Overview
While profiling a FOLIO instance running Juniper release, we found out that it is using Mark and Sweep Garbage Collection phases but were not sure which Garbage Collection algorithm it was using. We created
- POC to investigate Garbage Collection in FOLIO modulesJira Legacy server System Jira columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-201
...
Use -Xlog:gc flag to enable logging when a module loads up. For example, we used mod-data-export to experiment.
Experiments:
All experiments were carried out on openjdk-11.0
- Running mod-data-export locally on my laptop
java -jar -Xlog:gc=debug:file=gc.log target/mod-data-export-fat.jar
JVM automatically selected G1 Garbage Collector. G1 (Garbage First) Garbage Collector is designed for applications running on multi-processor machines with large memory space.
...
2. Running mod-data-export in AWS ECS in m5.xlarge EC2 instance type
JVM automatically selected Serial Garbage Collection. The Serial algorithm uses a single thread to do Garbage Collection. When JVM runs Garbage Collection, Java applications pause for few milliseconds.
mod-data-export is running inside a Docker Container which is running inside EC2 instance which is running Linux on 2 cores which are shared among 13 other backend FOLIO modules https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/cpu-options-supported-instances-values.html
3. Tried to override using -XX:+AlwaysActAsServerClassMachine flag. However, Data Export App in UI fails to poll the backend and crashes frequently.
3 4. Explicitly select G1 Garbage Collector when loading mod-data-export using flag XX:+UseG1GC. Also, tried to override using -XX:+AlwaysActAsServerClassMachine flag. However, mod-data-export module becomes stable but Data Export App in UI fails to poll the backend and crashes frequently.
45. Tested in a cap planning environment that is running 8 Cores(m5.2xlarge EC2 instance type) but JVM is still using Serial Garbage Collection
Observations
For Java 9+ version, Java 11 versions support the default Garbage Collection is as G1. So when running locally on my laptop, JVM automatically selected G1. However, there is a caveat that there must be enough CPU and memory resources in a multi-core OS. While EC2 instance with m5.xlarge instance type is running with default 2 CPU Cores. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/cpu-options-supported-instances-values.htmlHowever, running multiple(13) docker container services shared in a single EC2 instance makes JVM select the Serial Garbage Collection algorithm. Serial Garbage Collection algorithm is a blocking process. As per https://dzone.com/articles/pitfalls-in-jvm-and-docker-defaults, JVM needs more CPU and memory across multiple CPU cores to select G1 Garbage Collection which is a non-blocking and asynchronous process.
Links
https://serverfault.com/questions/691659/count-number-of-allowed-cpus-in-a-docker-container
...