JVM Garbage Collection
- Varun Javalkar
Overview
While profiling a FOLIO instance running Juniper release, we found out that it is using Mark and Sweep Garbage Collection phases but were not sure which Garbage Collection algorithm it was using. We created PERF-201 - Getting issue details... STATUS - POC to investigate Garbage Collection in FOLIO modules. Investigate if we can improve performance by using G1 Garbage Collection which is an asynchronous and non-blocking process.
Investigation Garbage Collection
Use -Xlog:gc flag to enable logging when a module loads up. For example, we used mod-data-export to experiment.
Experiments:
All experiments were carried out on openjdk-11.0
- Running mod-data-export locally on my laptop
java -jar -Xlog:gc=debug:file=gc.log target/mod-data-export-fat.jar
JVM automatically selected G1 Garbage Collector. G1 (Garbage First) Garbage Collector is designed for applications running on multi-processor machines with large memory space.
My laptop is running on 6 cores
2. Running mod-data-export in AWS ECS in m5.xlarge EC2 instance type
JVM automatically selected Serial Garbage Collection. The Serial algorithm uses a single thread to do Garbage Collection. When JVM runs Garbage Collection, Java applications pause for few milliseconds.
mod-data-export is running inside a Docker Container which is running inside EC2 instance which is running Linux on 2 cores which are shared among 13 other backend FOLIO modules https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/cpu-options-supported-instances-values.html
3. Tried to override using -XX:+AlwaysActAsServerClassMachine flag. However, Data Export App in UI fails to poll the backend and crashes frequently.
4. Explicitly select G1 Garbage Collector when loading mod-data-export using flag XX:+UseG1GC. mod-data-export module becomes stable but Data Export App in UI fails to poll the backend and crashes frequently.
5. Tested in a cap planning environment that is running 8 Cores(m5.2xlarge EC2 instance type) but JVM is still using Serial Garbage Collection
Observations
Java 11 versions support the default Garbage Collection as G1. However, running multiple(13) docker container services shared in a single EC2 instance makes JVM select the Serial Garbage Collection algorithm. Serial Garbage Collection algorithm is a blocking process. As per https://dzone.com/articles/pitfalls-in-jvm-and-docker-defaults, JVM needs more CPU and memory across multiple CPU cores to select G1 Garbage Collection.
With the current infrastructure, JVM cannot support asynchronous non-blocking G1 Garbage Collection
Links
https://serverfault.com/questions/691659/count-number-of-allowed-cpus-in-a-docker-container
http://www.herongyang.com/Java-GC/Serial-Collector-GC-Log-Message-Format.html
https://stackoverflow.com/questions/52474162/why-is-serialgc-chosen-over-g1gc
https://www.baeldung.com/jvm-garbage-collectors
https://dzone.com/articles/pitfalls-in-jvm-and-docker-defaults