Page Comparison

Overview

While profiling a FOLIO instance running Juniper release, we found out that it is using Mark and Sweep Garbage Collection phases but were not sure which Garbage Collection algorithm it was using. We created

...

. Investigate if we can improve performance by using G1 Garbage Collection which is an asynchronous and non-blocking process.

Investigation Garbage Collection

Use -Xlog:gc flag to enable logging when a module loads up. For example, we used mod-data-export to experiment. It was using

Experiments:

All experiments were carried out on openjdk-11. 0

Observations:

1. Running mod-data-export locally on my laptop

java -jar -Xlog:gc=debug:file=gc.log target/mod-data-export-fat.jar

Image Modified

JVM automatically selected G1 Garbage Collector. G1 (Garbage First) Garbage Collector is designed for applications running on multi-processor machines with large memory space. 

My laptop is running on 6 cores

Image Added

     2. Running mod-data-export in AWS ECS in m5.xlarge EC2 instance type

JVM automatically selected Serial Garbage Collection. The Serial algorithm uses a single thread to do Garbage Collection. When JVM runs Garbage Collection, Java applications pause for few milliseconds.

Start recording: execute the workflow

Image Removed

Stop recording and import to JMX file:

Image Removed

Open imported JMX file in JMeter:

Image Removed

Image Removed

Pros:

• Easy to install
• Easy to record HTTP(S) request in browser and import to JMeter script
• No need to set up proxy manually, everything is handled by the plugin
• It is open-source with 70k+ users
• The plugin is frequently maintained and upgraded
• The plugin generates clean and accurate JMeter script with almost no junk files 
• Replay recorded JMeter script without any changes
• Captures HTTP(S) calls to the backend which is difficult to track manually
• The JMeter script can be edited in the browser before importing
• Automates and simplifies JMeter script creation process

Cons:

• Recorded JMeter script need to be refactored to comply with best practices
• Need FOLIO UI modules installed

This plugin was tested against the check-in-check-out workflow and works well with all HTTP(S) verbs.

Outcome:

After weighing in both approaches and after discussing with the team, approach B was accepted. mod-data-export is running inside a Docker Container which is running inside EC2 instance which is running Linux on 2 cores which are shared among 13 other backend FOLIO modules https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/cpu-options-supported-instances-values.html

Image Added

       3. Tried to override using -XX:+AlwaysActAsServerClassMachine flag. However, Data Export App in UI fails to poll the backend and crashes frequently.

Image Added

Image Added

       4. Explicitly select G1 Garbage Collector when loading mod-data-export using flag XX:+UseG1GC. mod-data-export module becomes stable but Data Export App in UI fails to poll the backend and crashes frequently.

Image Added

         5. Tested in a cap planning environment that is running 8 Cores(m5.2xlarge EC2 instance type) but JVM is still using Serial Garbage Collection

Image Added 

Image Added

Observations

Java 11 versions support the default Garbage Collection as G1. However, running multiple(13) docker container services shared in a single EC2 instance makes JVM select the Serial Garbage Collection algorithm. Serial Garbage Collection algorithm is a blocking process. As per https://dzone.com/articles/pitfalls-in-jvm-and-docker-defaults, JVM needs more CPU and memory across multiple CPU cores to select G1 Garbage Collection.

With the current infrastructure, JVM cannot support asynchronous non-blocking G1 Garbage Collection

Links

https://guideserverfault.blazemeter.com/hcquestions/en-us/articles/206732579-The-BlazeMeter-Chrome-Extension-Record-JMeter-Selenium-or-Synchronized-JMeter-and-Selenium-The-BlazeMeter-Chrome-Extension:-Record-JMeter,-Selenium,-or-Synchronized-JMeter-and-Selenium691659/count-number-of-allowed-cpus-in-a-docker-container

http://www.herongyang.com/Java-GC/Serial-Collector-GC-Log-Message-Format.html

https://stackoverflow.com/questions/52474162/why-is-serialgc-chosen-over-g1gc

https://www.baeldung.com/jvm-garbage-collectors

https://jmeterdocs.apacheoracle.orgcom/usermanual/jmeter_proxy_step_by_step.htmlhttps://jmeter-plugins.org/javase/9/gctuning/available-collectors.htm#JSGCT-GUID-45794DA6-AB96-4856-A96D-FDE5F7DEE498

https://guide.blazemeterdzone.com/hc/en-us/articles/115004970329pitfalls-Chromein-Extensionjvm-Changelogand-Chromedocker-Extension-Changelogdefaults