Page Comparison

Overview

While profiling a FOLIO instance running Juniper release, we found out that it is using Mark and Sweep Garbage Collection phases but were not sure which Garbage Collection algorithm it was using. We created

Jira Legacy

server	System

Jira

JIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-201

- POC POC to investigate Garbage Collection in FOLIO modules

Requirements

Improve JMeter scripts creation process
Automate manual steps

Approach & Design

There are 2 approaches:

(A) Record HTTP(S) requests using Apache JMeter HTTP(S) Test Script Recorder as documented

https://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.html

Pros:

Easy to follow as it is already documented
Works fine for HTTP websites as it does not require SSL Certificate

Cons:

There are multiple steps to follow and could get lost if you miss anything
Manually open up a port for JMeter and update network proxy in browser settings/preferences - once you update network proxy settings, requests to outside network will stop working unless you revert to default proxy settings
For HTTPS websites, have to manually install SSL Certificate in browser
We have to manual set URL Patterns to exclude being captured by JMeter script
It captures a lot of redundant files such as HTML, gif, banners, images, config files
Need FOLIO UI modules installed

(B) Use open-source Blazemeter Chrome plugin

The plugin can be downloaded from https://chrome.google.com/webstore/detail/blazemeter-the-continuous/mbopgmdnpcbohhpnfglgohlbhfongabi/related

You will have to sign up to start using it. Sign-up is free.

Image Removed

Start recording: execute the workflow

Image Removed

Stop recording and import to JMX file:

Image Removed

Open imported JMX file in JMeter:

Image Removed

Pros:

Easy to install
Easy to record HTTP(S) request in browser and import to JMeter script
No need to set up proxy manually, everything is handled by the plugin
It is open-source with 70k+ users
The plugin is frequently maintained and upgraded
The plugin generates clean and accurate JMeter script with almost no junk files
Replay recorded JMeter script without any changes
Captures HTTP(S) calls to the backend which is difficult to track manually
The JMeter script can be edited in the browser before importing
Automates and simplifies JMeter script creation process

Cons:

Recorded JMeter script need to be refactored to comply with best practices
Need FOLIO UI modules installed

This plugin was tested against the check-in-check-out workflow and works well with all HTTP(S) verbs.

Outcome:

After weighing in both approaches and after discussing with the team, approach B was accepted.

Links

https://guide.blazemeter.com/hc/en-us/articles/206732579-The-BlazeMeter-Chrome-Extension-Record-JMeter-Selenium-or-Synchronized-JMeter-and-Selenium-The-BlazeMeter-Chrome-Extension:-Record-JMeter,-Selenium,-or-Synchronized-JMeter-and-Selenium

https://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.html

https://jmeter-plugins.org/

https://guide.blazemeter.com/hc/en-us/articles/115004970329-Chrome-Extension-Changelog-Chrome-Extension-Changelog. Investigate if we can improve performance by using G1 Garbage Collection which is an asynchronous and non-blocking process.

Investigation Garbage Collection

Use -Xlog:gc flag to enable logging when a module loads up. For example, we used mod-data-export to experiment.

Experiments:

All experiments were carried out on openjdk-11.0

Running mod-data-export locally on my laptop

java -jar -Xlog:gc=debug:file=gc.log target/mod-data-export-fat.jar

Image Added

JVM automatically selected G1 Garbage Collector. G1 (Garbage First) Garbage Collector is designed for applications running on multi-processor machines with large memory space.

My laptop is running on 6 cores

Image Added

2. Running mod-data-export in AWS ECS in m5.xlarge EC2 instance type

Image Added

JVM automatically selected Serial Garbage Collection. The Serial algorithm uses a single thread to do Garbage Collection. When JVM runs Garbage Collection, Java applications pause for few milliseconds.

mod-data-export is running inside a Docker Container which is running inside EC2 instance which is running Linux on 2 cores which are shared among 13 other backend FOLIO modules https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/cpu-options-supported-instances-values.html

Image Added

3. Tried to override using -XX:+AlwaysActAsServerClassMachine flag. However, Data Export App in UI fails to poll the backend and crashes frequently.

Image Added

4. Explicitly select G1 Garbage Collector when loading mod-data-export using flag XX:+UseG1GC. mod-data-export module becomes stable but Data Export App in UI fails to poll the backend and crashes frequently.

Image Added

5. Tested in a cap planning environment that is running 8 Cores(m5.2xlarge EC2 instance type) but JVM is still using Serial Garbage Collection

Image Added

Observations

Java 11 versions support the default Garbage Collection as G1. However, running multiple(13) docker container services shared in a single EC2 instance makes JVM select the Serial Garbage Collection algorithm. Serial Garbage Collection algorithm is a blocking process. As per https://dzone.com/articles/pitfalls-in-jvm-and-docker-defaults, JVM needs more CPU and memory across multiple CPU cores to select G1 Garbage Collection.

With the current infrastructure, JVM cannot support asynchronous non-blocking G1 Garbage Collection