FameFlower Test Results

• Overview
• Test Runs
• Results
  • 1. High level FameFlower results data
  • 2.  CPU Utilization comparisons
  • 3.  Memory trends
  • 4.  Database CPU trends  
  • 5.  Slow queries
  • 6. Missing indexes
• Summary
  • Issues

Overview

Using the Carrier-io framework for capturing and analyzing performance test results, the following tests for the export instance UUIDs workflows were executed.

The testing was provided with the modules:

Backend:
- mod-inventory-storage-19.1.2
- mod-inventory-14.1.3
- mod-authtoken-2.4.0
- mod-permissions-5.9.0
- okapi-2.38.0
Frontend:
- folio_inventory-2.0.2

Test Runs

• 30-min Runs for export instance UUIDs workflow:

  Test	Virtual Users	Duration	OKAPI log level	Profiled	Ramp up (total time in seconds)	Size of records
  1. FameFlower	1	30 min	INFO	No	5	10K~50K instances
  2. FameFlower	1	30 min	INFO	No	1	50K~100K instances
  3. FameFlower	1	30 min	INFO	Yes	10	10K~50K instances
  4. FameFlower	1	30 min	INFO	Yes	10	50K~100K instances
  5. FameFlower	5	30 min	INFO	No	50	10K~50K instances
  6. FameFlower	5	30 min	INFO	No	10	50K~100K instances
  7. FameFlower	5	30 min	INFO	Yes	50	10K~50K instances
  8. FameFlower	5	30 min	INFO	Yes	50	50K~100K instances

  
  

  Results

  *All numbers are in milliseconds except for those in the Delta % column, which indicates the difference in percentage going from 1 to 5 users, 10K~50K instances

  1. High level FameFlower results data

  1 and 5 users tests runs, 10K~50K instances

  
  

Image Modified

1 and 5 users tests runs, 50K~100K instancesImage Removed

• The chart shows the overall high-level API stats obtained by JMeter calling various APIs in the save instance UUIDs worfklow. It breaks down average response times for 1 and 5 users tests per API call. 

...

The charts below offer a clearer side-by-side comparison for the 1 and 5 users tests runs, 50K~100K instances

Image Added

A side-by-side comparison for the 1 and 5 users tests runs, 10K~50K instances

A side-by-side comparison for the 1 and 5 users tests runs, 50K~100K instances

...

Folio build was deployed with 50+ ECS services installed randomly across 4 m5.large instances in the fcp1-pvt cluster and the database was created on the db.r5.xlarge AWS RDS instance. Logging level was set to default INFO.

...

The service is 99% of allocated RAM memory.

During testing the workflow with 5 users and 50K~100K instances, mod-inventory-storage was crashing a few times due to OOM. There are 4 instances of mod-inventory-storage active in this test run. This means that it crashed 3 times and spun up new mod-inventory-storage instances

Image Modified

4. Database CPU trends  

...

Summary

        See Attached FameFlower Performance Test Runs.xlsx for details 

• FOLIO performs better without being profiled when the tests are running 

...

• Most of failed requests were related to GET_/inventory/instances and GET_/instance-bulk/ids that use mod-inventory-storage service
• GET_/inventory/instances and GET_/instance-bulk/ids have failed responses even for 1 user (30 min test run)
• mod-inventory-storage was crashing a few times due to OutOfMemoryError during the test runs

• The workflow with more than 100K records become unresponsive even with 1 user

• The workflow with more than 5 users become unresponsive
• Memory Issues: 
  • mod-inventory-storage has noticeable to significant gains in memory used.  
• fasterxml.jackson.databind.ObjectMapper.readValue method of mod-inventory-storage service overuses CPU resources as there are a lot of JSON decoding, this implementation could be reviewed and improved to reduce operations with JSON

...