FameFlower Test Results

• Overview
• Test Runs
• Results
  • 1. High level FameFlower results data
  • 2.  CPU Utilization comparisons
  • 3.  Memory trends
  • 4.  Database CPU trends  
  • 5.  Slow queries
  • 6. Missing indexes
  • 7. CPU Profiling result
• Summary
  • Issues

Overview

Using the Carrier-io framework for capturing and analyzing performance test results, the following tests for the export instance UUIDs workflows were executed.

The testing was provided with the modules:

Backend:
- mod-inventory-storage-19.1.2
- mod-inventory-14.1.3
- mod-authtoken-2.4.0
- mod-permissions-5.9.0
- okapi-2.38.0
Frontend:
- folio_inventory-2.0.2

Test Runs

• 30-min Runs for export instance UUIDs workflow:

  Test	Virtual Users	Duration	OKAPI log level	Profiled	Ramp up (total time in seconds)	Size of records
  1. FameFlower	1	30 min	INFO	No	5	10K~50K instances
  2. FameFlower	1	30 min	INFO	No	1	50K~100K instances
  3. FameFlower	1	30 min	INFO	Yes	10	10K~50K instances
  4. FameFlower	1	30 min	INFO	Yes	10	50K~100K instances
  5. FameFlower	5	30 min	INFO	No	50	10K~50K instances
  6. FameFlower	5	30 min	INFO	No	10	50K~100K instances
  7. FameFlower	5	30 min	INFO	Yes	50	10K~50K instances
  8. FameFlower	5	30 min	INFO	Yes	50	50K~100K instances

  
  

  Results

  *All numbers are in milliseconds except for those in the Delta % column, which indicates the difference in percentage going from 1 to 5 users, 10K~50K instances

  1. High level FameFlower results data

  1 and 5 users tests runs, 10K~50K instances

  
  

...

Folio build was deployed with 50+ ECS services installed randomly across 4 m5.large instances in the fcp1-pvt cluster and the database was created on the db.r5.xlarge AWS RDS instance. Logging level was set to default INFO.

...

During testing the workflow with 5 users and 50K~100K instances, mod-inventory-storage was crashing a few times due to OOM. There are 4 instances of mod-inventory-storage active in this test run. This means that it crashed 3 times and spun up new mod-inventory-storage instances

Image Modified

4. Database CPU trends  

...

CPU profiling of the most resources consuming mod-inventory-storage service showed 6 methods which had a high CPU usage and impact on the overall service performance.

Image Modified

fasterxml.jackson.databind.ObjectMapper.readValue method uses most of CPU capacity which leads to performance degradation

Image Modified

Summary

        See Attached FameFlower Performance Test Runs.xlsx for details 

• FOLIO performs better without being profiled when the tests are running 

...

• Most of failed requests were related to GET_/inventory/instances and GET_/instance-bulk/ids that use mod-inventory-storage service
• GET_/inventory/instances and GET_/instance-bulk/ids have failed responses even for 1 user (30 min test run)
• mod-inventory-storage was crashing a few times due to OutOfMemoryError during the test runs

• The workflow with more than 100K records become unresponsive even with 1 user

• The workflow with more than 5 users become unresponsive
• Memory Issues: 
  • mod-inventory-storage has noticeable to significant gains in memory used.  
• fasterxml.jackson.databind.ObjectMapper.readValue method of mod-inventory-storage service overuses CPU resources as there are a lot of JSON decoding, this implementation could be reviewed and improved to reduce operations with JSON

...