FameFlower Test Results

...

In this workflow we are checking the performance of the loading items on instance record (in holdings accordion) workflow running in the Fameflower release.  We will test it with 1, 5, 8, and 20 virtual users for 30 minutes.  A longevity test will also be executed to see if there were memory issues.

...

• 55 back-end modules deployed in 110 ECS services
• 3 okapi ECS services
• 8 m5.large  EC2 instances
• 1  db.r5.xlarge AWS RDS instance
• INFO logging level

High Level Summary

1. Overall check in, check out time in seconds

   1. Average check in time is 1.51 seconds for a typical use case of 8 users, 1.65 seconds for 20 users
   2. Average check out time is 1.75 seconds for a typical use case of 8 users, 1.90 seconds for 20 users
2. Slow APIs taking more than 100ms to run 
   1. POST checkout-by-barcode
   2. POST checkin-by-barcode 
   3. Get circulation/loans
   4. Get inventory/items
3. mod-inventory-storage log warnings for missing indexes - 64K lines in  45 minutes run. Logging level could be reduced to WARNING or INFO, but at the cost of having less data to work with should there be a need to troubleshoot. Adding the missing indexes could improve performance while stop logging these warnings 

   Jira Legacy
   server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key CIRCSTORE-215

4. JVM profiling shows JSON de/serialization operations one of the slowest operations, totaling more CPU time than other calls.  Since FOLIO modules retrieves and stores JSON objects, making sure that serialization and deserialization JSON efficient is essential, see Recommended Improvements

Test Runs

1. Overall load items workflow time in seconds

   	Average (seconds)	50th %tile (seconds)	75th %tile (seconds)	95th %tile  (seconds)
   1 user	1.015	1.234	0.961	14.27767 s	1.071	1.409	1.322	1.653
   5 users	1.2361.488	1.156	1.3931.464	1.8691.704	2.219
   8 users	1.5121.751	1.403	1.8521.741	2.0312.02	2.274
   20 users	1.6491.898	1.5351.996	1.896	2.2112.252	2.539

   
   

2. Slow APIs taking more than 100 ms to return

   API	1 user (75th %tile)	5 users (75th %tile)	8 users (75th %tile)	20 Users (75th %tile)

   GET /

   GET source-storage/formattedRecords/{id}

   5.

   24

   32 s

   905 ms906 ms988 msGET_/

   19.92 s	34.83 s	1.46 min
   GET inventory/instances keyword all "aba"	355.60 ms	753.86 ms	909.02 ms	2.87 s
   POST

   _/

   circulation/check-out-by-barcode

   283

   679.72 ms

   346 ms449 ms479 ms

   1.04 s	1.11 s	1.45 s
   GET inventory/items

   217

   164.45 ms

   232

   251.71 ms

   237

   286.87 ms

   281

   405.85 ms
   GET inventory/instances/{id}	119.93 ms	115.98 ms	123.88 ms	131.89 ms
   GET circulation/loans	221.95 ms	254.08 ms	251.45 ms	256.18 ms
   GET locations	221.09 ms	142.89 ms	152.26 ms	205.71 ms
   GET circulation/requests	81.77 ms	212.38 ms	257.80 ms	443.79 ms

   
   

3. Excess logging of missing indexes - 64K lines in  45 minutes run. Logging level could be reduced to WARNING or INFO, but at the cost of having less data to work with should there be a need to troubleshoot.

   
   

4. JVM profiling shows JSON de/serialization operations one of the slowest operations.

Test Runs

formattedRecords

Results

JVM Profiling

• Overall slow methods (between the modules profiled: okapi, mod-inventory, mod-inventory-storage)

...

The following WARNING statements of missing indexes were generated during a test run and logged by mod-source-inventoryrecord-storage:

...

Appendix

See Attached FameFlower Performance Test Runs.xlsx for details