FameFlower Test Results

Overview

PERF-40 - Getting issue details... STATUS

In this workflow we are checking the performance of the loading items on instance record (in holdings accordion) workflow running in the Fameflower release. We will test it with 1, 5, 8, and 20 virtual users for 30 minutes. A longevity test will also be executed to see if there were memory issues.

Backend:
- mod-inventory-storage-19.1.2
- mod-inventory-14.1.3
- mod-authtoken-2.4.0
- mod-permissions-5.9.0
- okapi-2.38.0

- mod-circulation-18.0.9

- mod-circulation-storage-11.0.8

Frontend:
- folio_inventory-2.0.2

Environment:

55 back-end modules deployed in 110 ECS services
3 okapi ECS services
8 m5.large EC2 instances
1 db.r5.xlarge AWS RDS instance
INFO logging level

High Level Summary

Overall load items time in seconds

	Average (seconds)		50th %tile (seconds)		75th %tile (seconds)		95th %tile (seconds)

1 user	1.015	1.234	0.96	1.277	1.071	1.409	1.322	1.653
5 users	1.236	1.488	1.156	1.393	1.464	1.869	1.704	2.219
8 users	1.512	1.751	1.403	1.852	1.741	2.031	2.02	2.274
20 users	1.649	1.898	1.535	1.996	1.896	2.211	2.252	2.539

Slow APIs taking more than 100 ms to return

API	1 user (75th %tile)	5 users (75th %tile)	8 users (75th %tile)	20 Users (75th %tile)
GET /source-storage/formattedRecords/{id}	5.24 s	905 ms	906 ms	988 ms
GET_/inventory/instances keyword all "aba"
POST_/circulation/check-out-by-barcode	283 ms	346 ms	449 ms	479 ms
GET inventory/items	217 ms	232 ms	237 ms	281 ms
GET inventory/instances/{id}
GET circulation/loans

Excess logging of missing indexes - 64K lines in 45 minutes run. Logging level could be reduced to WARNING or INFO, but at the cost of having less data to work with should there be a need to troubleshoot.
JVM profiling shows JSON de/serialization operations one of the slowest operations.

Test Runs

Test	Virtual Users	Duration	OKAPI log level	Profiled	Ramp up (total time in seconds)
1.	1	30 min	INFO	No	1
2.	5	30 min	INFO	No	50
3.	8	30 min	INFO	No	80
4.	20	30 min	INFO	No	200
5.	1	30 min	INFO	Yes	1
6.	5	30 min	INFO	Yes	50
7.	8	30 min	INFO	Yes	80
8.	20	30 min	INFO	Yes	200

Results

JVM Profiling

Overall slow methods (between the modules profiled: okapi, mod-inventory, mod-inventory-storage)

To see more drill down these methods at: http://ec2-3-93-19-104.compute-1.amazonaws.com/grafana/d/b2iP-9hiz/java-profiling-folio?orgId=1&var-service_tag=okapi&var-service_tag=mod-inventory-storage&var-service_tag=mod-inventory&var-aggregation=10s&from=1590418466176&to=1590420349447

Only slow Okapi methods:

When drilling down org.folio.okapi.managers.ProxyService.getModulesForRequest , we get the following tree. To see more click here: http://ec2-3-93-19-104.compute-1.amazonaws.com/grafana/d/U9JtDPLWz/stacktrace?orgId=1&class=org.folio.okapi.managers.ProxyService&method=getModulesForRequest&from=1590418466176&to=1590420349447

Slow mod-inventory methods:

Can drill down these methods for more information at : http://ec2-3-93-19-104.compute-1.amazonaws.com/grafana/d/b2iP-9hiz/java-profiling-folio?orgId=1&from=1589940195166&to=1589943457693&var-service_tag=mod-circulation&var-aggregation=10s

Database

Database does not show much CPU usage for 1, 5, 8 and 20 users runs.

The following WARNING statements of missing indexes were generated during a test run and logged by mod-inventory-storage:

CPU Utilization

	1 user		5 users		8 users		20 users
	Average %	Range %	Average %	Range %	Average %	Range %	Average %	Range %
Okapi	2.12	0.29-8.94
mod-inventory
mod-inventory-storage
mod-circulation
mod-circulation-storage

Memory

Memory was stable throughout the runs, only a spike here or there, but in a 30 minutes run they were consistent.

	1 user	5 users	8 users	20 users
	Average	Average	Average	Average
Okapi	40%	42%	40%	40%
mod-inventory	40%	43%	44%	43%
mod-inventory-storage	40%	40%	41%	41%
mod-circulation	72%	75%	72%	74%
mod-circulation-storage	32%	32%	32%	32%

Appendix

See Attached FameFlower Performance Test Runs.xlsx for details

Loading items on instance record (in holdings accordion) Test Report