Overview

In this workflow, we are checking the performance of exporting MARC Bib records workflow (default profile - does not involve items and holdings) running in the Fameflower release - PERF-91 - Getting issue details... STATUS

We tested with 1, 2 virtual users for 30 minutes.

Backend:

mod-data-export-2.1.2
mod-source-record-storage-4.0.2
mod-source-record-manager-2.3.1
okapi-3.1.1

Frontend:

folio_data-export-2.0.1

Environment:

7.2 million UChi SRS records
7.2 million inventory records
63 FOLIO back-end modules deployed in 110 ECS services
3 okapi ECS services
8 m5.large EC2 instances
1 db.r5.xlarge AWS RDS instance
INFO logging level

High-Level Summary

As the number of instances to export increases from 1 - 500,000 for 1, 2 users, performance improves and then becomes stable relatively.
Overall, this is a huge performance improvement compared to the previous Fameflower release

Test Runs

1 user and 2 users

Test	Total instances	Duration	OKAPI log level	OKAPI Version	Profiled	1 User - Avg Total time to Export instances - Ramp-up 0 seconds	2 Users - Avg Total time to Export instances - Ramp-up 5 seconds
1.	1	30 min	INFO	3.1.1	No	5.1 seconds	5 seconds
2.	5	30 min	INFO	3.1.1	No	5.3 seconds	5.4 seconds
3.	100	30 min	INFO	3.1.1	No	5.6 seconds	6.1 seconds
4.	10,000	30 min	INFO	3.1.1	No	4.78 minutes	8.3 minutes
5.	100,000	30 min	INFO	3.1.1	No	4.5 minutes	9 minutes
6.	500,000	30 min	INFO	3.1.1	No	23.23 minutes	43.5 minutes

Slow API taking more than 100 ms to return for 1 user:

API	1 user - 100 instances(Avg)	1 user - 10,000 instances(Avg)	1 user - 100,000 instances(Avg)	1 user - 500,000 instances(Avg)
POST_/data-export/fileDefinitions/{id}/upload	66 millisec	5.8 seconds	5.9 seconds	22.1 seconds

CPU Utilization:

CPU utilization stabilizes as the number of instances gradually increases.

mod-data-export:

mod-source-record-storage:

Appendix

For more raw data of the test run please see the attached data-export-test-report.xls for Goldenrod.

Browser not supported