Overview
- In this workflow, we are checking the performance of exporting MARC Bib records workflow (default profile - does not involve items and holdings) running in the Honeysuckle release - PERF-134 - Getting issue details... STATUS
We tested with 1, 2 virtual users for 30 minutes.
2. Check how many jobs can run in parallel - PERF-142 - Getting issue details... STATUS so data-export does not break
Backend:
- mod-data-export-3.0.3
- mod-source-record-storage-4.1.2
- mod-source-record-manager-2.4.2
- okapi-4.2.2
- mod-authtoken-2.6.0
Frontend:
- folio_data-export-3.0.1
Environment:
- 7.2 million UChi SRS records
- 7.2 million inventory records
- 63 FOLIO back-end modules deployed in 110 ECS services
- 3 okapi ECS services
- 10 m5.large EC2 instances
- 1 writer db.r5.xlarge 1 reader db.r5.xlarge AWS RDS instance
- INFO logging level
High-Level Summary
- As the number of instances to export increases from 1 - 500,000 for 1, 2 users, performance improves. For all cases, mod-data-export, mod-source-record-storage is stable and performant.
- data-export/job-executions API is polled every 3 seconds vs 5 seconds in Goldenrod. This means there are more requests overall but it also improves data-export execution. Data-export job completes faster(around 2 seconds) compared to the previous release. Other than that, data-export is stable and see performance improvements compared to the Goldenrod release.
- A new feature added for data-export is it will kill failed job every 6 hours
Test Runs
1 user, 2 users, and comparison with Goldenrod release
Test | Total instances | Duration | OKAPI log level | 1 User - Avg Total time to Export instances - Ramp-up 0 seconds | 2 Users - Avg Total time to Export instances - Ramp-up 5 seconds | Improvement from Goldenrod (avg percentage) for both 1 and 2 users |
1. | 1 | 30 min | INFO | 3.18 seconds | 3.11 seconds | 63% |
2. | 5 | 30 min | INFO | 3.18 seconds | 3.13 seconds | 58% |
3. | 100 | 30 min | INFO | 3.19 seconds | 3.18 seconds | 52% |
4. | 10,000 | 30 min | INFO | 20.38 seconds | 39.79 seconds | 10% |
| 5. | 100,000 | 30 min | INFO | 2.97 minutes | 9.65 minutes | 9% |
| 6. | 500,000 | 30 min | INFO | 18 minutes | 33.5 minutes | 76% |
Slow API taking more than 100 ms to return for 1 user:
| API | 1 user - 10,000 instances(Avg) | 1 user - 100,000 instances(Avg) | 1 user - 500,000 instances(Avg) |
|---|---|---|---|
| POST_/data-export/fileDefinitions/{id}/upload | 145 milliseconds | 627 milliseconds | 1.82 seconds |
Fast queries involved during data-export job:
| SQL query | Response time (Avg) |
|---|---|
| SELECT jsonb FROM fs09000000_mod_data_export.job_executions WHERE (jsonb->>'id') = '99fbccb3-c9bc-4278-b2e2-af130ef87030' | 1 millisecond |
| SELECT fs09000000_mod_data_export.count_estimate('SELECT * FROM fs09000000_mod_data_export.job_executions WHERE id=''99fbccb3-c9bc-4278-b2e2-af130ef87030''') | 2 milliseconds |
CPU Utilization
CPU utilization increase as the number of instances increases from 100 to 500k. CPU reaches maximum for 100k, 500k instances but there is not much difference for 1 user and 2 users.
Service Memory Utilization
Memory increases gradually as we start running tests especially for mod-data-export and then stabilizes. For all other modules such as mod-source-record-storage, the okapi remains constant between 80% - 100%.
Check how many jobs can run in parallel
Current memory allocation to mod-data-export service in ECS task definition container:
Soft memory limit - 360 MB
Hard memory limit - 512 MB
| Number of inventory instance records (Millions) | Average Memory Utilization (%) |
|---|---|
| 1 | 101.66 |
| 2 | 102.77 |
| 2.5 | 124.4 |
| 2.75 | 136.11 |
| 3 | 141 (service fails with OOM) |
When trying to export 3M records, POST data-export/file-definitions/d63d8a83-e339-44b2-8a2f-41caaf080221/upload fails with 503
Appendix
For more raw data of the test run please see the attached test-report-honeysuckle.xlsx for Honeysuckle.