PTF - Data Export Test Report (Honeysuckle)
- Varun Javalkar
Overview
- In this workflow, we are checking the performance of exporting MARC Bib records workflow (default profile - does not involve items and holdings) running in the Honeysuckle release - PERF-134 - Getting issue details... STATUS
We tested with 1, 2 virtual users for 30 minutes.
2. Check how many jobs can run in parallel - PERF-142 - Getting issue details... STATUS so data-export does not break
Backend:
- mod-data-export-3.0.3
- mod-source-record-storage-4.1.2
- mod-source-record-manager-2.4.2
- okapi-4.2.2
- mod-authtoken-2.6.0
Frontend:
- folio_data-export-3.0.1
Environment:
- 7.2 million UChi SRS records
- 7.2 million inventory records
- 63 FOLIO back-end modules deployed in 110 ECS services
- 3 okapi ECS services
- 10 m5.large EC2 instances
- 1 writer db.r5.xlarge 1 reader db.r5.xlarge AWS RDS instance
- INFO logging level
High-Level Summary
- As the number of instances to export increases from 1 - 500,000 for 1, 2 users, performance improves. For all cases, mod-data-export, mod-source-record-storage is stable and performant.
- data-export/job-executions API is polled every 3 seconds vs 5 seconds in Goldenrod. This means there are more requests overall but it also improves data-export execution. Data-export job completes faster(around 2 seconds) compared to the previous release. Other than that, data-export is stable and see performance improvements compared to the Goldenrod release.
- A new feature added for data-export is it will kill failed job every 6 hours
- How many data-export jobs can run in parallel? Multiple jobs can run in parallel and 2.75 Million instance records can be exported at any given point in time. If we try to export more than that for example 3 Million instance records, mod-data-export crashes with OOM(Out Of Memory)
Test Runs
1 user, 2 users, and comparison with Goldenrod release
Test | Total instances | Duration | OKAPI log level | 1 User - Avg Total time to Export instances - Ramp-up 0 seconds | 2 Users - Avg Total time to Export instances - Ramp-up 5 seconds | Improvement from Goldenrod (avg percentage) for both 1 and 2 users |
1. | 1 | 30 min | INFO | 3.18 seconds | 3.11 seconds | 63% |
2. | 5 | 30 min | INFO | 3.18 seconds | 3.13 seconds | 58% |
3. | 100 | 30 min | INFO | 3.19 seconds | 3.18 seconds | 52% |
4. | 10,000 | 30 min | INFO | 20.38 seconds | 39.79 seconds | 10% |
| 5. | 100,000 | 30 min | INFO | 2.97 minutes | 9.65 minutes | 9% |
| 6. | 500,000 | 30 min | INFO | 18 minutes | 33.5 minutes | 76% |
Slow API taking more than 100 ms to return for 1 user:
| API | 1 user - 10,000 instances(Avg) | 1 user - 100,000 instances(Avg) | 1 user - 500,000 instances(Avg) |
|---|---|---|---|
| POST_/data-export/fileDefinitions/{id}/upload | 145 milliseconds | 627 milliseconds | 1.82 seconds |
Fast queries involved during data-export job:
| SQL query | Response time (Avg) |
|---|---|
| SELECT jsonb FROM fs09000000_mod_data_export.job_executions WHERE (jsonb->>'id') = '99fbccb3-c9bc-4278-b2e2-af130ef87030' | 1 millisecond |
| SELECT fs09000000_mod_data_export.count_estimate('SELECT * FROM fs09000000_mod_data_export.job_executions WHERE id=''99fbccb3-c9bc-4278-b2e2-af130ef87030''') | 2 milliseconds |
CPU Utilization
CPU utilization increase as the number of instances increases from 100 to 500k. CPU reaches maximum for 100k, 500k instances but there is not much difference for 1 user and 2 users.
Service Memory Utilization
Memory increases gradually as we start running tests especially for mod-data-export and then stabilizes. For all other modules such as mod-source-record-storage, the okapi remains constant between 80% - 100%.
Check how many jobs can run in parallel
Multiple jobs can run in parallel but data-export fails if trying to export 3 Million instance records with the below configuration.
Current memory allocation to mod-data-export service in ECS task definition container:
Soft memory limit - 360 MB
Hard memory limit - 512 MB
Memory Utilization gradually increases from 101% to 141% as we increase the number of instance records where it eventually crashes.
| Number of inventory instance records (Millions) | Average Memory Utilization (%) |
|---|---|
| 1 | 101.66 |
| 2 | 102.77 |
| 2.5 | 124.4 |
| 2.75 | 136.11 |
| 3 | 141 (service fails with OOM) |
When trying to export 3M records, POST data-export/file-definitions/d63d8a83-e339-44b2-8a2f-41caaf080221/upload fails with 503
Appendix
For more raw data of the test run please see the attached test-report-honeysuckle.xlsx for Honeysuckle.