| Table of Contents | ||
|---|---|---|
|
Overview
- In this workflow, we are checking These tests are run to investigate the performance of exporting MARC Bib records workflow (default profile - does not involve items and holdings) running in the Honeysuckle release - difference of mod-data-export with logLevel=info vs logLevel=warn for Juniper release. This testing was part of
Jira Legacy server System Jira columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERFMDEXP-134
...
where we observed that mod-data-export is continuously writing a lot of data to log. For 100K DE job, mod-data-export is writing 42Million records. This could also result in a crash if there is not enough CPU, memory allocated to mod-data-export.394
2. Check how many jobs can run in parallel -
| Jira Legacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
Backend:
- mod-data-export-34.0.31.1 (snapshot version)
- mod-source-record-storage-45.1.24
- mod-source-record-manager-23.41.23
- okapi-4.28.2
- mod-authtoken-2.68.0
Frontend:
- folio_data-export-34.1.0.1
Environment:
- 7.2 million UChi SRS records
- 7.2 8 million inventory records
- 63 74 FOLIO back-end modules deployed in 110 144 ECS services
- 3 okapi ECS services
- 10 12 m5.large EC2 instances
- 1 writer db.r5r6g.xlarge 1 reader db.r5r6g.xlarge AWS RDS instance
- INFO logging level / WARN looging level
High-Level Summary
- As the number of instances to export increases from 1 - 500,000 for 1, 2 users, performance improves. For all cases, mod-data-export, mod-source-record-storage is stable and performant.With WARN level logging, seeing 9% improvement in memory utilization
- data-export/job-executions API is polled every 3 seconds vs 5 seconds in Goldenrod. This means there are more requests overall but it also improves data-export execution. Data-export job completes faster(around 2 seconds) compared to the previous release. Other than that, data-export is stable and see performance improvements compared to the Goldenrod release.
- A new feature added for data-export is it will kill failed job every 6 hours
- How many data-export jobs can run in parallel? Multiple jobs can run in parallel and 2.75 Million instance records can be exported at any given point in time. If we try to export more than that for example 3 Million instance records, mod-data-export crashes with OOM(Out Of Memory)
Test Runs
1 user , 2 users, and comparison with Goldenrod release- INFO level logging vs WARN level logging
Test | Total instances | Duration | OKAPI mod-data-export log level | 1 User - Avg Total time to Export instances - Ramp-up 0 seconds | 2 Users - Avg Total time to Export instances - Ramp-up 5 seconds | Improvement from Goldenrod (avg percentage) for both 1 and 2 users | |||
1. | 1 | 30 min | INFO | 3.18 seconds | 3.11 seconds | 63% | |||
2. | 5 | 30 min | INFO | 3.18 seconds | 3.13 seconds | 58% | |||
3. | 100 | 30 min | INFO | 3.19 seconds | 3.18 seconds | 52% | |||
4. | 10,000 | 30 min | INFO | 20.38 seconds | 39.79 seconds | 10% | |||
| 5. | 100complete exporting all instances, holdings, and items | ||||||||
| 1. | 100,000 | 30 min | INFO | 2.97 minutes9 | |||||
| 2.65 minutes | 9% | 6. | 500 | 100,000 | 30 min | INFOWARN | 18 minutes | 33.5 minutes | 76% |
Slow API taking more than 100 ms to return for 1 user:
...