OAI-PMH Baseline report (Honeysuckle)
- PERF-122Getting issue details... STATUS
Overview
The purpose of this document is to obtain baseline numbers for the OAI-PMH workflow from the edge API. These numbers include typical resource usage for one user test and response times for different calls.
There are two types of calls, and the only difference is in "metadataPrefix" which can be "marc21" or "marc21_withholdings". Each of these calls may have a "startDate" parameter.
Backend
- mod-oai-pmh:3.2.2, mod-oai-pmh:3.2.4
- edge-oai-pmh:2.3.0
- mod-inventory-storage:19.4.3
- okapi:4.2.2
Environment:
- 61 back-end modules deployed in 110 ECS services
- 3 okapi ECS services
- 8 m5.large EC2 instances
- 2 db.r5.xlarge AWS RDS instance (1 reader, 1 writer)
- 1 mod-oai-pmh ECS services (to easy troubleshoot)
- INFO Okapi logging level
- Load balancer timeouts increased to 4000 seconds.
- nginx timeouts extended to 3600 seconds.
Summary
"marc21" metadataPrefix working much faster than "marc21_withholdings" . Few basics numbers about both matadataPrefix :
1) marc21 metadataPrefix:
- Response time for both tests (including start date and without it) is 6.5 - 6.6 seconds on 75%/95% percentiles;
- mod-oai-pmh CPU usage did not grow higher than ~30%. (usually it not higher than 15%);
- Other modules didn't show visible RAM usage. Ram usage of OAI_PMH was near 110 -119 %;
- RDS CPU usage usually stay on 15 % rate.
2) marc21_withholdings metadataPrefix:
- Response time for test without start date 483-495 seconds on 75%/95% percentiles;
- Response time for test with start date is 530-539 seconds on 75%/95% percentiles;
- Higher CPU usage on modules than in previous cases.
- 170-200% on mod-inventory-storage;
- 9-100% on mod OAI-PMH;
- RAM usage mod-oai-pmh was near 110 -119 %;
- RDS CPU usage was 60-70 %.
3) The performance degrades as
- The <tenantId>_mod-oai-pmh.instances table is filled up with records. When the number of records approaches 40M, a call [oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=${APIKey}&start=${Date}] that used to take 7 minutes now takes longer than 40 minutes.
- After several runs (5-7), the oai-pmh calls take longer than 20 minutes even though the instances table is empty. mod-pubsub needs to be restarted for it to return results within 7 minutes again.
Request | Average (sec) | 75th %tile (sec) | 95th %tile (sec) | Request Rate (RPS) | note |
---|---|---|---|---|---|
/oai?verb=ListRecords&metadataPrefix=marc21&apikey=<apiKey> | 6.562 | 6.592 | 6.661 | 0.15 | |
/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey> | 452.200 | 483.202 | 495.325 | 0.002 | |
/oai?verb=ListRecords&metadataPrefix=marc21&apikey=<apiKey>&start=<date> | 6.566 | 6.592 | 6.634 | 0.15 | |
/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey> &start=<date> | 521.345 | 530.289 | 539.200 | 0.002 | |
/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey> &start=<date> | 462.685 | 471.778 | 485.882 | 0.002 | Truncated mod_oai_pmh.instances table |
Please Note: that on ptf env. getaway timeout value on public load-balancer was increased from 400 to 4000 seconds. While default value on our prod env. is 400 seconds, calls that includes marc21_withholdings metadataPrefix won't work and will fail with 504 "Getaway Timeout".
Test runs
Test | Virtual Users | Request | Duration (sec) | Version OAI_PMH | Notes |
---|---|---|---|---|---|
1 | 1 | GET https://<edge_url>/oai?verb=ListRecords&metadataPrefix=marc21&apikey=<apiKey> | 1800 | mod-oai-pmh:3.2.2 | |
2 | 1 | GET https://<edge_url>/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey> | 1800 | mod-oai-pmh:3.2.2 | |
3 | 1 | GET https://<edge_url>/oai?verb=ListRecords&metadataPrefix=marc21&apikey=<apiKey>&start=<date> | 1800 | mod-oai-pmh:3.2.2 | |
4 | 1 | GET https://<edge_url>/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey> &start=<date> | 1800 | mod-oai-pmh:3.2.2 | |
5 | 1 | GET https://<edge_url>/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey> &start=<date> | 1800 | mod-oai-pmh:3.2.4 | Truncated _mod_oai_pmh.instances table |
6 | 1 | GET https://<edge_url>/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey> | 1800 | mod-oai-pmh:3.2.4 | Truncated _mod_oai_pmh.instances table |
Detailed Results
Test 1
Request | Total calls | Min (sec) | 50th pct (sec) | 75th pct (sec) | 95th pct (sec) | 99th pct (sec) | Max(sec) |
---|---|---|---|---|---|---|---|
/oai?verb=ListRecords&metadataPrefix=marc21&apikey=[APIKey] | 283 | 5.307 | 6.562 | 6.592 | 6.661 | 6.779 | 9.683 |
Service CPU usage
MOD OAI-PMH CPU usage
MOD OAI-PMH Memory usage
Memory usage on other modules looks the same.
DB CPU utilization:
Test 2
Request | Total | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max |
---|---|---|---|---|---|---|---|
/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=[APIKey] | 5 | 427.606 | 452.200 | 483.202 | 495.325 | 495.325 | 495.325 |
Service CPU usage:
mod-oai-pmh memory usage
DB CPU Usage