PERF-122 - Getting issue details... STATUS

Overview

The purpose of this document is to obtain baseline numbers for the OAI-PMH workflow from the edge API. These numbers include typical resource usage for one user test and response times for different calls.

There are two types of calls, and the only difference is in "metadataPrefix" which can be "marc21" or "marc21_withholdings". Each of these calls may have a "startDate" parameter.

Backend

mod-oai-pmh:3.2.2, mod-oai-pmh:3.2.4
edge-oai-pmh:2.3.0
mod-inventory-storage:19.4.3
okapi:4.2.2

Environment:

61 back-end modules deployed in 110 ECS services
3 okapi ECS services
8 m5.large EC2 instances
2 db.r5.xlarge AWS RDS instance (1 reader, 1 writer)
1 mod-oai-pmh ECS services (to easy troubleshoot)
INFO Okapi logging level
Load balancer timeouts increased to 4000 seconds.
nginx timeouts extended to 3600 seconds.

Summary

"marc21" metadataPrefix working much faster than "marc21_withholdings" . Few basics numbers about both matadataPrefix :

1) marc21 metadataPrefix:

Response time for both tests (including start date and without it) is 6.5 - 6.6 seconds on 75%/95% percentiles;
mod-oai-pmh CPU usage did not grow higher than ~30%. (usually it not higher than 15%);
Other modules didn't show visible RAM usage. Ram usage of OAI_PMH was near 110 -119 %;
RDS CPU usage usually stay on 15 % rate.

2) marc21_withholdings metadataPrefix:

Response time for test without start date 483-495 seconds on 75%/95% percentiles;
Response time for test with start date is 530-539 seconds on 75%/95% percentiles;
Higher CPU usage on modules than in previous cases.
- 170-200% on mod-inventory-storage;
- 9-100% on mod OAI-PMH;
RAM usage mod-oai-pmh was near 110 -119 %;
RDS CPU usage was 60-70 %.

3) The performance degrades as

The <tenantId>_mod-oai-pmh.instances table is filled up with records. When the number of records approaches 40M, a call [oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=${APIKey}&start=${Date}] that used to take 7 minutes now takes longer than 40 minutes.
After several runs (5-7), the oai-pmh calls take longer than 20 minutes even though the instances table is empty. mod-pubsub needs to be restarted for it to return results within 7 minutes again.

Request	Average (sec)	75th %tile (sec)	95th %tile (sec)	Request Rate (RPS)	note
/oai?verb=ListRecords&metadataPrefix=marc21&apikey=<apiKey>	6.562	6.592	6.661	0.15
/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey>	452.200	483.202	495.325	0.002
/oai?verb=ListRecords&metadataPrefix=marc21&apikey=<apiKey>&start=<date>	6.566	6.592	6.634	0.15
/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey> &start=<date>	521.345	530.289	539.200	0.002
/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey> &start=<date>	462.685	471.778	485.882	0.002	Truncated mod_oai_pmh.instances table

Please Note: that on ptf env. getaway timeout value on public load-balancer was increased from 400 to 4000 seconds. While default value on our prod env. is 400 seconds, calls that includes marc21_withholdings metadataPrefix won't work and will fail with 504 "Getaway Timeout".

Test runs

Test	Virtual Users	Request	Duration (sec)	Version OAI_PMH	Notes
1	1	GET https://<edge_url>/oai?verb=ListRecords&metadataPrefix=marc21&apikey=<apiKey>	1800	mod-oai-pmh:3.2.2
2	1	GET https://<edge_url>/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey>	1800	mod-oai-pmh:3.2.2
3	1	GET https://<edge_url>/oai?verb=ListRecords&metadataPrefix=marc21&apikey=<apiKey>&start=<date>	1800	mod-oai-pmh:3.2.2
4	1	GET https://<edge_url>/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey> &start=<date>	1800	mod-oai-pmh:3.2.2
5	1	GET https://<edge_url>/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey> &start=<date>	1800	mod-oai-pmh:3.2.4	Truncated _mod_oai_pmh.instances table
6	1	GET https://<edge_url>/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey>	1800	mod-oai-pmh:3.2.4	Truncated _mod_oai_pmh.instances table

Browser not supported