PERF-122 - Getting issue details... STATUS

Overview

The purpose of this document is to obtain baseline numbers for the OAI-PMH workflow from the edge API.  These numbers include typical resource usage for one user test and response times for different calls.

There are two types of calls, and the only difference is in "metadataPrefix" which can be "marc21" or "marc21_withholdings".  Each of these calls may have a "startDate" parameter.

Backend

• mod-oai-pmh:3.2.2, mod-oai-pmh:3.2.4
• edge-oai-pmh:2.3.0
• mod-inventory-storage:19.4.3
• okapi:4.2.2

Environment:

• 61 back-end modules deployed in 110 ECS services
• 3 okapi ECS services
• 8 m5.large  EC2 instances
• 2 db.r5.xlarge AWS RDS instance (1 reader, 1 writer)
• 1 mod-oai-pmh ECS services (to easy troubleshoot)
• INFO Okapi logging level
• Load balancer timeouts increased to 4000 seconds.
• nginx timeouts extended to 3600 seconds.

Summary

"marc21" metadataPrefix working much faster than "marc21_withholdings" . Few basics numbers about both matadataPrefix :

1) marc21 metadataPrefix:

• Response time for both tests (including start date and without it) is 6.5 - 6.6 seconds on 75%/95% percentiles;
• mod-oai-pmh CPU usage did not grow higher than ~30%. (usually it not higher than 15%);
• Other modules didn't show visible RAM usage. Ram usage of OAI_PMH was near 110 -119 %;
• RDS CPU usage usually stay on 15 % rate. 

2) marc21_withholdings metadataPrefix:

• Response time for test without  start date 483-495 seconds on 75%/95% percentiles;
• Response time for test with start date is 530-539 seconds on 75%/95% percentiles;
• Higher CPU usage on modules than in previous cases.
  • 170-200% on mod-inventory-storage;
  • 9-100% on mod OAI-PMH;
• RAM usage mod-oai-pmh was near 110 -119 %;
• RDS CPU usage was 60-70 %. 

3) The performance degrades as

• The <tenantId>_mod-oai-pmh.instances table is filled up with records. When the number of records approaches 40M, a call [oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=${APIKey}&start=${Date}] that used to take 7 minutes now takes longer than 40 minutes.
• After several runs (5-7), the oai-pmh calls take longer than 20 minutes even though the instances table is empty. mod-pubsub needs to be restarted for it to return results within 7 minutes again.

Please Note: that on ptf env. getaway timeout value on public load-balancer was increased from 400 to 4000 seconds. While default value on our prod env. is 400 seconds, calls that includes marc21_withholdings metadataPrefix won't work and will fail with 504 "Getaway Timeout".

Test runs

Detailed Results

Test 1

Service CPU usage

MOD OAI-PMH CPU usage 

MOD OAI-PMH Memory usage

Memory usage on other modules looks the same. 

DB CPU utilization:

Test 2

Service CPU usage:

mod-oai-pmh memory usage

DB CPU Usage 

Test 3

Service CPU usage:

MOD OAI-PMH CPU usage

RDS CPU usage

Test 4

Service CPU usage 

Mod Inventory storage CPU usage

MOD OAI-PMH cpu usage

RDS CPU usage

Test 5

Service CPU usage:

Relevant modules: 

• mod-oai-pmh - 165% -189%;
• mod-inventory-storage - 160%-180%;
• okapi - 30%;

Service Memory usage:

DB CPU usage:

Test 6 

Service CPU usage:

mod-inventory-storage - 150%-220%

mod-oai-pmh - 130 -170

okapi 40-5% 

RDS CPU usage: