Overview

• The purpose of the OAI-PMH Full Harvesting tests and Incremental Harvesting tests is to measure performance of Poppy release and to find possible issues, bottlenecks per PERF-706 - Getting issue details... STATUS on MCPT environment.

• Two jMeter scripts for 2 scenarios were prepared to concurrently execute OAI-PMH by initiating harvesting from the member tenant level. To simulate the behaviour of EBSCO Harvester a delay after each request was used (150-300 ms) in the script.

• First script should trigger two full havests on 62 tenants creating 124 concurrent harvests. Second scenario should trigger 1 full and 1 incremental where tenants have 10k, 100k and 500k accordingly. In this second scenario incremental harvests should start from 10k, then 100k and 500k. 
• In total, 124 harvests for the first scenario and 143 harvests during 2 scenario (62 full + 57 incremental with 10k, + 23 with 100k + 1 with 500k).
• Baseline test for 1 Full + 1 incremental with 500k on 1 tenant should be run against tenant with highest number of instances.  

Summary

• In mcpt OAI-PMH can concurrently perform 124 harvests - 2 full harvests on each 62 member tenants with constant average throughput 10 requests or 1000 records per second during the test.
• Average response times (RTs) for /oai/records?verb=ListRecords&apikey=[APIKey]&resumptionToken=[resumptionToken] request in one tenant test - 0.580 sec,, in 62 tenants test - 18.3 sec. Harvesting work faster when RTs low and it happen when number of harvests decrease. Response times for 2 full harvest on 1 tenant only - 0.540 sec.
• When a number of harvests increase from 1 to 14 a total throughput grow up to 10 request per second. If number of harvests higher than 14 the throughput do not change. Just response times grow.
• Duration for 2 full harvests on 62 tenants - 4 hours 30 minutes in scenario 1
• Duration for 10k incremental harvest - 25 min, 100k - 1 hour 52 minutes, 500k - 3 hours - in scenario 2
• If to run 1 full and 1 incremental 500k on 1 tenant we have full harvest duration - 01:26:19. Incremental harvest duration - 01:09:46. Average response time 0.601 second. 
• Duration of full harvest triggered from central tenant level for all 62 tenants running sequentially - 13 hours 22 minutes.
• CPU utilization in mod-oai-pmh didn't exceed 44% over all tests.
• Memory utilization  in mod-oai-pmh was not higher 49%. Mod-inventory - 61% in scenario 2, 85% in scenario 1 (here it grew to this level because two 10k DI create jobs were tested along the oai-pmh). Additional DI along the test didn't affect oai-pmh response times. No memory leak trends found. More info in resource utilization table.
• RDS CPU utilization was 25% for first scenario, not higher 30% at the beginning of test in second scenario.
• DB connections - 700 in first scenario with SRS record source, 1400 with SRS + Inventory and 1350 in second scenario with SRS.

Recommendations & Jiras

Test Runs & Results

Two jMeter scripts for 2 scenarios were prepared to concurrently execute OAI-PMH by initiating harvesting from the member tenant level. To simulate the behaviour of EBSCO Harvester a delay after each request was used (150-300 ms) in the script.

In total, 124 harvests were executed concurrently during the test for the first scenario and 143 harvests during 2 scenario (62 full + 57 incremental with 10k, + 23 with 100k + 1 with 500k) .

This table contains durations for harvests in tenants with approximate numbers of records 25k, 50k and 100k for these scenarios. And also results of incremental harvests of 10k, 100k and 500k records.

Test #1

2 Full harvests on 62 member tenants. were carried out concurrently with SRS record source. During test 22 tenants were harvested fully without problems. The largest amount of records for these tenants was 53345 (schema - cs00000001_0048). The rest member tenants (40) stopped harvesting with 54700 records in Average and didn't finish successfully. Records were harvested equally all over the tenants. For some reason load generator stopped execution of the test. It was retested in test #2.

Average response times (RTs) for 100 records request /oai/records?verb=ListRecords&apikey=[APIKey]&resumptionToken=[resumptionToken] with 124 concurrent harvests was  22 seconds but with 80 harvests it decreased to 15 seconds.

Duration for 24225 records - 00:59:40,  53345 records - 01:48:40

Test #2

2 Full harvests on 62 member tenants with SRS record source. Harvesting finished successfully.

Average response times (RTs)  for request /oai/records?verb=ListRecords&apikey=[APIKey]&resumptionToken=[resumptionToken]  for harvesting on one tenant - 580 ms, for 62 tenants - 18.3 sec. Harvesting work faster when RTs low and it happen when number of harvests decrease.

Duration for 24225 records - 00:56:11,  53345 records - 01:42:20, 97588 records - 02:36:17

Test #3

2 Full harvests on 62 member tenants with Inventory record source. Finished successfully without any load. No data to harvest.

Test #4

2 Full harvests on 62 member tenants with SRS + Inventory record source. 

RTs for 100 records request with 124 concurrent harvests (at the beginning of the test) - 14,5 sec

Duration for 24225 records - 00:52:29,  53345 records - 01:40:50, 97588 records - 02:30:48

Test #5

1 Full harvest on 62 member tenants and 1 incremental harvest with 10k on 57 tenants, pause 2 minutes and then 1 incremental with 100k on 23 tenants, pause and 1 incremental with 500k records with SRS record source.

As far as harvests with 10k ended in 25 minutes response times improved for the rest of harvests from 14.4 seconds in Average to 7,4 seconds for full and 6.7 seconds for incremental. Closer to the end of test number of harvests decreased from 19 to 8 with following response times 1.32 seconds for full and 1.11  seconds for incremental. In range from 8 to 2 harvests response times 0.704 seconds for full and 0.622  seconds

Test #6

1 Full harvest + 1 incremental 500k concurrently on cs00000001_0042 tenant with 634k instances.

Full harvest duration - 01:26:19. 

Incremental harvest duration - 01:09:46

Test #7

1 Full harvest from the central tenant level trigger full harvests for all tenants sequentially. It was carried out from EBSCO harvester AWS windows machine. 

Duration for 62 tenants: 13 hours 22 minutes. Number of harvested records: 6495617. Additional info about test in Full Harvests Duration Table.

2 full harvests resources utilization

Test #1.  Record source = Source record storage

Service CPU Utilization

Service Memory Utilization

.

RDS CPU Utilization

.

RDS Database Connections

Response Times Table

Test #2.  Record source = Source record storage

Service CPU Utilization

Service Memory Utilization

.

RDS CPU Utilization

DB utilization during harvesting didn't exceed 25% in average. Spikes on the screen were up to 60% connected to DI create jobs with 10k records.

.

RDS Database Connections

Connection number for 124 concurrent harvests do not exceed 705.

Response Times Table

At this table we observe that response times for 124 concurrent harvests start from 20 sec per second.

Throughput Table

In this table active users equal to number of harvests. Here we observe that throughput start to decrease from 10 requests if number of harvests less than 14. At this point response times is 1.25 sec.

Test #3  Record source = Inventory

Harvests has completed. No harvests carried out because of lack of data at this data set. The following 1 sets was used for the harvesting: all.

Data needed to harvest should contain 'source' = 'CONSORTIUM-FOLIO' for shared instances or 'source' = 'FOLIO' for not shared ones in mod_inventory_storage.instance

Test #4.  Record source = Source record storage and inventory

Service CPU Utilization

Service Memory Utilization

.

RDS CPU Utilization

.

RDS Database Connections

Response Times Table

The results are almost the same as in SRS record source test.

Throughput Table

1 full + 1 incremental harvesting resources utilization

Test #5.  Record source = Source record storage

Service CPU Utilization

Service Memory Utilization

.

RDS CPU Utilization

.

RDS Database Connections

Response Times Table

Throughput Table

Database load

Top SQL

Test #6.  Record source = Source record storage

Service CPU Utilization

mod-oai-pmh - 13% for two harvests, 7% for one after first is finished.

Service Memory Utilization

mod-oai-pmh - 26%

.

RDS CPU Utilization

2 harvests - 6%, 1 harvest - 4%

.

RDS Database Connections

During test 1000 connections in Average.

Response Times Table

Throughput Table

Database load

Top SQL

SELECT * FROM cs00000001_0042_mod_oai_pmh.get_instances_with_marc_records inst
 WHERE inst.instance_id > '4fd0d955-b5dc-543e-be5b-bed10a46ce35'::uuid
    AND ( inst.source = 'MARC'
 OR inst.source = 'MARC_SHARED' OR inst.source = 'CONSORTIUM-MARC') ORDER BY instance_id
LIMIT 101;

SELECT * FROM cs00000001_0042_mod_oai_pmh.get_instances_with_marc_records inst
 WHERE inst.instance_id > 'aee9b375-f51a-588a-8afe-b89fefe87da0'::uuid
    AND ( inst.source = 'MARC'
 OR inst.source = 'MARC_SHARED' OR inst.source = 'CONSORTIUM-MARC')     AND inst.instance_updated_date >= cs00000001_0042_mod_inventory_storage.dateOrMin(timestamptz '2022-12-21T00:00:00Z')
    AND inst.instance_updated_date <= cs00000001_0042_mod_inventory_storage.dateOrMax(timestamptz '2024-01-07T00:00:00Z')
ORDER BY instance_id
LIMIT 101;

Test #7.  Record source = Source record storage

Service CPU Utilization

mod-oai-pmh - 10%

Service Memory Utilization

mod-oai-pmh - 28%

.

RDS CPU Utilization

RDS was in range from 4 to 8% in average. The highest spike was 15%.

.

RDS Database Connections

During test 1000 connections in average with spikes to 1400 connections regularly - every 30 minutes.

Full Harvests Duration Table

Database load

Top SQL

Appendix

To start full or incremental harvesting in parallel triggered from member tenant level use APIKEY of member tenant (host name should stay as for central tenant). It was implemented by jMeter script.

Preparing APIKEY for member tenants "s" parameter should be the same as it is in the central tenant.

Infrastructure