PERF-197 - Getting issue details... STATUS

Overview

As was mentioned in Jira - there is huge performance degradation between releases iris - juniper. According to Jira - previously harvesting process of 350K records takes ±2 hour and now on Juniper release it takes 4 hours.

Test flow

Test consist of few calls:

Initial call:

/oai/records?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=[APIKey] - performing only once

Harvesting call:

/oai/records?verb=ListRecords&apikey=[APIKey]&resumptionToken=[resumptionToken] - performing repeatedly, harvesting 100 records each time until there is no more data in [tenant]_mod_oai_pmh.instances table to harvest.

[resumptionToken] returning in initial call response and in each harvesting call until there is no more records to harvest. When all data has being harvested - resumptionToken will not return with the response.

Environments to test on:

ICP1 - Iris release;
JCP1 - Juniper release;
IMTC1 - Latest Juniper release (and more powerful environment).

Software versions

IMTC1 (Juniper)	JCP1 (Juniper)	ICP1 (Iris)
mod-oai-pmh:3.5.0	mod-oai-pmh:3.5.0	mod-oai-pmh:3.4.2
edge-oai-pmh:2.3.1	edge-oai-pmh:2.3.1	edge-oai-pmh:2.3.1
mod-source-record-storage:5.1.5	mod-source-record-storage:5.1.6	mod-source-record-storage:5.0.6
mod-source-record-manager:3.1.3	mod-source-record-manager:3.1.3	mod-source-record-manager:3.0.9
mod-inventory-storage:21.0.3	mod-inventory-storage:21.0.3	mod-inventory-storage:20.2.2
okapi:4.8.2	okapi:4.8.2	okapi:4.7.3

Test results

Harvesting of 350K records.

	ICP1 (Iris)	IMTC1 (Juniper)	JCP1 (Juniper)
Test 1	1 hr 47 min	1 hr 14 min	45 min
Test 2	1 hr 26 min	1 hr 10 min	42 min

Counts	ICP1 (Iris)	IMTC1 (Juniper)	JCP1 (Juniper)
Instances	8,072,048	756,417	7,252,127
SRS records	1,186,613	597,223	710,371

Conclusions:

As you can see from table above - we can say that Juniper release at least not slower than Iris release, or even faster.
We can see slowness on IMTC1 env. However it's still better or the same as ICP1 (Iris).
The only difference between IMTC1 and JCP1 is state of DB (different data set may affect the performance). Between IMTC1 and JCP1, the difference here is the proportion of SRS records relative to the instance records. IMTC1's harvesting time may be slower because there are more SRS records to process per instances than with JCP1 instances.
We'll need additional tests on same data set but on different releases.

Resource usage

We observe different CPU usage on mod-inventory storage between JCP and IMTC. I think that i can be explained with different numbers of instances that have SRS records.

IMTC

JCP1

ICP1