Overview

The purpose of the OAI-PMH Concurrent Incremental Harvesting tests is to measure performance of Poppy release and to find possible issues, bottlenecks PERF-786 - Getting issue details... STATUS on OCP3 environment.
The previous results of Incremental OAI-PMH PERF-660 - Getting issue details... STATUS

Summary

OAI-PMH - Incremental Harvesting:
- Three tests have been executed by JMeter script to check performance of harvesting the following number of records 10K, 25K, 50K, 500K and 1 MLN with different OAI-PMH Behaviors :
  - Test 1. Record source set to Source record storage ;
  - Test 2. Record source set to Inventory* (data set limit in OCP3 - 250k) ;
  - Test 3. Record source set to Source record storage and inventory.
- Number of multiple concurrent harvests:
  - 2 harvests;
  - 4 harvests;
  - 6 harvests.

CPU utilization during all tests was relevant to number of concurrent harvests.
- Test #1 mod-oai-pmh-b: 2 harvests - 5%, 4 harvests - 10%, 6 harvests - 15%
- Test #2 mod-oai-pmh-b: 2 harvests - 1%, 4 harvests - 3.7%, 6 harvests - 5.5%
- Test #3 mod-oai-pmh-b: 2 harvests - 10%, 4 harvests - 15%, 6 harvests - 25%
Memory consumption was stable except of mod-inventory which grew slowly and mod-oai-pmh that grew up from 46% to 56%. Tests:
- Tests #1 and #3 mod-oai-pmh-b didn't exceed 40%
- Test #2 mod-oai-pmh-b achieved 55%
RDS CPU utilization:
- The averages CPU usage for 2 harvests - 15%
- The averages CPU usage for 4 harvests - 20%
- The averages CPU usage for 6 harvests - 25%

Durations of harvests differed significantly in tests #1,3 (SRS) and test #2 (Inventory) because of the date creation distribution fromDate and untilDate parameters.
Durations were not degraded by increased number of concurrent harvests.
Response times for tests can be found in expanded links in section Test #. Record source

Improvements that can be noted in Poppy release:
1) Non-ECS environment with Poppy release can handle concurrent OAI-PMH

Recommendations & Jiras

To prepare tests it's good point to populate complete_updated_date column in {tenant}_mod_inventory_storage.instance using migration. More info in Appendix section.
To avoid degradation on OAI-PMH response times check that DB top queries do not have DELETE and INSERT for marc_id values after cluster restart
To have the same starting conditions before running test with different Record source sets the edge-oai-pmh service was restarted, it was done to return the service memory usage to its starting(after deployment) value;

Test Runs & Results

Incremental harvesting

Number of harvested records	Test 1. Record source = Source record storage Duration	Test 2. Record source = Inventory Duration	Test 3. Record source = Source record storage and inventory Duration	Test 1. Record source = Source record storage Duration	Test 2. Record source = Inventory Duration	Test 3. Record source = Source record storage and inventory Duration	Test 1. Record source = Source record storage Duration	Test 2. Record source = Inventory Duration	Test 3. Record source = Source record storage and inventory Duration
	2 concurrent Incremental OAI-PMH			4 concurrent Incremental OAI-PMH			6 concurrent Incremental OAI-PMH
10000 records(10K)	00:02:08	00:08:55	00:01:39	00:01:05	00:01:46	00:01:31	00:01:07	00:01:32	00:01:14
25000 records(25K)	00:04:09	00:16:25	00:04:27	00:02:38	00:21:00	00:04:34	00:02:52	00:20:32	00:02:57
50000 records(50K)	00:07:40	00:33:25	00:08:10	00:05:17	00:32:46	00:07:44	00:05:34	00:32:47	00:13:25
500000 records(500K) / 250000 records(250K) in test #2	01:56:40	02:33:30	01:51:24	01:58:34	02:35:29	01:48:48	01:34:29	02:37:45	01:44:42
1000000 records(1MLN)	02:50:17	not enough data	02:39:09	02:59:09	not enough data	02:50:29	03:04:30	not enough data	02:58:50

Incremental harvesting

Test 1. Record source = Source record storage

Results for Test 1. Record source = Source record storage

Test Label	Number of harvested records	Average Response Times, ms	Duration
SRS 2 concurrent 10k	10000	0.982	00:02:08
SRS 4 concurrent 10k	10000	0.356	00:01:05
SRS 6 concurrent 10k	10000	0.37	00:01:07
SRS 2 concurrent 25k	25000	0.689	00:04:09
SRS 4 concurrent 25k	25000	0.331	00:02:38
SRS 6 concurrent 25k	25000	0.385	00:02:52
SRS 2 concurrent 50k	50000	0.616	00:07:40
SRS 4 concurrent 50k	50000	0.334	00:05:17
SRS 6 concurrent 50k	50000	0.364	00:05:34
SRS 2 concurrent 500k	500000	0.903	01:56:40
SRS 4 concurrent 500k	500000	1.12	01:58:34
SRS 6 concurrent 500k	500000	0.829	01:34:29
SRS 2 concurrent 1Mln	1000000	0.718	02:50:17
SRS 4 concurrent 1Mln	1000000	0.77	02:59:09
SRS 6 concurrent 1Mln	1000000	0.802	03:04:30

This graph shows response times for GET request that retrieve data. For some reason for 4 and 6 concurrent harvests with 10k, 25k and 50k it decreases significantly affecting positively duration.

Service CPU Utilization

During five harvesting tests with 10K, 25k, 50K, 500K and 1MLN records CPU utilization remained steady for the same number of concurrent harvests.

The averages CPU usage for 2 harvests mod-oai-pmh-b = 5%, edge-oai-pmh-b = 3.5%, mod-source-record-storage-b = 2%, okapi-b = 1.5%, mod-inventory-storage-b = 1.5% .

The averages CPU usage for 4 harvests mod-oai-pmh-b = 9%, edge-oai-pmh-b = 5.4%, mod-source-record-storage-b = 1.5%, okapi-b = 1.7%, mod-inventory-storage-b = 0.7% .

The averages CPU usage for 6 harvests mod-oai-pmh-b = 15.5%, edge-oai-pmh-b = 9%, mod-source-record-storage-b = 1.5%, okapi-b = 2.4%, mod-inventory-storage-b = 1% .

A few minor fluctuations were at the the beginning of each test.

Service Memory Consumption

Memory consumption was stable.

The averages memory consumption didn't exceed mod-oai-pmh-b = 40%, edge-oai-pmh-b = 31%, mod-source-record-storage-b = 37%, okapi-b = 37%, mod-inventory-storage-b = 14% .

This graph for 10k, 25k, 50k records

This graph for 500k and 1 MLN records

This graph for 1 MLN records only

RDS CPU Utilization

Average CPU utilization was stable for the same number of concurrent harvests.

The averages CPU usage for 2 harvests - 15%

The averages CPU usage for 4 harvests - 20%

The averages CPU usage for 6 harvests - 25-30%

RDS Database Connections

Number of database connection was about 440,

Database load

This graph shows top sql queries for OAI-PMH 10k, 25k, 50k

This graph shows top sql queries for OAI-PMH 500k, 1 MLN

Marked query runs after cluster start until 16:30 UTC. This query was found in pcp1 cluster also.

This graph for 1 MLN only. 4 and 6 concurrent harvests

Test 2. Record source = Inventory

Service CPU Utilization

The averages CPU usage for 2 harvests mod-oai-pmh-b = 1%, edge-oai-pmh-b = 0.5%, mod-source-record-storage-b = 1.5%, okapi-b = 0.8%, mod-inventory-storage-b = 0.3% .

The averages CPU usage for 4 harvests mod-oai-pmh-b = 3.7%, edge-oai-pmh-b = 1.5%, mod-source-record-storage-b = 1.6%, okapi-b = 1.2%, mod-inventory-storage-b = 0.4% .

The averages CPU usage for 6 harvests mod-oai-pmh-b = 5.5%, edge-oai-pmh-b = 2%, mod-source-record-storage-b = 1.4%, okapi-b = 1.2%, mod-inventory-storage-b = 0.5% .

This graph for 10k, 25k, 50k.

This graph for 250k

Service Memory Consumption

For 10k, 25k, 50k memory consumption for mod-oai-pmh was 28% at the beginning and grew up to 46%

For 250k tests memory consumption for mod-oai-pmh was 55% at the beginning of 250k tests and stayed at this level

This graph for 10k, 25k, 50k.

This graph for 250k

RDS CPU Utilization

RDS for 10k, 25k, 50k

Fluctuations on the screen explained by DELETE, INSERT queries with marc_id values connected to daily cluster restart. After 14:30 this process was finished and we observe graph for the tests

RDS for 250k

RDS Database Connections

Connections are the same as for other tests - 440.

Test 3. Record source = Source record storage and inventory

Results for Test 3. Record source = Source record storage and Inventory

Test Label	Average Response Times, ms	Duration
SRS+INV 2 concurrent 10k	0.71	00:01:39
SRS+INV 4 concurrent 10k	0.617	00:01:31
SRS+INV 6 concurrent 10k	0.439	00:01:14
SRS+INV 2 concurrent 25k	0.773	00:04:27
SRS+INV 4 concurrent 25k	0.802	00:04:34
SRS+INV 6 concurrent 25k	0.407	00:02:57
SRS+INV 2 concurrent 50k	0.684	00:08:10
SRS+INV 4 concurrent 50k	0.629	00:07:44
SRS+INV 6 concurrent 50k	1.31	00:13:25
SRS+INV 2 concurrent 500k	1.03	01:51:24
SRS+INV 4 concurrent 500k	1	01:48:48
SRS+INV 6 concurrent 500k	0.953	01:44:42
SRS+INV 2 concurrent 1Mln	0.652	02:39:09
SRS+INV 4 concurrent 1Mln	0.721	02:50:29
SRS+INV 6 concurrent 1Mln	0.768	02:58:50

Service CPU Utilization

The averages CPU usage for 2 harvests mod-oai-pmh-b = 10%, edge-oai-pmh-b = 7%, mod-source-record-storage-b = 1.7%, okapi-b = 1.5%, mod-inventory-storage-b = 0.6% .

The averages CPU usage for 4 harvests mod-oai-pmh-b = 15%, edge-oai-pmh-b = 10%, mod-source-record-storage-b = 1.5%, okapi-b = 2%, mod-inventory-storage-b = 0.8% .

The averages CPU usage for 6 harvests mod-oai-pmh-b = 25%, edge-oai-pmh-b = 15%, mod-source-record-storage-b = 1.4%, okapi-b = 2.4%, mod-inventory-storage-b = 1% .

The graph shows 10k, 25k, 50k, and 2 harvests of 500k

The graph demonstrate 500k and 1 MLN harvests

The graph demonstrate 1 MLN harvests only

Service Memory Utilization

Memory consumption was stable from OAI-PMH related modules. Mod-inventory didn't exceed 72%.

The averages memory consumption didn't exceed mod-oai-pmh-b = 40%, edge-oai-pmh-b = 29%, mod-source-record-storage-b = 37%, okapi-b = 37%, mod-inventory-storage-b = 15% , mod-inventory = 72%

RDS CPU Utilization

Average CPU utilization was stable for the same number of concurrent harvests, close to results in test #1..

Fluctuations on DB graphs explained that after everyday cluster start we observed DELETE queries from marc_indexers table with specific condition. Producing high load which affect response times of OAI-PMH. It happens each time after cluster restart.

It deletes rows from the table marc_indexers based on certain conditions defined in two separate subqueries.
It captures the marc_id values of the deleted rows
It inserts the distinct marc_id values from both subqueries into the table marc_indexers_deleted_ids to keep track of the deleted marc_id values.

The averages CPU usage for 2 harvests - 15%

The averages CPU usage for 4 harvests - 20%

The averages CPU usage for 6 harvests - 25-30%

RDS Database Connections

Number of database connection was about 440 in all tests.

Database load

This graph shows 10k, 25k, 50k

Top query:

WITH deleted_rows AS ( delete from marc_indexers mi where exists( select 1 from marc_records_tracking mrt where mrt.is_dirty = true and mrt.marc_id = mi.marc_id and mrt.version > mi.version ) returning mi.marc_id), deleted_rows2 AS ( delete from marc_indexers mi where exists( select 1 from records_lb where records_lb.id = mi.marc_id and records_lb.state = 'OLD' ) returning mi.marc_id) INSERT INTO marc_indexers_deleted_ids SELECT DISTINCT marc_id FROM deleted_rows UNION SELECT marc_id FROM deleted_rows2

Appendix

Methodology/Approach

OAI-PMH (incremental harvesting) was carried out by JMeter script from carrier with 2 main requests:

/oai/records?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=[APIKey]
/oai/records?verb=ListRecords&apikey=[APIKey]&resumptionToken=[resumptionToken]

to extract the required number of records was used loop counter with following configuration:

98 loop counts for 10K records;
248 loop counts for 25K records;
498 loop counts for 50K records;
2498 loop counts for 250k records*
4998 loop counts for 500K records;
9998 loop counts for 1MLN records

* - Test #2 data set limit

To run the incremental harvesting test the next time ranges were defined by experimental means. The time range for Test 2* was extended due to the impossibility of harvesting the defined number of records, but the next tests were run after adding 800K instances to database.

	Start date	Until date
Test 1.	2022-12-21	2023-10-16
Test 2*.	1962-12-21	2023-10-23*
Test 3.	2022-12-21	2023-10-16

OAI-PMH

Before testing OAI-PMH, following database commands to optimize the tables were executed (from https://folio-org.atlassian.net/wiki/display/FOLIOtips/OAI-PMH+Best+Practices#OAIPMHBestPractices-SlowPerformance):

REINDEX index <tenant>_mod_inventory_storage.audit_item_pmh_createddate_idx ;
REINDEX index <tenant>_mod_inventory_storage.audit_holdings_record_pmh_createddate_idx;
REINDEX index <tenant>_mod_inventory_storage.holdings_record_pmh_metadata_updateddate_idx;
REINDEX index <tenant>_mod_inventory_storage.item_pmh_metadata_updateddate_idx;
REINDEX index <tenant>_mod_inventory_storage.instance_pmh_metadata_updateddate_idx;
analyze verbose <tenant>_mod_inventory_storage.instance;
analyze verbose <tenant>_mod_inventory_storage.item;
analyze verbose <tenant>_mod_inventory_storage.holdings_record;

Execute the following query in a related database for removing existed 'instances' created by previous harvesting request and a request itself:

TRUNCATE TABLE fs09000000_mod_oai_pmh.request_metadata_lb cascade

Execute migration for complete_updated_date column as described here Migration scripts for OAI-PMH (note that in step 2. Update command set search_path = "{tenant}_mod_inventory_storage", "public"; may not work for some reason). It's ok to skip the command in scope of OAI-PMH.

Infrastructure

Environment: OCP3
Release: Poppy (2023 R2)

9 m6i.2xlarge EC2 instances located in US East (N. Virginia)
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK tenant
- 4 brokers
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- og.retention.minutes=480
- default.replication.factor=3

Modules

All modules

OAI-PMH related modules:

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
ocp3-pvt
Fri Feb 16 10:23:52 UTC 2024
mod-remote-storage	15	mod-remote-storage:3.0.1	2	4920	4472	1024	3960	512	512	FALSE
mod-ncip	9	mod-ncip:1.14.4	2	1024	896	128	768	88	128	FALSE
mod-finance-storage	10	mod-finance-storage:8.5.0	2	1024	896	1024	700	88	128	FALSE
mod-agreements	10	mod-agreements:6.0.2	2	1592	1488	128	0	0	0	FALSE
mod-ebsconet	9	mod-ebsconet:2.1.1	2	1248	1024	128	700	128	256	FALSE
edge-sip2	9	edge-sip2:3.1.1	2	1024	896	128	768	88	128	FALSE
mod-organizations	10	mod-organizations:1.8.0	2	1024	896	128	700	88	128	FALSE
mod-settings	9	mod-settings:1.0.2	2	1024	896	200	768	88	128	FALSE
edge-dematic	9	edge-dematic:2.1.1	1	1024	896	128	768	88	128	FALSE
mod-data-import	43	mod-data-import:3.0.7	1	2048	1844	256	1292	384	512	FALSE
mod-search	36	mod-search:3.0.5	2	2592	2480	2048	1440	512	1024	FALSE
mod-tags	10	mod-tags:2.1.0	2	1024	896	128	768	88	128	FALSE
mod-authtoken	15	mod-authtoken:2.14.1	2	1440	1152	512	922	88	128	FALSE
edge-courses	2	edge-courses:1.3.0	2	1024	896	128	768	88	128	FALSE
mod-notify	9	mod-notify:3.1.0	2	1024	896	128	768	88	128	FALSE
mod-inventory-update	10	mod-inventory-update:3.2.1	2	1024	896	128	768	88	128	FALSE
mod-configuration	10	mod-configuration:5.9.2	2	1024	896	128	768	88	128	FALSE
mod-orders-storage	10	mod-orders-storage:13.6.0	2	1024	896	512	700	88	128	FALSE
edge-caiasoft	9	edge-caiasoft:2.1.0	2	1024	896	128	768	88	128	FALSE
mod-login-saml	9	mod-login-saml:2.7.2	2	1024	896	128	768	88	128	FALSE
mod-erm-usage-harvester	10	mod-erm-usage-harvester:4.4.1	2	1024	896	128	768	88	128	FALSE
mod-licenses	10	mod-licenses:5.0.2	2	2480	2312	128	1792	384	512	FALSE
mod-password-validator	10	mod-password-validator:3.1.0	2	1440	1298	128	768	384	512	FALSE
mod-gobi	9	mod-gobi:2.7.1	2	1024	896	128	700	88	128	FALSE
mod-bulk-operations	9	mod-bulk-operations:1.1.9	2	3072	2600	1024	1536	384	512	FALSE
mod-fqm-manager	12	mod-fqm-manager:1.0.3	2	3000	2600	128	2048	384	512	FALSE
mod-graphql	11	mod-graphql:1.12.0	2	1024	896	128	768	88	128	FALSE
mod-finance	9	mod-finance:4.8.0	2	1024	896	128	700	88	128	FALSE
mod-erm-usage	10	mod-erm-usage:4.6.0	2	1024	896	128	768	88	128	FALSE
mod-lists	13	mod-lists:1.0.5	2	3000	2600	128	2048	384	512	FALSE
mod-copycat	9	mod-copycat:1.5.0	2	1024	896	128	768	88	128	FALSE
mod-entities-links	10	mod-entities-links:2.0.4	2	2592	2480	400	1440	0	1024	FALSE
mod-permissions	23	mod-permissions:6.4.0	2	1684	1544	512	1024	384	512	FALSE
pub-edge	8	pub-edge:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-orders	9	mod-orders:12.7.1	2	2048	1440	1024	1024	384	512	FALSE
edge-patron	9	edge-patron:5.0.0	2	1024	896	256	768	88	128	FALSE
edge-ncip	9	edge-ncip:1.9.2	2	1024	896	128	768	88	128	FALSE
mod-users-bl	9	mod-users-bl:7.6.0	2	1440	1152	512	922	88	128	FALSE
mod-invoice	9	mod-invoice:5.7.2	2	1440	1152	512	922	88	128	FALSE
mod-inventory-storage	14	mod-inventory-storage:27.0.4	2	4096	3690	2048	3076	384	512	FALSE
mod-user-import	10	mod-user-import:3.8.0	2	1024	896	128	768	88	128	FALSE
mod-sender	9	mod-sender:1.11.0	2	1024	896	128	768	88	128	FALSE
edge-oai-pmh	9	edge-oai-pmh:2.7.2	2	1512	1360	1024	1440	384	512	FALSE
mod-data-export-worker	9	mod-data-export-worker:3.1.2	2	3072	2800	1024	2048	384	512	FALSE
mod-rtac	20	mod-rtac:3.5.0	2	1024	896	128	768	88	128	FALSE
mod-circulation-storage	18	mod-circulation-storage:17.1.7	2	2880	2592	1536	1814	384	512	FALSE
mod-source-record-storage	17	mod-source-record-storage:5.7.5	2	5600	5000	2048	3500	384	512	FALSE
mod-calendar	9	mod-calendar:2.5.0	2	1024	896	128	768	88	128	FALSE
mod-event-config	10	mod-event-config:2.6.0	2	1024	896	128	768	88	128	FALSE
mod-courses	9	mod-courses:1.4.8	2	1024	896	128	768	88	128	FALSE
mod-inventory	18	mod-inventory:20.1.7	2	2880	2592	1024	1814	384	512	FALSE
mod-email	9	mod-email:1.16.0	2	1024	896	128	768	88	128	FALSE
mod-circulation	12	mod-circulation:24.0.11	2	2880	2592	1536	1814	384	512	FALSE
mod-di-converter-storage	11	mod-di-converter-storage:2.1.5	2	1024	896	128	768	88	128	FALSE
mod-pubsub	13	mod-pubsub:2.11.3	2	1536	1440	1024	922	384	512	FALSE
edge-orders	9	edge-orders:2.9.1	2	1024	896	128	768	88	128	FALSE
edge-rtac	10	edge-rtac:2.6.2	2	1024	896	128	768	88	128	FALSE
mod-template-engine	9	mod-template-engine:1.19.1	2	1024	896	128	768	88	128	FALSE
mod-users	12	mod-users:19.2.2	2	1024	896	128	768	88	128	FALSE
mod-patron-blocks	11	mod-patron-blocks:1.9.0	2	1024	896	1024	768	88	128	FALSE
edge-fqm	12	edge-fqm:1.0.1	2	1024	896	128	768	88	128	FALSE
mod-audit	9	mod-audit:2.8.0	2	1024	896	128	768	88	128	FALSE
mod-source-record-manager	19	mod-source-record-manager:3.7.8	2	5600	5000	2048	3500	384	512	FALSE
nginx-edge	8	nginx-edge:2023.06.14	2	1024	896	128	0	0	0	FALSE
mod-quick-marc	9	mod-quick-marc:5.0.1	1	2288	2176	128	1664	384	512	FALSE
nginx-okapi	8	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	FALSE
okapi-b	9	okapi:5.1.1	3	1684	1440	1024	922	384	512	FALSE
mod-feesfines	10	mod-feesfines:19.0.0	2	1024	896	128	768	88	128	FALSE
mod-invoice-storage	10	mod-invoice-storage:5.7.0	2	1872	1536	1024	1024	384	512	FALSE
mod-service-interaction	10	mod-service-interaction:3.0.2	2	2048	1844	256	1290	384	512	FALSE
mod-patron	9	mod-patron:6.0.0	2	1024	896	128	768	88	128	FALSE
mod-data-export	11	mod-data-export:4.8.7	1	1024	896	1024	768	88	128	FALSE
mod-oai-pmh	11	mod-oai-pmh:3.12.8	2	4096	3690	2048	3076	384	512	FALSE
edge-connexion	9	edge-connexion:1.1.1	2	1024	896	128	768	88	128	FALSE
mod-notes	9	mod-notes:5.1.0	2	1024	896	128	952	384	512	FALSE
mod-kb-ebsco-java	9	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128	FALSE
mod-data-export-spring	14	mod-data-export-spring:3.0.2	1	2048	1844	256	1536	384	512	FALSE
mod-login	10	mod-login:7.10.1	2	1440	1298	1024	768	384	512	FALSE
mod-organizations-storage	10	mod-organizations-storage:4.6.0	2	1024	896	128	700	88	128	FALSE
pub-okapi	8	pub-okapi:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-eusage-reports	9	mod-eusage-reports:2.0.0	2	1024	896	128	768	88	128	FALSE

Browser not supported