Overview

The purpose of the concurrent OAI-PMH, data import and CI/CO tests is to determine how these workflows affect each other. The report contain results for the PERF-492 - Getting issue details... STATUS

Summary

The OAI-PMH has an influence on CI/CO response times - it worsen results up to 13%. DI worsens CI/CO results mostly with create job profile up to 55% with 1.000 records and up to 71% with 100.000 records. DI with update profile worsens CI/CO results less than create profile - up to 25% with 1.000 records and 36% with 100.000 records. OAI-PMH (incremental) duration was in range from 04:20 min till 05:20 min for all tests of DI with 1.000 records and without it. OAI-PMH duration calculation is described in Methodology/Approach section.
For major services memory usage didn't exceed the level of 60%. The highest level was registered for mod-source-record-manager 107% and mod-inventory-b 98%. After tests for Scenario 1 it achieved its stable level and didn't change.
Running OAI-PMH, DI and CI/CO simultaneously it has been shown that the environment can handle such load.
CI/CO response times during DI and OAI-PMH has degradation depending on the profile that was used. It was a number of consequent DI operations (create and update job profiles).
After 90 minutes of full harvest the growth of CPU utilization up to 188 % was observed for mod-oai-pmh-b. This increased CPU utilization lasted during 10 minutes. After it got back to steady state ( 5-7 % ).
Service CPU Utilization at the beginning of DI mostly used by mod-di-converter-storage-b ( 253 % ), mod-inventory-b ( 172 % ), mod-quick-marc-b ( 108 % ). For the rest of modules it was under 70%. At the highest level it was mod-di-converter-storage-b ( 453 % ), mod-inventory-b ( 190 % ), mod-quick-marc-b ( 121 % ).
RDS CPU Utilization during incremental harvesting didn't exceed 60 % for all DI job profiles (1.000 records). Data export took 40% But for full harvesting with DI Create job profile (100.000 records) it became instantly 96 % and stayed on this level major part of process. DI Update used up to 90%.
All oai-pmh tests were executed by EBSCO Harvester in the AWS ptf-windows instance.
During full harvesting (504) Gateway Timeout issue happened after all DI create and update were done so it didn't affect the results. It happened during all two Full harvesting runs with returned instances count ( during first OAI-PMH full - 1764989 records, second - 1166089 out of total 10433728 ).

Recommendations & Jiras

Allocate more CPU resources to mod-di-convertor-storage and mod-inventory-b

Test Runs & Results

Data import duration and CI/CO response times with DI & OAI-PMH results

Test #	CI/CO 10 users	Scenario	Job profile	OAI-PMH only / instance amount	OAI-PMH + DI + CI/CO Duration	DI Duration	CI average	CO average	Load level	Comments
Scenario 1 OAI-PMH incremental	40 min	DI MARC Bib Create	PTF - Create 2	00:04:46 8000	00:05:18	00:00:48	0.961	1.398	For scenario 1 1K (with pause ~5 min)	All incremental harvests were stopped manually after ~ 8000 instances
		DI MARC Bib Update	PTF - Updates Success - 1	00:05:14 8000	00:05:18	00:00:56	0.706	1.125
		DI MARC Bib Create	PTF - Create 2	00:05:11 8000	00:04:20	00:00:43	0.843	1.402
		DI MARC Bib Update	PTF - Updates Success - 1	00:04:24 8000	00:04:20	00:00:44	0.848	1.335
Scenario 2 OAI-PMH full mode	5 hours	DI MARC Bib Create	PTF - Create 2	1764989	04:42:20	00:53:30	1.078	1.545	For scenario 2 100K (with pause ~5 min)	During scenario 2 full harvests stopped due to ERROR: Error saving an xml document: The remote server returned an error: (504) Gateway Timeout.
		DI MARC Bib Update	PTF - Updates Success - 1			01:04:38	0.725	1.231
		DI MARC Bib Update	PTF - Updates Success - 1			01:05:48	0.69	1.249
	5 hours	DI MARC Bib Update	PTF - Updates Success - 1	1166089	03:44:20	01:17:58	0.903	1.333
		DI MARC Bib Update	PTF - Updates Success - 1			01:18:08	0.737	1.221
		DI MARC Bib Update	PTF - Updates Success - 1			01:21:21	0.62	1.106		Last 30 minutes without OAI-PMH

Comparisons

Comparison table for CI/CO response times

	CI/CO only	CI/CO + OAI-PMH	CI/CO + OAI-PMH + DI Create 1k	CI/CO + OAI-PMH + DI Update 1k	CI/CO after	CI/CO + OAI-PMH + DI Create 100k	CI/CO between	CI/CO + OAI-PMH + DI Update 100k	CI/CO after
Requests	Average	Average	Average	Average		Average		Average
Check-Out Controller	0.904	1.024 ↑13.27%	1.398 ↑54.65%	1.125 ↑24.45%	0.900	1.545 ↑70.91%	0.914	1.231 ↑36.17%	0.926
Check-In Controller	0.629	0.666 ↑5.88%	0.961 ↑52.78%	0.706 ↑12.24%	0.625	1.078 ↑71.38%	0.569	0.725 ↑15.26%	0.515

Scenario 1

Response time

This table shows s40 minutes of CI/CO

Service CPU Utilization

Service Memory Utilization

RDS CPU Utilization

Scenario 2

Response time

The table shows first 5 hours of CI/CO (it contains Create and 2 Updates with 100.000 records file)

The table shows second 5 hours of CI/CO (it contains 3 Updates with 100.000 records file)

Service CPU Utilization

Service Memory Utilization

RDS CPU Utilization

Errors

Scenario 1 - no errors

Scenario 2

All errors are connected to

Check-Out Controller

Request name	Number
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)_POST_422	8
GET_inventory/items (Submit_barcode_checkout)_GET_200	6
GET_groups_ID (Submit_patron_barcode)_GET_400	1

Appendix

Methodology/Approach

OAI-PMH (incremental) was carried out with manual stop from AWS instance machine after approximately 8000 instances and holdings were harvested up. To define time duration for the certain harvest just find difference between timestamps of second call and the last one in the definite log file in log folder.

Circulation rules should be modified before CI/CO test in Circulation rules editor to run it without issues from POST_circulation/check-out-by-barcode (Submit_barcode_checkout) side.

Partitions number should be equal to 2 in all DI related topics.

Before running OAI-PMH with full harvest, following database commands to optimize the tables should be executed (from https://folio-org.atlassian.net/wiki/display/FOLIOtips/OAI-PMH+Best+Practices#OAIPMHBestPractices-SlowPerformance):

REINDEX index <tenant>_mod_inventory_storage.audit_item_pmh_createddate_idx ;
REINDEX index <tenant>_mod_inventory_storage.audit_holdings_record_pmh_createddate_idx;
REINDEX index <tenant>_mod_inventory_storage.holdings_record_pmh_metadata_updateddate_idx;
REINDEX index <tenant>_mod_inventory_storage.item_pmh_metadata_updateddate_idx;
REINDEX index <tenant>_mod_inventory_storage.instance_pmh_metadata_updateddate_idx;
analyze verbose <tenant>_mod_inventory_storage.instance;
analyze verbose <tenant>_mod_inventory_storage.item;
analyze verbose <tenant>_mod_inventory_storage.holdings_record;

Execute the following query in a related database for removing existed 'instances' created by previous harvesting request and a request itself:

TRUNCATE TABLE fs09000000_mod_oai_pmh.request_metadata_lb cascade

Infrastructure

8 m6i.2xlarge EC2 instances located in US East (N. Virginia)
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK ptf-kakfa-3
- 4 brokers
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- og.retention.minutes=480
- default.replication.factor=3
Front End:
- Item Check-in (folio_checkin-8.0.100000491)
- Item Check-out (folio_checkout-9.0.100000595)

Modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
ocp2-pvt
Mon Jul 03 14:54:13 UTC 2023
mod-inventory-storage	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:26.0.0	2	2208	1952	1024	1440	384	512	FALSE
mod-inventory	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.0.0-SNAPSHOT.392	2	2880	2592	1024	1814	384	512	FALSE
mod-source-record-storage	5	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.6.5	2	5600	5000	2048	3600	384	512	FALSE
mod-source-record-manager	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.6.0-SNAPSHOT.197	2	4096	3688	1024	2048	384	512	FALSE
mod-data-import	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:2.7.0-SNAPSHOT.101	1	2048	1844	256	1292	384	512	FALSE
mod-di-converter-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.1.0-SNAPSHOT.32	2	1024	896	128	768	88	128	FALSE
mod-data-import-converter-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import-converter-storage:1.16.0-SNAPSHOT.132	2	1024	896	128	768	88	128	FALSE
mod-remote-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:2.0.0-SNAPSHOT.83	2	4920	4472	1024	3960	512	512	FALSE
mod-users	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.2.0-SNAPSHOT.584	2	1024	896	128	768	88	128	FALSE
mod-configuration	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.9.2-SNAPSHOT.291	2	1024	896	128	768	88	128	FALSE
mod-circulation-storage	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:16.1.0-SNAPSHOT.305	2	1536	1440	1024	896	384	512	FALSE
mod-circulation	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:23.5.0-SNAPSHOT.556	2	1024	896	1024	768	88	128	FALSE
mod-authtoken	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.14.0-SNAPSHOT.238	2	1440	1152	512	922	88	128	FALSE
mod-pubsub	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.10.0-SNAPSHOT.124	2	1536	1440	1024	922	384	512	FALSE
pub-okapi	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2022.03.02	2	1024	896	128	768	0	0	FALSE
okapi-b	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.1.0-SNAPSHOT.1352	3	1684	1440	1024	922	384	512	FALSE

Partitions

Click here to expand partitions

Name	Partitions	% Preferred	# Under-replicated	Custom Config
ocp2.Default.fs09000000.DI_COMPLETED	2	100%	0	No
ocp2.Default.fs09000000.DI_ERROR	2	100%	0	No
ocp2.Default.fs09000000.DI_INITIALIZATION_STARTED	2	100%	0	No
ocp2.Default.fs09000000.DI_INVENTORY_HOLDING_CREATED	2	100%	0	No
ocp2.Default.fs09000000.DI_INVENTORY_HOLDING_MATCHED	2	100%	0	No
ocp2.Default.fs09000000.DI_INVENTORY_HOLDING_NOT_MATCHED	2	100%	0	No
ocp2.Default.fs09000000.DI_INVENTORY_HOLDING_UPDATED	2	100%	0	No
ocp2.Default.fs09000000.DI_INVENTORY_INSTANCE_CREATED	2	100%	0	No
ocp2.Default.fs09000000.DI_INVENTORY_INSTANCE_CREATED_READY_FOR_POST_PROCESSING	2	100%	0	No
ocp2.Default.fs09000000.DI_INVENTORY_ITEM_MATCHED	2	100%	0	No
ocp2.Default.fs09000000.DI_LOG_SRS_MARC_BIB_RECORD_CREATED	2	100%	0	No
ocp2.Default.fs09000000.DI_MARC_FOR_UPDATE_RECEIVED	2	100%	0	No
ocp2.Default.fs09000000.DI_PARSED_RECORDS_CHUNK_SAVED	2	100%	0	No
ocp2.Default.fs09000000.DI_RAW_RECORDS_CHUNK_PARSED	2	100%	0	No
ocp2.Default.fs09000000.DI_RAW_RECORDS_CHUNK_READ	2	100%	0	No
ocp2.Default.fs09000000.DI_SRS_MARC_BIB_INSTANCE_HRID_SET	2	100%	0	No
ocp2.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_CREATED	2	100%	0	No
ocp2.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_MATCHED	2	100%	0	No
ocp2.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_MATCHED_READY_FOR_POST_PROCESSING	2	100%	0	No
ocp2.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_MODIFIED	2	100%	0	No
ocp2.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_MODIFIED_READY_FOR_POST_PROCESSING	2	100%	0	No
ocp2.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_NOT_MATCHED	2	100%	0	No

Folio Development Teams

OAI-PMH performance dependencies between CI/CO and data import

Overview

Summary

Recommendations & Jiras

Test Runs & Results

Comparisons

Service Memory Utilization

RDS CPU Utilization

Scenario 2

Service Memory Utilization

RDS CPU Utilization

Errors

Appendix

Methodology/Approach

Infrastructure