Overview

The purpose of the OAI-PMH Full Harvesting tests and Incremental Harvesting tests is to measure performance of Poppy release and to find possible issues, bottlenecks per PERF-706 - Getting issue details... STATUS on MCPT environment.
Two jMeter scripts for 2 scenarios were prepared to concurrently execute OAI-PMH by initiating harvesting from the member tenant level. To simulate the behaviour of EBSCO Harvester a delay after each request was used (150-300 ms) in the script.
First script should trigger two full havests on 62 tenants creating 124 concurrent harvests. Second scenario should trigger 1 full and 1 incremental where tenants have 10k, 100k and 500k accordingly. In this second scenario incremental harvests should start from 10k, then 100k and 500k.
In total, 124 harvests for the first scenario and 143 harvests during 2 scenario (62 full + 57 incremental with 10k, + 23 with 100k + 1 with 500k).
Baseline test for 1 Full + 1 incremental with 500k on 1 tenant should be run against tenant with highest number of instances.

Overview PCON

The purpose of the OAI-PMH Full Harvesting tests and Incremental Harvesting tests is to measure performance of Poppy release and to find possible issues, bottlenecks per PERF-706 - Getting issue details... STATUS on PCON environment.
jMeter scripts were prepared to concurrently execute two OAI-PMH harvests on each tenant by initiating harvesting from the member tenant level. To simulate the behavior of EBSCO Harvester a delay after each request was used (200-400 ms) in the script. 2 scripts were triggered from carrier-io concurrently to carry out 1 Full and 1 Incremental harvesting on each member tenant. One was for Full harvesting and second one was for Incremental. In total, 10 harvests were executed simultaneously during the test.

Summary

In mcpt OAI-PMH can concurrently perform 124 harvests - 2 full harvests on each 62 member tenants with constant average throughput 10 requests or 1000 records per second during the test.
Average response times (RTs) for /oai/records?verb=ListRecords&apikey=[APIKey]&resumptionToken=[resumptionToken] request in one tenant test - 0.580 sec,, in 62 tenants test - 18.3 sec. Harvesting work faster when RTs low and it happen when number of harvests decrease. Response times for 2 full harvest on 1 tenant only - 0.540 sec.
When a number of harvests increase from 1 to 14 a total throughput grow up to 10 request per second. If number of harvests higher than 14 the throughput do not change. Just response times grow.
Duration for 2 full harvests on 62 tenants - 4 hours 30 minutes in scenario 1
Duration for 10k incremental harvest - 25 min, 100k - 1 hour 52 minutes, 500k - 3 hours - in scenario 2
If to run 1 full and 1 incremental 500k on 1 tenant we have full harvest duration - 01:26:19. Incremental harvest duration - 01:09:46. Average response time 0.601 second.
Duration of full harvest triggered from central tenant level for all 62 tenants running sequentially - 13 hours 22 minutes.
CPU utilization in mod-oai-pmh didn't exceed 44% over all tests.
Memory utilization in mod-oai-pmh was not higher 49%. Mod-inventory - 61% in scenario 2, 85% in scenario 1 (here it grew to this level because two 10k DI create jobs were tested along the oai-pmh). Additional DI along the test didn't affect oai-pmh response times. No memory leak trends found. More info in resource utilization table.
RDS CPU utilization was 25% for first scenario, not higher 30% at the beginning of test in second scenario.
DB connections - 700 in first scenario with SRS record source, 1400 with SRS + Inventory and 1350 in second scenario with SRS.

Summary PCON

OAI-PMH can concurrently operate for a minimum of 5 member tenants with negligible performance decline. The time taken is directly proportional to the quantity of requests made.
Average response times (ms) for /oai/records?verb=ListRecords&apikey=[APIKey]&resumptionToken=[resumptionToken] was 0.618

Recommendations & Jiras

Test Runs & Results

Two jMeter scripts for 2 scenarios were prepared to concurrently execute OAI-PMH by initiating harvesting from the member tenant level. To simulate the behaviour of EBSCO Harvester a delay after each request was used (150-300 ms) in the script.

In total, 124 harvests were executed concurrently during the test for the first scenario and 143 harvests during 2 scenario (62 full + 57 incremental with 10k, + 23 with 100k + 1 with 500k) .

This table contains durations for harvests in tenants with approximate numbers of records 25k, 50k and 100k for these scenarios. And also results of incremental harvests of 10k, 100k and 500k records.

RESULTS for tests #1 - #5			Record source
			SRS		Inventory		SRS+Inventory
	Full / Incremental	Tenant	Instances	Duration	Instances	Duration	Instances	Duration
Scenario #1 2 Full Harvests	Full	cs00000001_0037	24226	00:56:11		No data	24226	00:52:27
		cs00000001_0048	53346	01:39:24		No data	53346	01:40:50
		cs00000001_0030	97588	02:36:17		No data	97588	02:30:48
		cs00000001_0042	634673	04:28:34		No data	634673	04:21:41

Scenario #2 1 Full Harvest + 1 Incremental	Full	cs00000001_0037	24226	00:49:52		No data
		cs00000001_0048	53346	01:28:33		No data
		cs00000001_0030	97588	02:15:58		No data
		cs00000001_0042	634673	03:41:44		No data
	Incremental	cs00000001_0037	10k	00:25:02		No data
		cs00000001_0046	100k	01:52:01		No data
		cs00000001_0042	500k	02:59:13		No data

Test #1

2 Full harvests on 62 member tenants. were carried out concurrently with SRS record source. During test 22 tenants were harvested fully without problems. The largest amount of records for these tenants was 53345 (schema - cs00000001_0048). The rest member tenants (40) stopped harvesting with 54700 records in Average and didn't finish successfully. Records were harvested equally all over the tenants. For some reason load generator stopped execution of the test. It was retested in test #2.

Average response times (RTs) for 100 records request /oai/records?verb=ListRecords&apikey=[APIKey]&resumptionToken=[resumptionToken] with 124 concurrent harvests was 22 seconds but with 80 harvests it decreased to 15 seconds.

Duration for 24225 records - 00:59:40, 53345 records - 01:48:40

Test #2

2 Full harvests on 62 member tenants with SRS record source. Harvesting finished successfully.

Average response times (RTs) for request /oai/records?verb=ListRecords&apikey=[APIKey]&resumptionToken=[resumptionToken] for harvesting on one tenant - 580 ms, for 62 tenants - 18.3 sec. Harvesting work faster when RTs low and it happen when number of harvests decrease.

Duration for 24225 records - 00:56:11, 53345 records - 01:42:20, 97588 records - 02:36:17

Test #3

2 Full harvests on 62 member tenants with Inventory record source. Finished successfully without any load. No data to harvest.

Test #4

2 Full harvests on 62 member tenants with SRS + Inventory record source.

RTs for 100 records request with 124 concurrent harvests (at the beginning of the test) - 14,5 sec

Duration for 24225 records - 00:52:29, 53345 records - 01:40:50, 97588 records - 02:30:48

Test #5

1 Full harvest on 62 member tenants and 1 incremental harvest with 10k on 57 tenants, pause 2 minutes and then 1 incremental with 100k on 23 tenants, pause and 1 incremental with 500k records with SRS record source.

As far as harvests with 10k ended in 25 minutes response times improved for the rest of harvests from 14.4 seconds in Average to 7,4 seconds for full and 6.7 seconds for incremental. Closer to the end of test number of harvests decreased from 19 to 8 with following response times 1.32 seconds for full and 1.11 seconds for incremental. In range from 8 to 2 harvests response times 0.704 seconds for full and 0.622 seconds

Test #6

1 Full harvest + 1 incremental 500k concurrently on cs00000001_0042 tenant with 634k instances.

Full harvest duration - 01:26:19.

Incremental harvest duration - 01:09:46

API_KEY_FH - FULL API_KEY_IH - INCREMENTAL All test	avg
active users	2.00
/oai/records?verb=ListRecords&apikey=[API_KEY_FH]&resumptionToken=[resumptionToken]	588.09 ms
/oai/records?verb=ListRecords&apikey=[API_KEY_IH]&resumptionToken=[resumptionToken]	608.90 ms
/oai/records?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=[API_KEY_FH]	2.52 s
/oai/records?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=[API_KEY_IH]	5.46 s

Full and Incremental harvesting together	avg
active users	2.00
/oai/records?verb=ListRecords&apikey=[API_KEY_FH]&resumptionToken=[resumptionToken]	601.60 ms
/oai/records?verb=ListRecords&apikey=[API_KEY_IH]&resumptionToken=[resumptionToken]	604.38 ms

Full only (after Incremental finished)	avg
active users	1.00
/oai/records?verb=ListRecords&apikey=[API_KEY_FH]&resumptionToken=[resumptionToken]	508.29 ms

Test #7

1 Full harvest from the central tenant level trigger full harvests for all tenants sequentially. It was carried out from EBSCO harvester AWS windows machine.

Duration for 62 tenants: 13 hours 22 minutes. Number of harvested records: 6495617. Additional info about test in Full Harvests Duration Table.

CPU and Memory utilization for tests #1 - #5

CPU Utilization				Memory Consumption
1 scenario		2 scenario		1 scenario		2 scenario
Module	%	Module	%	Module	%	Module	%
mod-oai-pmh-b	43.87	mod-oai-pmh-b	42.95	mod-inventory-b	84.99	mod-inventory-b	61.23
edge-oai-pmh-b	29.36	edge-oai-pmh-b	28.92	mod-quick-marc-b	63.63	okapi-b	58.17
mod-inventory-b	25.79	mod-inventory-b	23.63	okapi-b	62.01	mod-source-record-manager-b	53.74
nginx-okapi	18.68	nginx-okapi	19.66	mod-source-record-manager-b	57.09	mod-oai-pmh-b	50.54
mod-source-record-storage-b	9.58	mod-quick-marc-b	9.05	mod-oai-pmh-b	48.57	mod-quick-marc-b	49.45
mod-quick-marc-b	8.21	okapi-b	8.32	mod-source-record-storage-b	35.72	mod-source-record-storage-b	47.91
pub-edge	8.15	mod-source-record-storage-b	7.81	edge-oai-pmh-b	33.32	edge-oai-pmh-b	35.10
okapi-b	6.79	pub-edge	7.64	mod-circulation-storage-b	26.79	mod-circulation-storage-b	33.32
mod-source-record-manager-b	3.31	mod-source-record-manager-b	3.14	mod-inventory-storage-b	17.26	mod-inventory-storage-b	13.35
mod-inventory-storage-b	2.71	mod-inventory-storage-b	2.40	pub-edge	4.80	nginx-okapi	4.80
mod-circulation-storage-b	0.55	mod-circulation-storage-b	1.23	nginx-okapi	4.80	pub-edge	4.74
pub-okapi	0.13	pub-okapi	0.14	pub-okapi	4.58	pub-okapi	4.35

RESULTS PCON

jMeter script was prepared to concurrently execute two OAI-PMH harvests by initiating harvesting from the member tenant level. To simulate the behaviour of EBSCO Harvester a delay after each request was used (200-400 ms) in the script.

2 scripts were triggered from carrier-io concurrently to carry out 1 Full and 1 Incremental harvesting on each member tenant. One was for Full harvesting and second one was for Incremental. In total, 10 harvests were executed simultaneously during the test.

ALL SCENARIO TEST RESULTS			Record source
			SRS		Inventory		SRS+Inventory
	Number of records	Tenant	Harvested number	Duration	Harvested number	Duration	Harvested number	Duration
Scenario #1 2 Full Harvests	Full	cs00000int_0001	313465	01:00:25	39711	00:07:54	353176	01:07:17
		cs00000int_0002	170001	00:35:13	30000	00:06:05	200001	00:41:47
		cs00000int_0003	183867	00:40:37	18378	00:04:12	202245	00:43:56
		cs00000int_0004	174793	00:38:41	58232	00:09:41	233025	00:47:40
		cs00000int_0005	362891	01:07:21	46634	00:09:00	409525	01:21:33
Scenario #2 1 Full Harvest + 1 Incremental	Full	cs00000int_0001	313465	01:07:05	39711	00:06:28	353176	01:06:05
		cs00000int_0002	170001	00:35:22	30000	00:05:00	200001	00:38:42
		cs00000int_0003	183867	00:42:50	18378	00:03:20	202245	00:40:59
		cs00000int_0004	174793	00:40:13	58232	00:08:46	233025	00:44:32
		cs00000int_0005	362891	01:14:55	46634	00:07:49	383996	01:10:23
Scenario # 2 1 Full Harvest + 1 Incremental	10k	cs00000int_0001	10k	00:02:06	10k	00:02:00	10k	00:02:11
		cs00000int_0002		00:01:57		00:01:57		00:01:58
		cs00000int_0003		00:02:10		00:01:58		00:02:09
		cs00000int_0004		00:02:05		00:01:44		00:02:00
		cs00000int_0005		00:02:08		00:02:00		00:02:06
	50k	cs00000int_0001	50k	00:10:36	39711	00:06:07	50k	00:10:32
		cs00000int_0002		00:09:50	30000	00:04:42		00:10:08
		cs00000int_0003		00:10:38	18378	00:03:12		00:10:27
		cs00000int_0004		00:10:20	50100	00:06:17		00:10:08
		cs00000int_0005		00:10:43	46634	00:07:00		00:10:36
	500k	cs00000int_0001	313465	00:56:32		unavailable	276298	00:49:17
		cs00000int_0002	170001	00:31:36		unavailable	200001	00:36:19
		cs00000int_0003	183867	00:37:00		unavailable	202245	00:38:31
		cs00000int_0004	174793	00:34:53		unavailable	233025	00:41:38
		cs00000int_0005	362891	01:03:14		unavailable	270297	00:48:40

1 FULL HARVEST SEQUENTIALLY TEST #1
		Tenant	Duration, hh:mm:ss	returned_instances_counter	failed_instances_counter
EBSCO Harvester	SRS	cs00000int_0001	00:34:44	313465	No logs for failed instances
		cs00000int_0002	00:17:22	170001
		cs00000int_0003	00:21:25	183867
		cs00000int_0004	00:19:20	174887
		cs00000int_0005	00:41:58	363361
		Duration (for all jobs):	02:14:49
			Total (full harvest):	1205581

2 full harvests resources utilization

Test #1. Record source = Source record storage

Service CPU Utilization

Service Memory Utilization

RDS CPU Utilization

RDS Database Connections

Response Times Table

Test #2. Record source = Source record storage

Service CPU Utilization

Service Memory Utilization

RDS CPU Utilization

DB utilization during harvesting didn't exceed 25% in average. Spikes on the screen were up to 60% connected to DI create jobs with 10k records.

RDS Database Connections

Connection number for 124 concurrent harvests do not exceed 705.

Response Times Table

At this table we observe that response times for 124 concurrent harvests start from 20 sec per second.

Throughput Table

In this table active users equal to number of harvests. Here we observe that throughput start to decrease from 10 requests if number of harvests less than 14. At this point response times is 1.25 sec.

Test #3 Record source = Inventory

Harvests has completed. No harvests carried out because of lack of data at this data set. The following 1 sets was used for the harvesting: all.

Data needed to harvest should contain 'source' = 'CONSORTIUM-FOLIO' for shared instances or 'source' = 'FOLIO' for not shared ones in mod_inventory_storage.instance

Test #4. Record source = Source record storage and inventory

Service CPU Utilization

Service Memory Utilization

RDS CPU Utilization

RDS Database Connections

Response Times Table

The results are almost the same as in SRS record source test.

Throughput Table

1 full + 1 incremental harvesting resources utilization

Test #5. Record source = Source record storage

Service CPU Utilization

Service Memory Utilization

RDS CPU Utilization

RDS Database Connections

Response Times Table

Throughput Table

Database load

Top SQL

Top SQL comparing

Here we compare queries: one with high average latency and other with low. Difference only in schemas. cs00000001_0042 has 630k and cs00000001_0003 - 136k.

Harvesting for 0003 started with full + incremental 10k. after 10k finished it started a new incremental 100k
Harvesting for 0042 started with full + incremental 10k, after 10k finished it started a new incremental 100k and 500k.

--- Calls/sec - 0.09, Rows/sec - 9.56, Avr. Latency - 237.63 ---

SELECT * FROM cs00000001_0003_mod_oai_pmh.get_instances_with_marc_records inst
WHERE inst.instance_id > 'e74b436b-ac91-5fd1-9790-bc825329ec84'::uuid
AND ( inst.source = 'MARC'
OR inst.source = 'MARC_SHARED' OR inst.source = 'CONSORTIUM-MARC') ORDER BY instance_id
LIMIT 101;

--- Calls/sec - 0.44, Rows/sec - 44.64, Avr. Latency - 55.90 ---
SELECT * FROM cs00000001_0042_mod_oai_pmh.get_instances_with_marc_records inst
WHERE inst.instance_id > 'c8ed4322-7ec9-5f81-b01a-18075b06c12e'::uuid
AND ( inst.source = 'MARC'
OR inst.source = 'MARC_SHARED' OR inst.source = 'CONSORTIUM-MARC') ORDER BY instance_id
LIMIT 101;

Test #6. Record source = Source record storage

Memory and CPU utilization, %

Memory		CPU
mod-inventory-b	58.99	mod-inventory-b	24.12
mod-quick-marc-b	58.51	mod-oai-pmh-b	13.07
okapi-b	55.28	mod-pubsub-b	10.13
mod-source-record-manager-b	46.05	edge-oai-pmh-b	9.11
mod-source-record-storage-b	38.19	mod-quick-marc-b	6.99
mod-circulation-storage-b	33.91	okapi-b	5.85
mod-oai-pmh-b	26.95	mod-source-record-storage-b	3.87
mod-pubsub-b	26.88	mod-source-record-manager-b	3.23
edge-oai-pmh-b	24.90	nginx-okapi	1.78
mod-inventory-storage-b	15.04	mod-circulation-storage-b	1.22
nginx-okapi	4.91	mod-inventory-storage-b	0.93
pub-okapi	4.46	pub-okapi	0.13

Service CPU Utilization

mod-oai-pmh - 13% for two harvests, 7% for one after first is finished.

Service Memory Utilization

mod-oai-pmh - 26%

RDS CPU Utilization

2 harvests - 6%, 1 harvest - 4%

RDS Database Connections

During test 1000 connections in Average.

Response Times Table

Throughput Table

Database load

Top SQL

SELECT * FROM cs00000001_0042_mod_oai_pmh.get_instances_with_marc_records inst
 WHERE inst.instance_id > '4fd0d955-b5dc-543e-be5b-bed10a46ce35'::uuid
    AND ( inst.source = 'MARC'
 OR inst.source = 'MARC_SHARED' OR inst.source = 'CONSORTIUM-MARC') ORDER BY instance_id
LIMIT 101;

SELECT * FROM cs00000001_0042_mod_oai_pmh.get_instances_with_marc_records inst
 WHERE inst.instance_id > 'aee9b375-f51a-588a-8afe-b89fefe87da0'::uuid
    AND ( inst.source = 'MARC'
 OR inst.source = 'MARC_SHARED' OR inst.source = 'CONSORTIUM-MARC')     AND inst.instance_updated_date >= cs00000001_0042_mod_inventory_storage.dateOrMin(timestamptz '2022-12-21T00:00:00Z')
    AND inst.instance_updated_date <= cs00000001_0042_mod_inventory_storage.dateOrMax(timestamptz '2024-01-07T00:00:00Z')
ORDER BY instance_id
LIMIT 101;

Test #7. Record source = Source record storage

Memory and CPU utilization, %

Memory		CPU
mod-quick-marc-b	60.89	mod-pubsub-b	10.31
okapi-b	57.48	mod-oai-pmh-b	9.37
mod-inventory-b	54.56	mod-inventory-b	9.06
mod-source-record-manager-b	48.82	mod-quick-marc-b	7.46
mod-source-record-storage-b	40.16	edge-oai-pmh-b	6.23
mod-pubsub-b	35.55	okapi-b	4.68
mod-circulation-storage-b	35.41	mod-source-record-storage-b	2.41
edge-oai-pmh-b	34.19	mod-di-converter-storage-b	2.21
mod-di-converter-storage-b	29.24	mod-source-record-manager-b	1.67
mod-oai-pmh-b	28.28	nginx-okapi	1.32
mod-inventory-storage-b	17.06	mod-inventory-storage-b	0.67
nginx-okapi	5.02	pub-okapi	0.10
pub-okapi	4.80

Service CPU Utilization

mod-oai-pmh - 10%

Service Memory Utilization

mod-oai-pmh - 28%

RDS CPU Utilization

RDS was in range from 4 to 8% in average. The highest spike was 15%.

RDS Database Connections

During test 1000 connections in average with spikes to 1400 connections regularly - every 30 minutes.

Full Harvests Duration Table

All harvests duration, instances per second

Filtered by returned_instances_counter (sort largest to smallest)

tenant	started_date	last_updated_date	returned_instances_counter	duration	Duration, in seconds	Instances per second
cs00000001_0042	2024-01-30 07:58:05.270091+00	2024-01-30 09:24:25.143512+00	634673	01:26:19	5179	122.55
cs00000001_0024	2024-01-30 23:32:06.328008+00	2024-01-31 00:12:09.513664+00	385121	00:40:03	2403	160.27
cs00000001_0051	2024-01-31 05:19:16.762325+00	2024-01-31 05:55:05.370195+00	351323	00:35:48	2148	163.56
cs00000001_0006	2024-01-30 19:47:56.994035+00	2024-01-30 20:18:04.047742+00	293579	00:30:07	1807	162.47
cs00000001_0007	2024-01-30 20:18:04.928054+00	2024-01-30 20:45:31.841288+00	279491	00:27:26	1646	169.80
cs00000001_0012	2024-01-30 21:44:15.496785+00	2024-01-30 22:07:21.604411+00	227487	00:23:06	1386	164.13
cs00000001_0038	2024-01-31 02:09:28.951308+00	2024-01-31 02:30:32.110773+00	225562	00:21:03	1263	178.59
cs00000001_0050	2024-01-31 05:00:49.438666+00	2024-01-31 05:19:16.116316+00	190734	00:18:26	1106	172.45
cs00000001_0055	2024-01-31 06:25:03.701594+00	2024-01-31 06:43:58.086662+00	176007	00:18:54	1134	155.21
cs00000001_0056	2024-01-31 06:43:58.675942+00	2024-01-31 06:59:40.582524+00	160975	00:15:41	941	171.07
cs00000001_0008	2024-01-30 20:45:32.795785+00	2024-01-30 21:05:08.613622+00	160023	00:19:35	1175	136.19
cs00000001_0026	2024-01-31 00:23:26.324646+00	2024-01-31 00:39:18.350149+00	153911	00:15:52	952	161.67
cs00000001_0016	2024-01-30 22:21:16.557905+00	2024-01-30 22:38:40.512729+00	153332	00:17:23	1043	147.01
cs00000001_0052	2024-01-31 05:55:06.081995+00	2024-01-31 06:11:33.012233+00	137938	00:16:26	986	139.90
cs00000001_0003	2024-01-30 19:02:32.304572+00	2024-01-30 19:22:49.097716+00	136290	00:20:16	1216	112.08
cs00000001_0035	2024-01-31 01:36:48.717188+00	2024-01-31 01:47:48.176875+00	135703	00:10:59	659	205.92
cs00000001_0039	2024-01-31 02:30:32.811619+00	2024-01-31 02:44:00.517573+00	128520	00:13:27	807	159.26
cs00000001_0058	2024-01-31 07:05:18.862821+00	2024-01-31 07:22:08.347929+00	123816	00:16:49	1009	122.71
cs00000001_0036	2024-01-31 01:47:48.70498+00	2024-01-31 02:02:30.044027+00	114168	00:14:41	881	129.59
cs00000001_0040	2024-01-31 02:44:01.342285+00	2024-01-31 02:57:34.101665+00	111991	00:13:32	812	137.92
cs00000001_0009	2024-01-30 21:05:09.266434+00	2024-01-30 21:20:52.835695+00	110917	00:15:43	943	117.62
cs00000001_0005	2024-01-30 19:33:59.876671+00	2024-01-30 19:47:55.908675+00	110091	00:13:56	836	131.69
cs00000001_0046	2024-01-31 04:33:45.182805+00	2024-01-31 04:44:16.75419+00	109676	00:10:31	631	173.81
cs00000001_0030	2024-01-31 00:56:23.730637+00	2024-01-31 01:08:34.146005+00	97588	00:12:10	730	133.68
cs00000001_0011	2024-01-30 21:32:30.564789+00	2024-01-30 21:44:14.603914+00	94169	00:11:44	704	133.76
cs00000001_0023	2024-01-30 23:20:35.961953+00	2024-01-30 23:32:05.687917+00	91516	00:11:29	689	132.82
cs00000001_0061	2024-01-31 07:36:35.789584+00	2024-01-31 07:48:44.358093+00	87115	00:12:08	728	119.66
cs00000001_0022	2024-01-30 23:08:15.819582+00	2024-01-30 23:20:34.978638+00	86899	00:12:19	739	117.59
cs00000001_0025	2024-01-31 00:12:10.143554+00	2024-01-31 00:23:25.769649+00	84214	00:11:15	675	124.76
cs00000001_0021	2024-01-30 22:56:27.762303+00	2024-01-30 23:08:14.66468+00	83985	00:11:46	706	118.96
cs00000001_0010	2024-01-30 21:20:53.410569+00	2024-01-30 21:32:29.870923+00	83253	00:11:36	696	119.62
cs00000001_0032	2024-01-31 01:14:54.996634+00	2024-01-31 01:24:50.266775+00	80073	00:09:55	595	134.58
cs00000001_0045	2024-01-31 04:21:11.747423+00	2024-01-31 04:33:44.488302+00	78318	00:12:32	752	104.15
cs00000001_0004	2024-01-30 19:22:50.065381+00	2024-01-30 19:33:58.807122+00	77784	00:11:08	668	116.44
cs00000001_0031	2024-01-31 01:08:35.025465+00	2024-01-31 01:14:54.197098+00	71894	00:06:19	379	189.69
cs00000001_0044	2024-01-31 04:11:28.963726+00	2024-01-31 04:21:11.011287+00	70870	00:09:42	582	121.77
cs00000001_0001	2024-01-30 18:47:33.401981+00	2024-01-30 18:57:25.506314+00	65358	00:09:52	592	110.40
cs00000001_0028	2024-01-31 00:46:46.489521+00	2024-01-31 00:55:33.616289+00	64554	00:08:47	527	122.49
cs00000001_0053	2024-01-31 06:11:33.683951+00	2024-01-31 06:21:27.297008+00	61833	00:09:53	593	104.27
cs00000001_0041	2024-01-31 02:57:34.910847+00	2024-01-31 03:06:51.97583+00	61089	00:09:17	557	109.68
cs00000001_0048	2024-01-31 04:51:58.861674+00	2024-01-31 04:59:49.819602+00	53345	00:07:50	470	113.50
cs00000001_0047	2024-01-31 04:44:17.709472+00	2024-01-31 04:51:58.309158+00	48530	00:07:40	460	105.50
cs00000001_0027	2024-01-31 00:39:18.818234+00	2024-01-31 00:46:45.719918+00	48175	00:07:26	446	108.02
cs00000001_0033	2024-01-31 01:24:51.028727+00	2024-01-31 01:33:02.341647+00	45884	00:08:11	491	93.45
cs00000001_0020	2024-01-30 22:49:10.153732+00	2024-01-30 22:56:27.024515+00	44257	00:07:16	436	101.51
cs00000001_0060	2024-01-31 07:28:54.094613+00	2024-01-31 07:36:34.714826+00	38394	00:07:40	460	83.47
cs00000001_0059	2024-01-31 07:22:09.021211+00	2024-01-31 07:28:53.445342+00	37713	00:06:44	404	93.35
cs00000001_0015	2024-01-30 22:15:32.594419+00	2024-01-30 22:21:15.705491+00	26898	00:05:43	343	78.42
cs00000001_0037	2024-01-31 02:02:30.987839+00	2024-01-31 02:09:28.411798+00	24225	00:06:57	417	58.09
cs00000001_0057	2024-01-31 06:59:41.571592+00	2024-01-31 07:05:17.933701+00	22595	00:05:36	336	67.25
cs00000001_0002	2024-01-30 18:57:26.363148+00	2024-01-30 19:02:31.4724+00	22369	00:05:05	305	73.34
cs00000001_0018	2024-01-30 22:42:19.32858+00	2024-01-30 22:48:26.683217+00	22262	00:06:07	367	60.66
cs00000001_0014	2024-01-30 22:10:54.509386+00	2024-01-30 22:15:31.769242+00	20744	00:04:37	277	74.89
cs00000001_0013	2024-01-30 22:07:22.488948+00	2024-01-30 22:10:53.870081+00	19130	00:03:31	211	90.66
cs00000001_0034	2024-01-31 01:33:03.311241+00	2024-01-31 01:36:48.087995+00	18239	00:03:44	224	81.42
cs00000001_0017	2024-01-30 22:38:41.130227+00	2024-01-30 22:42:18.265948+00	16490	00:03:37	217	75.99
cs00000001_0054	2024-01-31 06:21:27.853456+00	2024-01-31 06:25:02.81652+00	16296	00:03:34	214	76.15
cs00000001_0049	2024-01-31 04:59:50.428684+00	2024-01-31 05:00:48.785603+00	6951	00:00:58	58	119.84
cs00000001_0019	2024-01-30 22:48:27.522764+00	2024-01-30 22:49:09.628092+00	5311	00:00:42	42	126.45
cs00000001_0029	2024-01-30 15:58:19.508812+00	2024-01-30 15:59:25.424545+00	5139	00:01:05	65	79.06
cs00000001_0043	2024-01-31 04:11:22.798694+00	2024-01-31 04:11:28.361622+00	830	00:00:05	5	166.00
cs00000001_0062	2024-01-31 07:48:45.009151+00	2024-01-31 07:48:45.009151+00	0

Database load

Top SQL

Appendix

Methodology/Approach MCPT, PCON

Methodology/Approach

OAI-PMH (incremental harvesting) was carried out by JMeter script from carrier with 2 main requests:

/oai/records?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=[APIKey]
/oai/records?verb=ListRecords&apikey=[APIKey]&resumptionToken=[resumptionToken]

to extract the required number of records was used loop counter with following configuration:

98 loop counts for 10K records;
498 loop counts for 50K records;
4999 loop counts for 500K records;

To run the incremental harvesting test the next time ranges were defined by experimental means. The time range for Test 2* was extended due to the impossibility of harvesting the defined number of records, but the next tests were run after adding 800K instances to database.

	Start date	Until date
Test 1.	2022-12-21	2023-12-10

OAI-PMH (full harvesting)

Before running OAI-PMH with full harvest, following database commands to optimize the tables were executed (from https://folio-org.atlassian.net/wiki/display/FOLIOtips/OAI-PMH+Best+Practices#OAIPMHBestPractices-SlowPerformance):

REINDEX index <tenant>_mod_inventory_storage.audit_item_pmh_createddate_idx ;
REINDEX index <tenant>_mod_inventory_storage.audit_holdings_record_pmh_createddate_idx;
REINDEX index <tenant>_mod_inventory_storage.holdings_record_pmh_metadata_updateddate_idx;
REINDEX index <tenant>_mod_inventory_storage.item_pmh_metadata_updateddate_idx;
REINDEX index <tenant>_mod_inventory_storage.instance_pmh_metadata_updateddate_idx;
analyze verbose <tenant>_mod_inventory_storage.instance;
analyze verbose <tenant>_mod_inventory_storage.item;
analyze verbose <tenant>_mod_inventory_storage.holdings_record;

Execute the following query in a related database for removing existed 'instances' created by previous harvesting request and a request itself:

TRUNCATE TABLE fs09000000_mod_oai_pmh.request_metadata_lb cascade

Full harvesting test with consortia enabled triggered from central tenant level was running from ptf-windows machine using EBSCO Harvester. The following cmd command (cmd should be run in the same directory as EBSCO Harvester) start EBSCO Harvester:

OAIPMHHarvester.exe -HarvestMode=full -DefinitionId=pcon-marc21-with-holdings -HarvesterWebClientTimeout_Seconds=0s=0

To start full harvesting sequentially for all member tenants use APIKEY of central tenant. With definition:

Harvest definition

<?xml version="1.0" encoding="UTF-8"?>
<HarvestDefinition xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="HarvestDefinition.xsd">
<id>PCON</id>
<Description>PCON</Description>
<Urls>

<Url>https://edge-ptf-consortium.int.aws.folio.org/oai/eyJzIjoiYm5PUWN4dVZabyIsInQiOiJjczAwMDAwaW50IiwidSI6IkVCU0NPRWRnZSJ9</Url>

</Urls>

<ThrottledInMiliseconds>0</ThrottledInMiliseconds>


<MetadataFormat>marc21_withholdings</MetadataFormat>




<Sets use="Harvested">
</Sets>



</HarvestDefinition>

To start full or incremental harvesting in parallel triggered from member tenant level use APIKEY of member tenant (host name should stay as for central tenant). It was implemented by jMeter script.

Preparing APIKEY for member tenants "s" parameter should be the same as it is in the central tenant.

Infrastructure

Infrastructure MCPT

Infrastructure

Environment: MCPT
Release: Poppy (2023 R2)

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)
2 instances of db.r6.4xlarge database instances, one writer
MSK tenant
- 4 brokers
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- og.retention.minutes=480
- default.replication.factor=3

Modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mcpt-pvt
Tue Jan 30 09:03:55 UTC 2024
mod-remote-storage	10	mod-remote-storage:3.0.1	2	4920	4472	1024	3960	512	512	FALSE
mod-inventory-update	6	mod-inventory-update:3.2.1	2	1024	896	128	768	88	128	FALSE
pub-edge	1	pub-edge:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-inventory-storage	10	mod-inventory-storage:27.0.4	2	4096	3690	2048	3076	384	512	FALSE
edge-oai-pmh	9	edge-oai-pmh:2.7.2	2	1512	1360	1024	1440	384	512	FALSE
mod-circulation-storage	10	mod-circulation-storage:17.1.7	2	2880	2592	1536	1814	384	512	FALSE
mod-source-record-storage	10	mod-source-record-storage:5.7.5	2	5600	5000	2048	3500	384	512	FALSE
mod-source-record-manager	10	mod-source-record-manager:3.7.7	2	5600	5000	2048	3500	384	512	FALSE
mod-quick-marc	10	mod-quick-marc:5.0.1	1	2288	2176	128	1664	384	512	FALSE
nginx-okapi	2	nginx-okapi:2023.06.14	2	1024	896	512	0	0	0	FALSE
okapi-b	1	okapi:5.1.2	3	1684	1440	1024	922	384	512	FALSE
mod-oai-pmh	9	mod-oai-pmh:3.12.8	2	4096	3690	2048	3076	384	512	FALSE
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	128	768	0	0	FALSE

Infrastructure PCON

Infrastructure

Environment: PCON
Release: Poppy (2023 R2)

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK tenant
- 4 brokers
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- og.retention.minutes=480
- default.replication.factor=3

Modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
pcon-pvt

pub-edge	2	pub-edge:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-inventory-storage	1	mod-inventory-storage:27.0.0	2	4096	3690	2048	3076	384	512	FALSE
edge-oai-pmh	1	edge-oai-pmh:2.7.0	2	1512	1360	1024	1440	384	512	FALSE
mod-circulation-storage	1	mod-circulation-storage:17.1.0	2	2880	2592	1536	1814	384	512	FALSE
mod-source-record-storage	1	mod-source-record-storage:5.7.0	2	5600	5000	2048	3500	384	512	FALSE
mod-inventory	1	mod-inventory:20.1.0	2	2880	2592	1024	1814	384	512	FALSE
mod-source-record-manager	1	mod-source-record-manager:3.7.0	2	5600	5000	2048	3500	384	512	FALSE
mod-quick-marc	1	mod-quick-marc:5.0.0	1	2288	2176	128	1664	384	512	FALSE
nginx-okapi	2	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	FALSE
okapi-b	2	okapi:5.1.1	3	1684	1440	1024	922	384	512	FALSE
mod-oai-pmh	1	mod-oai-pmh:3.12.0	2	4096	3690	2048	3076	384	512	FALSE
pub-okapi	2	pub-okapi:2023.06.14	2	1024	896	128	768	0	0	FALSE

Browser not supported