Overview

In the scope of PERF-698 - Getting issue details... STATUS - * Poppy Release | Data import | Create MARC authority Records it's needed to run tests to answer questions:

Determine the time it takes to complete Data-import on PCP1 env
Determine the main modules that are involved in the process (if obvious or if known)
Test specific settings or items or scenarios:
Check-in and Checkout (CICO) is in progress and there are 5 concurrent users, and run DI for 1K - 5K- 10K-25K-50K
Test concurrent DI jobs with multi-tenants in the same cluster.

Summary

1K, 5K, 10K, 50K data import jobs finished successfully without errors or issues. 22K completed with 67 errors, problem in data, because same errors were on 3 tenants.
General summary. Data Import MARC Authorities work on Poppy like on the previous releases. DI duration is about the same and resource utilization has the same behavior as in the previous releases.
The environment is stable and can process 5 DI jobs in parallel on 3 tenants.

Test 1. DI Primary tenant 1k-5K-10K-22K-50K

DI performance for:
- 1K is faster by Orchid but on the same level as Morning Glory and Nolana
- 5K and 10K DI duration is about the same as Orchid, Morning Glory, and Nolana
- 23K and 50K DI duration is longer on Poppy in comparison to Orchid, but faster than on Morning Glory and Nolana
Resource Utilization during data imports was stable during the test.
- Maximal CPU consumption was for mod-quick-marc service about 55-60% of CPU and spiked up to 80% on mod-data-import-b, when the 50K records file was uploading;
- No memory leaks;
- RDS utilization was about 95%;

Test 2. DI Primary tenant 1k-5K-10K-22K-50K + CI/CO 5VU

Average response time is slower from 2 up to 18% for small files(1K, 5K, and 10K ), but for big files (23K, 50K);
The duration of DI during CICO is similar to results "without CICO";
Memory and CPU utilization were without problems. The most consuming CPU service is mod-inventory-b about 80%. RDS utilization was about 95%;

Test 3. Multitenant DI 50K-22K-10K-5K-1K

All DI finished successfully(except 23K, DI file has 67 broken records), with no memory leaks.

Test Runs & Results

Job Profile Default - Create SRS MARC Authority

Test #	# of records	% creates	File	DI duration Morning Glory	DI duration Nolana	DI duration Orchid	DI duration Poppy
1	1,000	100	1k_marc_authority.mrc?api=v2	24 s	27 s	41 sec	29 sec
2	5,000	100	LC_SUBJ_msplit00000000.mrc?api=v2	1 min 21 s	1 min 15 s	1min 21s	1 min 38 sec
3	10,000	100	msplit00000000.mrc?api=v2	2 min 32 s	2 min 31 s	2min 53s	2 min 53 sec
4	22778	100	msplit00000013.mrc?api=v2	11 min 14 s	7 min 7 s	5 min 42s	6 min 24 sec
5	50,000	100	50000_authorityrecords.mrc?api=v2	22 min	11 min 24 s	11 min 11s	13 min 48 sec

Test with CICO 5 concurrent users and DI 1K and 5K and 50K

	CICO, Median time without DI	CICO, 95% time without DI
Check-In	516 ms	567 ms
Check-Out	910 ms	2094 ms

Test #	# of records	DI Duration with CICO	CI time Avg without	Baseline CI delta	CI time 95th pct	Baseline CI delta	CO time Avg	Baseline CO Avg Delta	CO time 95th pct	Baseline CO delta
1	1,000	35 sec	0.525	+2%	0.576	+1%	1.078	+18%	1.326	-27%
2	5,000	1 min 41 sec	0.513	-1%	0.612	+8%	0.9	-0.001%	1.019	-52%
3	10,000	3 min 4 sec	0.581	+12%	0.685	+20%	1.016	+11%	1.321	-27%
4	22,778	6 min 32 sec	0.598	+16%	1.542	+171%	1.244	+36%	1.729	-18%
5	50,000	13 min 48 sec	0.671	+30%	1.953	+244%	1.51	+65%	2.09	-0.01%

Multitenant testing

File with 50K, 23K, 10K, 5K, 1K loaded sequences and launched on Tenant1, Tenant2, and Tenant3

Test	Num of records	Tenant 1 pcp1-00 duration	Tenant 2 pcp1-01 duration	Tenant 3 pcp1-02 duration
1.	1000	66 min	75 min	74 min
2.	5000	67 min	76 min	75 min
3.	10,000	67 min	76 min	74 min
4.	22,778	67 min*	73 min*	71 min*
5.	50,000	62 min	56 min	54 min

*Completed with errors

Error detailes

And on all of the tenants test 4 finished with 67 errors, with the same messages

Error message:
io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"JSON parse error: Cannot deserialize value of type `java.util.UUID` from String \"76c74801-afec-45a0-aad7-3ff23591e147 76c74801-afec-45a0-aad7-3ff23591e147\": UUID has to be represented by standard 36-char representation","type":"HttpMessageNotReadableException","code":"validation"}],"total_records":1}

Resource utilization

Test 1. DI Primary tenant 1k-5K-10K-22K-50K

Data-import 17:43 → 1000 records import; 17:50 → 5000 records import; 17:55-18:00 →10K records import; 18:05- 18:10 → 23K records import; 18:20- 18:35→ 50K records import.

Service CPU Utilization

CPU Utilization during data-imports was stable during the test, mod-quick-marc-b consumed about 55-60% of CPU and Spikes up to 80% on mod-data-import-b, but we saw the same behavior when large files were uploaded for data-import.
CPU utilization after tests returned to its before-test statements.

Memory utilization

RDS utilization was about 95%

Database load

Top SQL

Test 2. DI Primary tenant 1k-5K-10K-22K-50K + CI/CO 5VU

CICO Responce time graph
Data import 1k- 23:53,15 - 23:53,50; 5k- 23:57,17 - 23:59; 10k- 00:04 - 00:07; 23k- 00:09 - 00:15; 3k- 00:18 - 00:32;

Service CPU Utilization

CPU Utilization during data imports was stable during the test, and after the tests finished returned to its before-test statements.

Memory utilization was stable during the tests.

RDS utilization was about 95%

Number of DB connections

Database load

Top SQL

Test 3. Multitenant DI 50K-22K-10K-5K-1K.

Service CPU Utilization

CPU Utilization during data imports was stable during the test, except 2 spikes on mod-permission-b services to about 110%, and after the tests finished returned to its before-test statements.

Memory utilization was stable during the tests.

Maximal RDS utilization was about 96%

Number of DB connections

Database Load

Appendix

Infrastructure

PTF -environment pcp1

11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK tenant
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
pcp1-pvt
pcp1-pvt											mod-entities-links	13	mod-entities-links:2.0.3	2	2592	2480	400	1440	0	1024	FALSE
mod-data-import	11	mod-data-import:3.0.3	1	2048	1844	256	1292	384	512	FALSE
mod-permissions	32	mod-permissions:6.4.0	2	1684	1544	512	1024	384	512	FALSE
mod-inventory-storage	12	mod-inventory-storage:27.0.3	2	4096	3690	2048	3076	384	512	FALSE
mod-source-record-storage	15	mod-source-record-storage:5.7.3	2	5600	5000	2048	3500	384	512	FALSE
mod-inventory	11	mod-inventory:20.1.3	2	2880	2592	1024	1814	384	512	FALSE
mod-di-converter-storage	15	mod-di-converter-storage:2.1.2	2	1024	896	128	768	88	128	FALSE
mod-circulation	12	mod-circulation:24.0.8	2	2880	2592	1536	1814	384	512	FALSE
mod-source-record-manager	14	mod-source-record-manager:3.7.4	2	5600	5000	2048	3500	384	512	FALSE
mod-quick-marc	9	mod-quick-marc:5.0.0	1	2288	2176	128	1664	384	512	FALSE
nginx-okapi	9	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	FALSE
okapi-b	11	okapi:5.1.2	3	1684	1440	1024	922	384	512	FALSE

Browser not supported