Table of Contents
Overview

...

Table of Contents
Overview

This document contains the results of testing concurrent Data Import with file splitting feature for MARC Bibliographic records in the Poppy release.

...

Ticket:

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-715

Summary

Data import Import duration approximately doubling with of 10k and 25k jobs approximately doubles when increasing the number of concurrent jobs on different tenants. This trend is consistent across the central main/first tenant and other tenants.

Smoke test with 50k didn't reveal some issues. Duration for 3 three concurrent DI Create jobs was three times 3x higher than one DI on central the main tenant that ; this only confirm previous statement about the concurrency affecteffect.

Maximum average CPU utilization was different during create and update jobs. Top two modules during DI Create jobs in mod-inventory-b - 123%, mod-quick-marc-b - 76%, Update jobs mod-inventory-b - 182%, mod-quick-marc-b - 122%.

...

Top long query for failed job on third tenant during DI Create job with 10k- SELECT jsonb,id FROM fs07000002_mod_inventory_storage.instance_holdings_item_view. Average latency- 386455.99 ms/call

Test Runs

Test #	Scenario	Load level

Comment

1 - Concurrent Create imports	DI MARC Bib Create	10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants
2 - Concurrent Update imports	DI MARC Bib Update	10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants
3

- Concurrent Create imports ("smoke test") of 50K

DI MARC Bib Create

50k concurrently on 3

tenants - smoke test

tenants

Test Results

...

As the number of concurrent Data Import jobs increases and file size grow grows, the duration of DI jobs grow grows proportionally.
Smoke tests Test finished successfully for 3 concurrent DI Create jobs with 50kof 50K each.
10k
DI Create # of test Number of concurrent jobs Central
Main tenant
(fs09000000)
Second tenant
(fs07000001)
Third tenant
(fs07000002)
10K Baseline 1 00:04:56

1 2 00:10:43 00:10:37
2 3 00:21:12 00:21:06 00:20:57 *
25k25K Baseline 1 00:11:24

3 2 00:23:44 00:23:30
4 3 00:37:11 00:37:05 00:36:58
DI Update

10k10K Baseline 1 00:06:32

5 2 00:09:47 00:11:26
6 3 00:19:08 00:19:06 00:18:31
25k25K Baseline 1 00:15:13

7 2 00:30:49 00:30:52
8 3 00:47:47 00:48:17 00:47:54
DI Create (smoke Smoke test)

50k50K 9 1 00:22:31

10 3 01:12:54 01:12:44 01:12:35
* - Errors occurred only in 10k 10K DI Create jobs running on third tenant during 3 concurrent jobs test. The errors did not reproduce during subsequent tests.
...
Update jobs

DB Connections

Create jobs DB connections for 2 tenants Create jobs tenants - 710, for 3 tenants Create jobs - - 870

Update jobs DB connections for 2 tenants Create jobs tenants - 630, for 3 tenants Create jobs - 785

DB connections needed for every additional job processing concurrently on different tenant - 150.

DB Connections for Create jobs

DB Connections for Update jobs

DB load

Create jobs

...

Expand

title	Logs

pcp1/mod-search

failure in bulk execution - 186 errors during all update jobs, >4000 errors during create jobs

10:59:59 [] [] [] [] WARN KafkaMessageListener Failed to index resource event [eventType: CREATE, tenantId: fs09000000, id: 5cc8ef78-cb05-49fa-8274-1cba1d660aad]

index [pcp1_instance_fs09000000], id [f7aea9b8-614e-4050-9dbd-e2f8a884c06b], message [OpenSearchException[OpenSearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [indices:data/write/bulk[s]] would be [16502737514/15.3gb], which is larger than the limit of [16320875724/15.1gb], real usage: [16499671264/15.3gb], new bytes reserved: [3066250/2.9mb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=3103382/2.9mb]]]]

feign.FeignException$Unauthorized: [401 Unauthorized] during [GET] to [http://inventory-view/instances?query=id%3D%3D%28%221e9b752b-6cc3-433b-ae90-cbafdc307cb6%22%29&limit=1] [InventoryViewClient#getInstances(CqlQuery,int)]: [Invalid token]

org.folio.search.exception.SearchOperationException: Failed to perform elasticsearch request [index=pcp1_contributor_fs09000000, type=bulkApi, message: 30,000 milliseconds timeout on connection http-outgoing-265 [ACTIVE]]

Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-294 [ACTIVE]

WARN essageBatchProcessor Failed to process batch, attempting to process resources one by one

feign.RetryableException: timeout executing GET http://inventory-view/instances?query=id%3D%3D%28%22c4ac5388-4b64-4de5-8930-d3be806c1b7f%22%20or%20%229228d6de-c8e3-4e62-85c7-f5b8b45fb649%22%20or%20%2288913517-17dc-4d90-be8b-9560c1f30a01%22%20or%20%220514f12a-680d-415d-b262-ba82f4dd3e76%22%20or%20%2205253541-bf8a-465a-b91d-b2cb4ff0944d%22%20or%20%222f7b4924-8781-4834-b25f-36fc430d8f5d%22%20or%20%22b2dad08a-05e9-41d2-8742-25c98daf7fbe%22%20or%20%22be30bb11-58da-4b06-beea-e204a0823438%22%20or%20%22930b4625-8eb3-492a-bae8-388480364e67%22%20or%20%2223925f8a-efb4-41a2-9002-48292f6419f3%22%20or%20%22765cb2e6-96a4-4b33-99fd-b38011be999f%22%20or%20%22a3666aca-4963-4e94-95ab-dd3d790ffdd3%22%20or%20%2226bb45e4-dfc0-4915-9ec8-0187d334651d%22%20or%20%220c536304-7330-46d2-a19e-9b69ca13591a%22%20or%20%22ee237647-875c-405e-8ce1-fd55f701d83b%22%20or%20%22869ba2aa-f465-42b4-b4ba-b47ccd29d6ac%22%20or%20%221aa2e06e-b647-4e7b-8fa8-9804c65e1dc1%22%20or%20%22a7096f26-0363-49b4-803c-31e5661b12de%22%20or%20%22042395fb-34a7-4412-a78a-d541ef948922%22%20or%20%2273342f96-f5b5-42d6-ab41-1ef121aef0d5%22%20or%20%22fa028d18-e45e-434d-85d2-e8c4e5db9519%22%20or%20%22c0d1a681-66ea-4cc8-bf68-5771cf8a93e9%22%20or%20%22d7ece421-5dae-4519-9d09-100f22b47007%22%20or%20%2225ca20d1-d0e1-414a-b228-b74ceaba2512%22%20or%20%2274fcad26-9ddf-4cf9-8206-f665374a37f3%22%20or%20%221fd756cd-e573-4287-9c3e-c29408ee8709%22%20or%20%22a322007c-f578-4c93-9239-e6558a393710%22%20or%20%22a2ed3841-a9d2-4364-a3fe-939e2bffbe24%22%20or%20%22b31f92f9-027e-40bf-a5b1-bb4e50241a46%22%20or%20%2204175b0b-9dea-403b-911c-82d5f6a2fbe2%22%20or%20%22f4ebc0f9-adb7-4458-a4ae-8d7996b3b4f8%22%20or%20%22e1e6f55c-7720-4a23-a73a-3202746c7c75%22%20or%20%2238a43c32-d2de-454f-b7d7-7725b5bab61e%22%20or%20%22b8f17eec-f61c-4b29-9769-3b8d91a6dae4%22%20or%20%22e93b2a0c-939b-4c31-98f9-c85cb52081eb%22%20or%20%22983fd8c1-cd4a-4087-a4b9-b1c3dd11c08e%22%20or%20%222b8a75db-2ec2-4190-8702-c4dca1067bf6%22%20or%20%22f73f7d21-deab-4af4-9163-7f374f1d56d2%22%20or%20%229819b074-3174-429e-8f7f-1d6312d9630f%22%20or%20%2256743d6d-3df9-4bbf-9495-cc9f2b95e60b%22%20or%20%229f146268-ad02-46b2-8f0d-a5f64fc8579b%22%20or%20%22fb26676f-2d20-4309-8fc5-aa1c09962618%22%20or%20%222b39c2de-6374-46af-b78e-1c83d669c991%22%20or%20%22a7fd3130-6752-40b4-b2a4-cc6f1fee0349%22%20or%20%22e7561b18-c9da-43f8-97d4-9125261de4b6%22%20or%20%221106c6ed-519b-4efa-a73c-deae1dc0570d%22%20or%20%220ea912aa-6107-490f-8855-350c7edc0060%22%20or%20%225ec701f4-af40-4464-b6a4-ba9f14ca7d28%22%20or%20%22e2a15de7-b793-4949-beed-9f56bd9cde9d%22%20or%20%22879210f9-cae8-4202-9dcd-89657a5f8113%22%29&limit=50


pcp1/mod-authtoken

org.folio.auth.authtokenmodule.tokens.TokenValidationException: Access token has expired

Infrastructure

PTF -environment pcp1

...

2 database instances, writer/reader

...

db.r6g.xlarge

...

4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3

...

number of errors - 23400. The errors happen only during DI in fs07000002 tenant

| filter @logStream like "pcp1/mod-authtoken"

| filter @message like "ERROR FilterApi"

13:48:00 [595516/users] [fs07000002] [] [mod-authtoken] ERROR FilterApi Unable to retrieve permissions for system-user: User does not exist: 8cc96687-ea63-44cb-ab5f-a73bc6985324 request took 7 ms

Infrastructure

PTF -environment pcp1

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader

Name Memory GIB vCPUs max_connections
db.r6g.xlarge
32 GiB 4 vCPUs 2731
Number of records in DB:
- fs09000000
  - instances - 25.129.941
  - items - 26.299.440
  - holdings - 25.392.570
- fs07000001
  - nstances - 10.039.613
  - items - 1.423.844
  - holdings - 10.461.259
- fs07000002
  - nstances - 1.114.273
  - items - 1.106.537
  - holdings - 1.106.539
MSK tenant
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
pcp1-pvt	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
mod-remote-storage	10(11)*	3.0.0	2	4920	4472	1024	3960	512	512
mod-data-import	18(20)*	3.0.7	1	2048	1844	256	1292	384	512
mod-authtoken	13(16)*	2.14.1	2	1440	1152	512	922	88	128
mod-configuration	9(10)*	5.9.2	2	1024	896	128	768	88	128
mod-users-bl	9(10)*	7.6.0	2	1440	1152	512	922	88	128
mod-inventory-storage	12(15)*	27.0.3(27.0.4)*	2	4096	3690	2048	3076	384	512
mod-circulation-storage	12(14)*	17.1.3(17.1.7)*	2	2880	2592	1536	1814	384	512
mod-source-record-storage	15(18)*	5.7.3(5.7.5)*	2	5600	5000	2048	3500	384	512
mod-inventory	11(14)*	20.1.3(20.1.7)*	2	2880	2592	1024	1814	384	512
mod-di-converter-storage	15(18)*	2.1.2(2.1.5)*	2	1024	896	128	768	88	128
mod-circulation	12(14)*	24.0.8(24.0.11)*	2	2880	2592	1536	1814	384	512
mod-pubsub	11(13)*	2.11.2(2.11.3)*	2	1536	1440	1024	922	384	512
mod-patron-blocks	9(10)*	1.9.0	2	1024	896	1024	768	88	128
mod-source-record-manager	14(17)*	3.7.4(3.7.8)*	2	5600	5000	2048	3500	384	512
mod-quick-marc	9(11)*	5.0.0(5.0.1)*	1	2288	2176	128	1664	384	512
nginx-okapi	9	2023.06.14	2	1024	896	128	0	0	0
okapi-b	11	5.1.2	3	1684	1440	1024	922	384	512
mod-feesfines	10(11)*	19.0.0	2	1024	896	128	768	88	128
pub-okapi	9	2023.06.14	2	1024	896	128	768	0	0

Expand

title	All modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
pcp1-pvt
Fri Mar 15 17:12:45 UTC 2024
mod-remote-storage	11	mod-remote-storage:3.0.0	2	4920	4472	1024	3960	512	512
mod-ncip	10	mod-ncip:1.14.4	2	1024	896	128	768	88	128
mod-finance-storage	10	mod-finance-storage:8.5.0	2	1024	896	1024	700	88	128
mod-agreements	10	mod-agreements:6.0.2	2	1592	1488	128	0	0	0
mod-ebsconet	10	mod-ebsconet:2.1.1	2	1248	1024	128	700	128	256
edge-sip2	8	edge-sip2:3.1.1	2	1024	896	128	768	88	128
mod-organizations	10	mod-organizations:1.8.0	2	1024	896	128	700	88	128
mod-settings	11	mod-settings:1.0.2	2	1024	896	200	768	88	128
edge-dematic	10	edge-dematic:2.1.0	1	1024	896	128	768	88	128
mod-data-import	20	mod-data-import:3.0.7	1	2048	1844	256	1292	384	512
mod-search	20	mod-search:3.0.5	2	2592	2480	2048	1440	512	1024
mod-tags	10	mod-tags:2.1.0	2	1024	896	128	768	88	128
mod-authtoken	16	mod-authtoken:2.14.1	2	1440	1152	512	922	88	128
edge-courses	2	edge-courses:1.3.0	2	1024	896	128	768	88	128
mod-notify	10	mod-notify:3.1.0	2	1024	896	128	768	88	128
mod-inventory-update	10	mod-inventory-update:3.2.1	2	1024	896	128	768	88	128
mod-configuration	10	mod-configuration:5.9.2	2	1024	896	128	768	88	128
mod-orders-storage	10	mod-orders-storage:13.6.0	2	1024	896	512	700	88	128
edge-caiasoft	10	edge-caiasoft:2.1.0	2	1024	896	128	768	88	128
mod-login-saml	18	mod-login-saml:2.7.1	2	1024	896	128	768	88	128
mod-erm-usage-harvester	11	mod-erm-usage-harvester:4.4.1	2	1024	896	128	768	88	128
mod-password-validator	10	mod-password-validator:3.1.0	2	1440	1298	128	768	384	512
mod-licenses	10	mod-licenses:5.0.2	2	2480	2312	128	1792	384	512
mod-gobi	10	mod-gobi:2.7.1	2	1024	896	128	700	88	128
mod-fqm-manager	9	mod-fqm-manager:1.1.0-SNAPSHOT.1078	2	1024	896	128	768	88	128
mod-bulk-operations	9	mod-bulk-operations:1.1.7	2	3072	2600	1024	1536	384	512
mod-graphql	16	mod-graphql:1.12.0	0	1024	896	128	768	88	128
mod-finance	10	mod-finance:4.8.0	2	1024	896	128	700	88	128
mod-erm-usage	13	mod-erm-usage:4.6.0	2	1024	896	128	768	88	128
mod-copycat	10	mod-copycat:1.5.0	2	1024	896	128	768	88	128
mod-lists	8	mod-lists:1.1.0-SNAPSHOT.1261	2	1024	896	128	768	88	128
mod-entities-links	15	mod-entities-links:2.0.4	2	2592	2480	400	1440	0	1024
mod-permissions	47	mod-permissions:6.5.0-SNAPSHOT.369	2	1684	1544	512	1024	384	512
pub-edge	9	pub-edge:2023.06.14	2	1024	896	128	768	0	0
mod-orders	10	mod-orders:12.7.1	2	2048	1440	1024	1024	384	512
edge-patron	10	edge-patron:5.0.0	2	1024	896	256	768	88	128
edge-ncip	11	edge-ncip:1.9.2	2	1024	896	128	768	88	128
mod-users-bl	10	mod-users-bl:7.6.0	2	1440	1152	512	922	88	128
mod-inventory-storage	15	mod-inventory-storage:27.0.4	2	4096	3690	2048	3076	384	512
mod-invoice	10	mod-invoice:5.7.2	2	1440	1152	512	922	88	128
mod-user-import	10	mod-user-import:3.8.0	2	1024	896	128	768	88	128
mod-sender	10	mod-sender:1.11.0	2	1024	896	128	768	88	128
edge-oai-pmh	8	edge-oai-pmh:2.7.1	2	1512	1360	1024	1440	384	512
mod-data-export-worker	10	mod-data-export-worker:3.1.2	2	3072	2800	1024	2048	384	512
mod-rtac	10	mod-rtac:3.5.0	2	1024	896	128	768	88	128
mod-circulation-storage	14	mod-circulation-storage:17.1.7	2	2880	2592	1536	1814	384	512
mod-calendar	10	mod-calendar:2.5.0	2	1024	896	128	768	88	128
mod-source-record-storage	18	mod-source-record-storage:5.7.5	2	5600	5000	2048	3500	384	512
mod-event-config	10	mod-event-config:2.6.0	2	1024	896	128	768	88	128
mod-courses	10	mod-courses:1.4.8	2	1024	896	128	768	88	128
mod-inventory	15	mod-inventory:20.1.8	2	2880	2592	1024	1814	384	512
mod-email	10	mod-email:1.16.0	2	1024	896	128	768	88	128
mod-di-converter-storage	18	mod-di-converter-storage:2.1.5	2	1024	896	128	768	88	128
mod-circulation	14	mod-circulation:24.0.11	2	2880	2592	1536	1814	384	512
mod-pubsub	13	mod-pubsub:2.11.3	2	1536	1440	1024	922	384	512
edge-orders	10	edge-orders:2.9.1	2	1024	896	128	768	88	128
edge-rtac	7	edge-rtac:2.6.2	2	1024	896	128	768	88	128
mod-template-engine	10	mod-template-engine:1.19.1	2	1024	896	128	768	88	128
mod-users	34	mod-users:19.3.0-SNAPSHOT.677	2	1024	896	128	768	88	128
mod-patron-blocks	10	mod-patron-blocks:1.9.0	2	1024	896	1024	768	88	128
edge-fqm	21	edge-fqm:1.0.1	2	1024	896	128	768	88	128
mod-audit	10	mod-audit:2.8.0	2	1024	896	128	768	88	128
mod-source-record-manager	17	mod-source-record-manager:3.7.8	2	5600	5000	2048	3500	384	512
nginx-edge	9	nginx-edge:2023.06.14	2	1024	896	128	0	0	0
mod-quick-marc	11	mod-quick-marc:5.0.1	1	2288	2176	128	1664	384	512
nginx-okapi	9	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0
okapi-b	11	okapi:5.1.2	3	1684	1440	1024	922	384	512
mod-feesfines	11	mod-feesfines:19.0.0	2	1024	896	128	768	88	128
mod-invoice-storage	10	mod-invoice-storage:5.7.0	2	1872	1536	1024	1024	384	512
mod-service-interaction	10	mod-service-interaction:3.0.2	2	2048	1844	256	1290	384	512
mod-data-export	12	mod-data-export:4.8.7	1	1024	896	1024	768	88	128
mod-patron	10	mod-patron:6.0.0	2	1024	896	128	768	88	128
mod-oai-pmh	5	mod-oai-pmh:3.12.8	2	4096	3690	2048	3076	384	512
edge-connexion	10	edge-connexion:1.1.0	2	1024	896	128	768	88	128
mod-kb-ebsco-java	10	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128
mod-notes	10	mod-notes:5.1.0	2	1024	896	128	952	384	512
mod-organizations-storage	10	mod-organizations-storage:4.6.0	2	1024	896	128	700	88	128
mod-data-export-spring	12	mod-data-export-spring:3.0.2	1	2048	1844	256	1536	384	512
mod-login	10	mod-login:7.10.1	2	1440	1298	1024	768	384	512
pub-okapi	9	pub-okapi:2023.06.14	2	1024	896	128	768	0	0

...


mod-eusage-reports	13	mod-eusage-reports:2.0.0	2	1024	896	128	768	88	128

Methodology/Approach

DI tests were started from UI with 5-minute pauses between the testsconcurrently with 1 job on each tenant, fs09000000 first and then on fs07000001 so in total two jobs on two tenants. Then 1 job on three tenants concurrently with several seconds delay - started with tenant fs09000000, second tenant -fs07000001 and third tenant - fs07000002.
DI Create jobs were conducted with 10k and 25k first. Then DI Update jobs.

Version	Old Version 8	New Version Current
Changes made by	Leonid Kolesnykov	Leonid Kolesnykov
Saved on	Mar 22, 2024	Mar 28, 2024

Versions Compared

Key

Table of Contents
Overview

Table of Contents
Overview

Summary

Test Runs

Test Results

DB Connections

DB load

Infrastructure

Infrastructure

Methodology/Approach

DI Create	# of test	Number of concurrent jobs	Central Main tenant (fs09000000)	Second tenant (fs07000001)	Third tenant
10K	Baseline	1	00:04:56
	1	2	00:10:43	00:10:37
	2	3	00:21:12	00:21:06	00:20:57 *
25k25K	Baseline	1	00:11:24
	3	2	00:23:44	00:23:30
	4	3	00:37:11	00:37:05	00:36:58
DI Update
10k10K	Baseline	1	00:06:32
	5	2	00:09:47	00:11:26
	6	3	00:19:08	00:19:06	00:18:31
25k25K	Baseline	1	00:15:13
	7	2	00:30:49	00:30:52
	8	3	00:47:47	00:48:17	00:47:54
DI Create (smoke Smoke test)
50k50K	9	1	00:22:31
50k50K	10	3	01:12:54	01:12:44	01:12:35

Name	Memory GIB	vCPUs	max_connections
db.r6g.xlarge	32 GiB	4 vCPUs	2731

Page Comparison

Versions Compared

Key

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-13">#ea5c5c}[data-colorid=ddlmqxa5jf]{color:#a31515} html[data-color-mode=dark] [data-colorid=ddlmqxa5jf]{color:#ea5c5c}</span> Table of ContentsOverview

Summary

Test Runs

Test Results

DB Connections

DB load

Infrastructure

Infrastructure

Methodology/Approach

Table of Contents
Overview