Table of Contents
Overview

...

Data import create holdings job durations increased significantly in Quesnelia release. 4 times longer with 10k file. Failed to complete with 80k file because it was stopped after 4 hours of test run with only 46 committed jobs (total for the test was 81).
Top CPU utilization: mod-inventory-b - 16%, nginx-okapi - 5%, mod-source-record-storage-b - 4%, mod-quick-marc-b - 7%. Such low resource utilization from modules side can be explained by DB queries huge average latency during INSERT and UPDATE processes which had lock on the same tuple.
Top memory consumption: mod-inventory-storage-b - 85%, mod-data-import-b - 52%, mod-source-record-storage-b - 45%, mod-source-record-manager-b - 43%. Growing trend was defined in tests set #1 for mod-inventory-storage-b - 85%
DI job duration for the same file size grew from test to test if to use the same instance HRID to create holdings
DI perform faster if to use files with 1 unique instance HRID for every 1000 records. DI duration corresponds to file size with such approach. Memory utilized without growing trend. CPU and RDS utilization increased because there are less locks in DB.

Recommendations & Jiras

Investigate growing trend for mod-inventory-storage in tests set #1 (using 1 instance HRID to create all Holdings)
Define high number of Holdings associated with one instance HRID that's still realistic
Consider limit the request /inventory/items-by-holdings-id with limit. Now limit=0.
Jira Legacy
server System Jira
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key MODINVSTOR-1229

Errors

error status for 32'd split job during 80k file importing- SNAPSHOT_UPDATE_ERROR

Log message: 
ERROR taImportKafkaHandler org.folio.inventory.dataimport.exceptions.CacheLoadingException: Error loading jobProfileSnapshot by id: 'aee287c2-0d40-4e8d-9879-4c1c54bcd819', status code: 503

...

Set 2 - Files used to test DI create Holdings had 1 unique instance HRID for every 1000 created Holdings (new approach)

TestFileDuration: PoppyDuration: Quesnelia set #1Difference absoluteDifference percentage11k00:00:3200:01:2200:00:50156%25k00:02:1400:08:0000:05:46258%310k00:04:3500:22:4000:18:05395%480k00:36:2504:13:0003:36:35595%

Test	File	Duration: Orchid (previous results)	Duration: Poppy (previous results)	Duration: Quesnelia [ECS] Set #1	Status and Errors Quesnelia [ECS] Set #1	Duration: Quesnelia [ECS] Set #2	Status and Errors Quesnelia [ECS] Set #2
1	1k	45s	32s	1 min 22 sec	Success	1 min 3 sec	Success
2	5k	7m 47s	2m 14s	8 min	Success	4 min 16 sec	Success
3	10k	19m 46s	4m 35s	22 min 40 sec	Success	8 min 59 sec	Success
4	80k	20m (error*)	36m 25s	4 hours 13 min	Stopped by user after 46 job COMMITTED from 81 - 56% finished 1 job status - ERROR, with error status - SNAPSHOT_UPDATE_ERROR (job number - 32, file_name = '1718290065265-80k_holdings_Create_32.mrc')	52 min 5 sec	Success

Compared with results in previous test report: Data Import Create MARC holdings records [Poppy]

Comparison

Table contains comparison between Quesnelia and Poppy

Set #1 (Compared Poppy and Quesnelia with different data sets)

UPDATE_ERROR

(job number - 32, file_name = '1718290065265-80k_holdings_Create_32.mrc')

52 min 5 sec

Success

Previous test report: Data Import Create MARC holdings records [Poppy]

Service CPU Utilization

Expand

title	CPU utilization, 1k, 5k, 10k, 80k

Set #1

Module	CPU (1k)	CPU (5k)	CPU (10k)	CPU (80k)
mod-inventory-b	20.23	24.03	20.5	16.34
mod-di-converter-storage-b	12.3	13.51	6.21	3.94
nginx-okapi	10.67	13.07	10.37	5.13
mod-inventory-storage-b	9.93	14.49	12.87	10.95
mod-quick-marc-b	8.24	7.34	7.95	6.82
mod-source-record-storage-b	7.99	8.42	6.47	3.79
mod-users-b	7.16	5.95	5.93	6.58
okapi-b	6.76	7.8	6.15	3.75
mod-pubsub-b	5.69	5.73	5.78	5.69
mod-data-import-b	5.15	1.27	1.6	1.23
mod-source-record-manager-b	3.7	4.55	4.01	3.06
mod-authtoken-b	3.69	1.26	1.46	0.99
mod-password-validator-b	2.36	2.34	2.34	3.33
mod-feesfines-b	2.34	2.12	2.2	2.16
mod-configuration-b	2.22	2.08	1.98	2.71
mod-permissions-b	1.66	0.8	1.11	0.94
mod-circulation-storage-b	0.61	0.61	0.61	0.68
mod-circulation-b	0.36	0.39	0.33	0.4
pub-okapi	0.19	0.13	0.13	0.13

Set #2

Module	CPU (1k)	CPU (5k)	CPU (10k)	CPU (80k)
mod-inventory-b	11.93	28.85	32.97	33.23
mod-quick-marc-b	7.19	8.1	8.19	7.64
mod-pubsub-b	6.2	6.56	6.72	6.55
mod-users-b	5.79	6.47	6.09	6.82
mod-configuration-b	3.12	3.21	3.47	3.4
mod-feesfines-b	2.39	2.47	2.55	2.34
mod-password-validator-b	2.26	2.4	3.27	2.41
mod-di-converter-storage-b	1.95	10.22	9.39	8.98
mod-source-record-storage-b	1.79	11.45	11.64	10.84
mod-source-record-manager-b	1.65	6.44	6.42	5.87
mod-data-import-b	1.36	1.48	1.83	1.48
okapi-b	1.28	13.81	14	14.93
mod-authtoken-b	1.02	1.24	1.6	1.6
mod-circulation-storage-b	0.7	0.71	0.71	0.73
nginx-okapi	0.59	20.99	20.39	22.89
mod-permissions-b	0.48	4.95	1.65	1.63
mod-circulation-b	0.35	0.36	0.35	0.36
mod-inventory-storage-b	0.33	14.24	14.54	13.96
pub-okapi	0.17	0.24	0.24	0.23

...

Name	Memory GIB	vCPUs	Engine version	Architecture settings
db.r6g.xlarge	32 GB	4 vCPUs	16.1	Non-multitenant architecture

...

Expand

title	Modules

16/06/2024

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	qcon-pvt
mod-inventory-b	3	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512
mod-quick-marc-b	1	mod-quick-marc:5.1.0	2	2288	2176	128	1664	384	512
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	128
mod-di-converter-storage-b	2	mod-di-converter-storage:2.2.2	2	1024	896	128	768	88	128
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-source-record-storage-b	1	mod-source-record-storage:5.8.0	2	5600	5000	2048	3500	384	512
mod-source-record-manager-b	1	mod-source-record-manager:3.8.0	2	5600	5000	2048	3500	384	512
mod-inventory-storage-b	2	mod-inventory-storage:27.2.0-SNAPSHOT.738	2	4096	3690	2048	3076	384	512
mod-pubsub-b	1	mod-pubsub:2.13.0	2	1536	1440	1024	922	384	922
mod-users-b	1	mod-users:19.3.1	2	1024	896	128	768	88	128
mod-data-import-b	1	mod-data-import:3.1.0	1	2048	1844	256	1292	384	512
mod-organizations-storage-b	1	mod-organizations-storage:4.7.0	2	1024	896	128	700	88	128
mod-notes-b	1	mod-notes:5.2.0	2	1024	896	128	952	384	512
mod-gobi-b	1	mod-gobi:2.8.0	2	1024	896	128	700	88	128
mod-permissions-b	2	mod-permissions:6.5.0	2	1684	1544	512	1024	384	512
mod-search-b	9	mod-search:3.3.0-SNAPSHOT.224	2	2592	2480	2048	1440	512	1024
mod-circulation-storage-b	1	mod-circulation-storage:17.2.0	2	2880	2592	1536	1814	384	512
mod-circulation-b	2	mod-circulation:24.2.1	2	2880	2592	1536	1814	384	512
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	128	768

...

Versions Compared

Old Version 14

New Version 15

Key

Table of Contents
Overview

Recommendations & Jiras

Errors

Comparison

Service CPU Utilization

Page Comparison

Versions Compared

Old Version 14

New Version 15

Key

Recommendations & Jiras

Errors

Comparison

Service CPU Utilization