O

verview

This document contains the results of Central Ordering testing scenarios to create order with PO lines plus Change instance connection and Receive all pieces for Ramsons [ECS] environments.

PERF-1027 - Getting issue details... STATUS

Summary

All tests were successfully completed, and all API calls were executed without errors.
Response time for Change Instance Connection for Test №2 increased by 45% compared to Test №1 and for Test №3 Response time increased by 125% compared to Test 1. It appears that the Find or Create New Holdings action performed multiple operations, which explains the extended response time for this action.
In Test 3, the Create Order operation involved creating orders with 50 Order Lines, compared to 1 Order Lines in Test 1 and 10 Order Lines in Test 2. This explains the significant increase in response times for the Open Order and Unopen Order operations:
- Open Order: Response times increased nearly 10x in Test 3 compared to Test 1 due to the higher load (50 items per order).
- Unopen Order: Similarly, response times grew approximately 4-5x in Test 3.
For other operations (Create Order, Create Order Line, Receive Pieces, and Change Instance Connection), response times increased moderately but remained within acceptable ranges as the load increased.
Conclusion: The increase in item count per order directly impacted the performance of Open Order and Unopen Order, highlighting their sensitivity to load.
In Test №2, the Create New Holdings operation for Change Instance Connection triggered the reindex function. As a result, we observed that mod-search initiated reindexing, as evidenced by SQL queries N1 and N4 appearing in the TOP 5 SQL Statements table. This process placed a significant load on the database and continued for over 3 hours after the tests were completed.

Test Runs

Test №	Date	Test Conditions	Results
1	12/20/24, 13:15 AM UTC	View Order history, View Order Line history and View Pieces history with 10, 50, 100 and 1000 changes on Data Set with 500k records	Completed
2	12/20/24, 13:11 AM UTC	View Order history, View Order Line history and View Pieces history with 10, 50, 100 and 1000 changes on Data Set with 1 million records	Completed
3	12/20/24, 15:45 PM UTC	View Order history, View Order Line history and View Pieces history with 10, 50, 100 and 1000 changes on Data Set with 5 million records	Completed

Test Results

Test №1

PO with PO Lines (1) + locations (5) + Change instance connection (Move holdings)

	Average	pct90	Average	pct90	Average	pct90
Transactions	Response Time (milliseconds)
	Data size 500k		Data size 1 million		Data size 5 million
TC_open order 10	813	2513	872	2526	894	2650
TC_open order 50	446	687	453	687	469	647
TC_open order 100	737	1343	739	1346	814	1722
TC_open order 1000	670	884	657	858	680	838
TC_open order line 10	1363	3159	1468	3464	1560	4083
TC_open order line 50	913	1343	925	1591	936	1458
TC_open order line 100	1181	1784	1170	1978	1235	2242
TC_open order line 1000	1197	1534	1192	1485	1213	1529
TC_open pieces 10	1268	2643	1249	2621	1269	2645
TC_open pieces 50	880	1065	881	1254	915	1368
TC_open pieces 100	1111	1844	1126	1531	1144	1811
TC_open pieces 1000	1062	1286	1110	1305	1092	1489
TC_view order history 10	541	1652	558	1613	544	1388
TC_view order history 50	254	383	249	314	265	380
TC_view order history 100	477	805	486	886	505	849
TC_view order history 1000	452	647	445	528	464	681
TC_view order line history 10	754	1610	738	1564	823	1974
TC_view order line history 50	452	619	449	587	489	788
TC_view order line history 100	695	1068	677	1037	718	968
TC_view order line history 1000	733	900	760	1207	746	926
TC_view pieces history 10	753	1433	745	1422	782	1801
TC_view pieces history 50	452	701	449	632	454	626
TC_view pieces history 100	670	1008	656	1086	701	1081
TC_view pieces history 1000	710	945	701	1095	694	862

	Data size 500k	Data size 1 million	Data size 5 million	Data size 500k	Data size 1 million	Data size 5 million
Transactions	Response Time (milliseconds)
	Average	Average	Average	pct90	pct90	pct90
GET audit-data/acquisition/order/[order_10]	27	28	34	72	74	89
GET audit-data/acquisition/order-line/[po_line_id_10]	29	27	29	77	70	90
GET audit-data/acquisition/piece/]pieces_10]/status-change-history	28	28	27	64	68	69
GET audit-data/acquisition/order/[order_50]	18	16	18	58	47	48
GET audit-data/acquisition/order-line/[po_line_id_50]	20	23	40	53	60	344
GET audit-data/acquisition/piece/]pieces_50]/status-change-history	17	20	19	44	51	51
GET audit-data/acquisition/order/[order_100]	22	20	23	62	59	66
GET audit-data/acquisition/order-line/[po_line_id_100]	27	25	32	65	63	75
GET audit-data/acquisition/piece/]pieces_100]/status-change-history	22	21	23	61	62	60
GET audit-data/acquisition/order/[order_1000]	47	52	48	100	98	81
GET audit-data/acquisition/order-line/[po_line_id_1000]	122	126	132	212	231	222
GET audit-data/acquisition/piece/]pieces_1000]/status-change-history	60	50	53	117	105	96

table_name	table_size	indexes_size	row_count
cs00000int_mod_audit.acquisition_order_log	2811 MB	358 MB	5.300.543
cs00000int_mod_audit.acquisition_order_line_log	11 GB	366 MB	5.400.089
cs00000int_mod_audit.acquisition_piece_log	3957 MB	374 MB	5.220.071

Comparison

Test №1-№2-№3

The following table compares the results between tests №1, №2 and №3.

Open Order: Response times increased nearly 10x in Test 3 compared to Test 1 due to the higher load (50 items per order).
Unopen Order: Similarly, response times grew approximately 4-5x in Test 3.

Resource utilization for Test #1-2-3

Resource utilization table

CPU		RAM
mod-inventory-b	1.40%	mod-inventory-b	75%
mod-search-b	0.20%	mod-finance-b	68%
mod-inventory-storage-b	0.10%	mod-orders-b	55%
mod-orders-storage-b	0.10%	mod-orders-storage-b	44%
mod-orders-b	0.10%	mod-audit-b	39%
mod-audit-b	0.10%	mod-search-b	39%
mod-finance-b	0.10%	mod-finance-storage-b	32%
mod-finance-storage-b	0.10%	mod-inventory-storage-b	23%

Service CPU Utilization

Here we can see that mod-search initiated reindexing for Test #2.

Service Memory Utilization

Here we can see that all modules show a stable trend.

Kafka metrics

OpenSearch Data Nodes metrics

DB CPU Utilization

DB CPU in the average were 20% for Scenario 1, 30% for Scenario 2 and 40% for Scenario 3.

DB Connections

Max DB connections was 1560.

DB load

Top SQL-queries

Appendix

Infrastructure

PTF - environment Ramsons (rcon)

10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
1 instance of db.r6.xlarge database instance: Writer instance
OpenSearch
- domain: ptf-test
- Number of nodes: 7
- Version: OpenSearch_2_13_R20240520-P5
MSK - fse-tenant
- 4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

RCON modules memory and CPU parameters

rcon-pvt (Wed Dec 11 13:14:17 UTC 2024)

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	7	mod-remote-storage:3.3.2	2	4920	4472	3960	512	512
mod-finance-storage	7	mod-finance-storage:8.7.3	2	1024	896	700	88	128
mod-ncip	7	mod-ncip:1.15.6	2	1024	896	768	88	128
mod-agreements	7	mod-agreements:7.1.2	2	1592	1488	0	0	0
mod-ebsconet	7	mod-ebsconet:2.3.0	2	1248	1024	700	128	256
mod-organizations	7	mod-organizations:2.0.0	2	1024	896	700	88	128
mod-consortia	9	mod-consortia:1.2.1	2	5136	4776	2048	512	1024
edge-sip2	7	edge-sip2:3.3.1	2	1024	896	768	88	128
mod-serials-management	7	mod-serials-management:1.1.1	2	2480	2312	1792	384	512
mod-settings	7	mod-settings:1.1.0	2	1024	896	768	88	128
mod-data-import	7	mod-data-import:3.2.1	1	2048	1844	1292	384	512
mod-search	15	mod-search:4.0.2	2	2592	2480	1440	512	1024
edge-dematic	7	edge-dematic:2.3.1	1	1024	896	768	88	128
mod-inn-reach	4	mod-inn-reach:3.2.1-SNAPSHOT.102	2	3600	3240	2880	512	1024
mod-record-specifications	7	mod-record-specifications:1.0.2	2	1024	896	768	88	128
mod-tags	7	mod-tags:2.3.0	2	1024	896	768	88	128
mod-authtoken	9	mod-authtoken:2.16.1	2	1440	1152	922	88	128
edge-courses	7	edge-courses:1.5.0	2	1024	896	768	88	128
mod-notify	7	mod-notify:3.3.0	2	1024	896	768	88	128
mod-inventory-update	7	mod-inventory-update:4.0.0	2	1024	896	768	88	128
mod-configuration	7	mod-configuration:5.11.0	2	1024	896	768	88	128
mod-orders-storage	7	mod-orders-storage:13.8.3	2	1024	896	700	88	128
edge-caiasoft	7	edge-caiasoft:2.3.2	2	1024	896	768	88	128
mod-login-saml	7	mod-login-saml:2.9.3	2	1024	896	768	88	128
mod-erm-usage-harvester	7	mod-erm-usage-harvester:5.0.1	2	1024	896	768	88	128
mod-gobi	7	mod-gobi:2.9.0	2	1024	896	700	88	128
mod-licenses	7	mod-licenses:6.1.2	2	2480	2312	1792	384	512
mod-password-validator	7	mod-password-validator:3.3.0	2	1440	1298	768	384	512
edge-dcb	8	edge-dcb:1.2.1	2	1024	896	768	88	128
mod-bulk-operations	7	mod-bulk-operations:2.1.4	2	3072	2600	1536	384	512
mod-fqm-manager	9	mod-fqm-manager:3.0.3	2	3000	2600	768	88	128
mod-graphql	8	mod-graphql:1.13.0	2	1024	896	768	88	128
mod-finance	7	mod-finance:5.0.0	2	1024	896	700	88	128
mod-erm-usage	7	mod-erm-usage:5.0.0	2	2800	2550	1800	384	512
mod-batch-print	7	mod-batch-print:1.2.0	2	1024	896	768	88	128
mod-tlr	4	mod-tlr:1.0.0-SNAPSHOT.8	2	1024	896	768	88	128
mod-lists	11	mod-lists:3.0.3	2	6000	2600	768	88	128
mod-copycat	7	mod-copycat:1.7.0	2	1024	512	768	88	128
mod-entities-links	10	mod-entities-links:3.1.1	2	2592	2480	1440	0	1024
mod-permissions	11	mod-permissions:6.6.0	2	1684	1544	1024	384	512
pub-edge	7	pub-edge:2023.06.14	0	1024	896	768	0	0
mod-orders	7	mod-orders:12.9.2	2	2048	1740	1024	384	512
edge-patron	7	edge-patron:5.2.0	2	1024	896	768	88	128
mod-marc-migrations	15	mod-marc-migrations:1.0.1	2	1024	896	768	88	128
edge-ncip	8	edge-ncip:1.10.1	2	1024	896	768	88	128
edge-inn-reach	5	edge-inn-reach:3.3.0-SNAPSHOT.69	2	1024	896	768	88	128
mod-users-bl	7	mod-users-bl:7.9.3	2	1440	1152	922	88	128
mod-oa	4	mod-oa:2.1.0-SNAPSHOT.66	2	1024	896	768	88	128
mod-inventory-storage	8	mod-inventory-storage:28.0.2	2	4096	3690	3076	512	1024
mod-invoice	7	mod-invoice:5.9.1	2	1440	1152	922	88	128
mod-user-import	7	mod-user-import:3.9.0	2	1024	896	768	88	128
mod-sender	7	mod-sender:1.13.0	2	1024	896	768	88	128
edge-oai-pmh	7	edge-oai-pmh:2.10.0	2	1512	1360	1440	384	512
mod-data-export-worker	8	mod-data-export-worker:3.3.3	2	3072	2048	2048	384	512
mod-rtac	7	mod-rtac:3.7.0	2	1024	896	768	88	128
mod-circulation-storage	7	mod-circulation-storage:17.3.1	2	2880	2592	1814	384	512
mod-source-record-storage	12	mod-source-record-storage:5.9.3	2	5600	5000	3500	384	512
mod-calendar	7	mod-calendar:3.2.0	2	1024	896	768	88	128
mod-event-config	7	mod-event-config:2.8.0	2	1024	896	768	88	128
mod-courses	8	mod-courses:1.4.11	2	1024	896	768	88	128
mod-circulation-item	7	mod-circulation-item:1.1.0	2	1024	896	0	0	0
mod-inventory	7	mod-inventory:21.0.2	2	2880	2592	1814	384	512
mod-email	7	mod-email:1.18.0	2	2800	2550	1800	384	512
mod-requests-mediated	4	mod-requests-mediated:1.0.0-SNAPSHOT.4	2	1024	896	768	88	128
mod-circulation	7	mod-circulation:24.3.2	2	2880	2592	1814	384	512
mod-pubsub	7	mod-pubsub:2.15.2	2	1536	1440	922	384	512
mod-di-converter-storage	7	mod-di-converter-storage:2.3.1	2	1024	896	768	88	128
edge-rtac	7	edge-rtac:2.8.0	2	1024	896	768	88	128
edge-orders	7	edge-orders:3.1.0	2	1024	896	768	88	128
mod-users	7	mod-users:19.4.3	2	1024	896	768	88	128
mod-template-engine	7	mod-template-engine:1.21.0	2	1024	896	768	88	128
mod-patron-blocks	7	mod-patron-blocks:1.11.1	2	1024	896	768	88	128
mod-audit	7	mod-audit:2.10.1	2	1024	896	768	88	128
edge-fqm	8	edge-fqm:3.0.1	2	1024	896	768	88	128
mod-source-record-manager	7	mod-source-record-manager:3.9.3	2	5600	5000	3500	384	512
nginx-edge	7	nginx-edge:2023.06.14	0	1024	896	0	0	0
mod-quick-marc	7	mod-quick-marc:6.0.0	1	2288	2176	1664	384	512
nginx-okapi	7	nginx-okapi:2023.06.14	2	1024	896	0	0	0
okapi-b	7	okapi:6.1.0	2	1684	1440	922	384	512
mod-feesfines	7	mod-feesfines:19.2.1	2	1024	896	768	88	128
mod-invoice-storage	7	mod-invoice-storage:5.9.1	2	1872	1536	1024	384	512
mod-reading-room	7	mod-reading-room:1.0.0	2	1024	896	768	88	128
mod-dcb	7	mod-dcb:1.2.3	2	1024	896	768	88	128
mod-service-interaction	7	mod-service-interaction:4.1.1	2	2048	1844	1290	384	512
mod-patron	7	mod-patron:6.2.1	2	1024	896	768	88	128
mod-data-export	12	mod-data-export:5.1.2	1	2048	1844	0	0	0
mod-oai-pmh	7	mod-oai-pmh:3.14.3	2	4096	3690	3076	384	512
edge-connexion	7	edge-connexion:1.3.1	2	1024	896	768	88	128
mod-notes	7	mod-notes:6.0.0	2	1024	896	952	384	512
mod-kb-ebsco-java	8	mod-kb-ebsco-java:5.0.0	2	1024	896	768	88	128
mod-organizations-storage	7	mod-organizations-storage:4.8.1	2	1024	896	700	88	128
mod-data-export-spring	7	mod-data-export-spring:3.4.2	1	2048	1844	1536	384	512
mod-login	7	mod-login:7.12.1	2	1440	1298	768	384	512
pub-okapi	7	pub-okapi:2023.06.14	2	1024	896	768	0	0
edge-erm	5	edge-erm:1.3.0	2	1024	896	768	88	128
mod-eusage-reports	7	mod-eusage-reports:3.0.0	2	1024	896	768	88	128

Methodology/Approach

All tests scenarios were started from Load Generator i-066a4ed5e6ec777bc on Ramsons ECS (rcon) environment.

Parameters for Tests:

Number of users: 1
Test duration: 20min
Each test scenario was conducted with the following dataset sizes for mod_audit.acquisition_order_log, mod_audit.acquisition_order_line_log and mod_audit.acquisition_piece_log tables:
- 500k records
- 1 million records
- 5 million records
No pause between user action

Test scenario №1

Open Order tub
Open Order (with 10 changes in history)
View Order history
Open Order Line (with 10 changes in history)
View Order Line history
Open Pieces (with 10 changes in history)
View Pieces history

Test scenario №2

Open Order tub
Open Order (with 50 changes in history)
View Order history
Open Order Line (with 50 changes in history)
View Order Line history
Open Pieces (with 50 changes in history)
View Pieces history

Test scenario №3

Open Order tub
Open Order (with 100 changes in history)
View Order history
Open Order Line (with 100 changes in history)
View Order Line history
Open Pieces (with 100 changes in history)
View Pieces history

Test scenario №4

Open Order tub
Open Order (with 1000 changes in history)
View Order history
Open Order Line (with 1000 changes in history)
View Order Line history
Open Pieces (with 1000 changes in history)
View Pieces history

Attached files

Ramsons ECS (rcon) module configuration parameters:

JMeter script:

RCON [Ramsons] ECS - Acquisition Audit Log Performance testing

O