Overview

In the scope of PERF-527 - Getting issue details... STATUS it's needed to run tests to answer questions:

- Typical KPIs:
  - Service CPU
  - Service Memory
  - DB CPU
  - DB Memory
- Response times
- Durations of long workflows

Summary

The current system can accommodate an average load only without Data Import. With data import, we will have 'HTTP 500 Internal Server Error. If the issue persists, please report it to EBSCO Connect.' for several workflows in the source-storage/records/{id}/formatted requests PERF-582 - Getting issue details... STATUS , and the general response time will be longer up to 2 times for all other workflows.
Service CPU usage did not exceed 44 % for all modules. We can observe spikes in CPU usage of DI modules at the beginning of the Data Import jobs. CPU usage of mod-inventory-storage grows up to 31% but decreased automatically to 6% (a possible reason is garbage collection process work). Without DI spikes average CPU usage was up to 13% for all other modules.
Service memory utilization was stable, and no memory leaks were suspected during tests.
Average instance CPU utilization was up to 8.5% and spiked at the beginning of Data Import up to 15%
Average DB CPU usage is up to 55%. For part of the test with DI job- 620 connections count. 395 connections - average usage during the test.

Recommendations & Jiras

Jiras

PERF-582 - Getting issue details... STATUS

Test Runs & Results

Test #	# configuration	Test duration	comments
1	FYR workflow started at the beginning of the test	10 hours	Res 500 ERROR during Data Import (DI) process
2	FYR workflow started at the end of the test	24 hours (planned - not performed yet)	Will be performed after DI issues fix

Test results from 1st test run:

Test # 1	Workflow name	Total time it takes to complete workflow										Time-consuming requests for each workflow during DI, finished with Response body: HTTP 500 Internal Server Error. If the issue persists, please report it to EBSCO Connect.
		1st hour of the test Avg + FYR	1st hour of the test 95th pct + FYR	2nd hour of the test Avg	2nd hour of the test 95th pct	10th hour of the test Avg	10th hour of the test 95th pct	Avg With DI (sec) normal load	95th pct with DI (sec) normal load	Avg no DI (sec) normal load	95th pct no DI (sec) normal load
1	CICO_Checkin	1.227	1.716	1.196	1.661	1.074	1.355	1.238	1.506	1.054	1.591
2	CICO_Checkout	1.978	2.694	1.952	2.635	1.767	2.168	2.156	2.829	1.650	1.948
3	IO_View invoices	0.945	1.221	0.932	1.142	0.904	1.012	0.907	1.305	0.763	0.913
4	IO_Create invoices	1.718	2.126	1.769	2.187	1.804	2.061	1.433	1.815	1.174	1.370
5	IO_Edit invoices	2.170	2.497	2.195	2.442	2.149	2.432	1.983	2.422	1.581	1.897
6	IO_Delete invoices	1.119	1.348	1.090	1.296	1.141	1.307	1.070	1.196	0.804	0.927
7	AIE_Approving Invoices	1.174	1.940	1.232	1.946	1.662	2.276	1.752	2.211	1.453	1.940
8	VAR_View Authority records	0.310	0.496	0.305	0.444	0.278	0.418	22.037	30.604	0.289	0.381	VAR_GET /source-storage/records/marc_id/formatted
9	VTT_View MARC tag table	1.422	2.392	1.473	2.370	1.039	1.369	41.272	61.935	0.987	1.284	VTT_GET source-storage/records/{id}/formatted *2
10	VH_View holdings records	1.733	2.507	1.626	2.296	1.361	1.616	27.328	33.579	1.526	1.922	VH_GET source-storage/records/{id}/formatted
11	VB_View Bib	1.179	2.011	1.177	1.881	0.850	1.129	22.851	31.634	0.841	1.168	VB_GET source-storage/records/{id}/formatted
12	PRO_View patron records	0.636	1.032	0.628	1.069	0.559	0.869	0.672	1.118	0.566	0.883
13	PRO_Delete patron records	0.703	1.181	0.844	1.113	0.716	1.049	0.892	1.336	0.638	1.070
14	PRO_Update patron records	1.163	1.915	1.011	1.703	1.091	1.692	1.386	2.097	1.043	1.625
15	PRO_Create patron records	1.210	1.482	1.192	1.482	1.189	1.399	1.547	1.979	1.098	1.261
16	LO_View Ledger	0.048	0.086	0.048	0.070	0.077	0.120	0.122	0.458	0.050	0.088
17	LO_Create ledger	0.635	0.799	0.631	0.744	0.710	0.848	0.684	0.840	0.616	0.761
18	LO_Edit ledger	0.050	0.087	0.050	0.081	0.079	0.128	0.076	0.094	0.054	0.085
19	LO_Delete a ledger	0.046	0.077	0.047	0.073	0.080	0.115	0.080	0.129	0.046	0.080
20	DE_Export bib "Default instances export job profile"	5 sec	-	4 sec	-	5 sec	-	11 sec (5000 records)	-	5 sec (5000 records)	-
21	DE_Export holdings "Default holdings export job profile"	44 sec	-	28 sec	-	27 sec	-	3 min 16 sec (5000 records)	-	26 sec (5000 records)	-
22	DE_Export authority records "Default authority export job profile"	4 sec	-	3 sec	-	3 sec	-	8 sec (5000 records)	-	3 sec (5000 records)	-
23	DI "DISC HRID match"	-	-	-	-	-	-	1sec (1 record)	-	-	-
24	DI "DS LA edeposit records update"	-	-	-	-	-	-	17 min 5 sec	-	-	-	job duration 17 min
25	DI "DISC New edeposit records"	-	-	-	-	-	-	4 sec	-	-	-
26	DI "DISC New NON edeposit records"	-	-	-	-	-	-	4 sec (5 records)	-	-	-
27	IRO_View item records	1.443	2.230	1.758	2.346	1.380	1.787	24.771	32.327	1.289	1.649
28	IRO_update item records	1.360	1.971	1.332	1.819	1.020	1.186	17.226	33.950	0.998	1.250
29	IRO_delete item records	1.443	2.230	1.519	2.346	1.033	1.376	23.686	31.497	0.927	1.099
30	MPS_Monitoring Pick Slips and Requests GET /circulation/requests	0.476	0.756	0.476	0.756	0.336	0.476	0.434	0.527	0.359	0.480
31	MPS_Monitoring Pick Slips and Requests GET /circulation/pick-slips/	0.181	0.484	0.208	0.558	0.167	0.417	0.106	0.312	0.112	0.256
32	MPS_Monitoring Pick Slips and Requests	0.050	0.050	0.050	0.050	0.030	0.030	0.297	0.297	0.303	0.303
33	ULR_Users loan renewal	1.622	1.954	1.696	2.278	1.514	1.765	1.899	2.395	1.467	1.661
34	ILR_Item-level requests	0.768	1.518	0.775	1.287	0.700	0.956	0.835	1.267	0.669	0.973
35	VRO_View vendor records	0.703	1.262	0.657	1.217	0.792	1.286	1.315	2.305	0.713	1.165
36	VRO_Edit vendor records	5.981	6.575	6.078	6.267	6.095	6.326	7.429	9.980	5.199	6.190
37	VRO_Create vendor records	1.135	1.337	1.127	1.387	1.115	1.307	1.436	1.829	1.064	1.200
38	VRO_Delete vendor records	0.459	0.670	0.463	0.670	0.426	0.598	0.635	0.885	0.412	0.522
39	POO_Create purchase orders	1.768	1.970	1.781	1.932	1.746	2.009	1.974	2.393	1.625	1.733
40	POO_View purchase orders	1.036	1.449	1.197	1.447	1.248	1.501	1.502	1.603	1.205	1.435
41	POO_Edit purchase orders	1.904	2.917	2.555	2.833	2.213	2.941	3.118	3.914	2.076	2.984
42	POO_Delete purchase orders	1.140	1.760	1.477	1.794	1.554	1.787	2.036	3.387	1.432	1.830
43	RIH_Retrieving instances and holdings	0.047	0.090	0.045	0.088	0.036	0.070	19.737	30.084	0.035	0.073	RIH_GET source-storage/source-records
44	ETT_Edit MARC tag table	4.078	5.171	4.075	5.409	3.121	3.613	118.391	128.508	3.424	4.257	ETT_GET /source-storage/records/instance_id/formatted ETT_GET /records-editor/records
45	FYR_Fiscal close - end of FY rollover		-	-	-	-	-	10 min 30 s	-	11 min	-
46	VIR_Blacklight: View an inventory record JMeter script	0.899	1.224	0.894	1.207	0.819	1.042	1.030	1.602	0.821	1.042
47	BLS_Blacklight: Create a Request JMeter script	1.335	1.696	1.269	1.696	1.189	1.501	1.336	1.605	1.122	1.404
48	PRV_Blacklight: Create a View Patron record JMeter script	0.077	0.126	0.077	0.125	0.071	0.114	0.106	0.136	0.073	0.110
49	VIH_View instance holdings details	3.093	4.175	2.956	3.792	1.534	1.761	21.241	31.698	1.456	1.572	VIH_GET /source-storage/records/instanceId/formatted

*Note that workflows that have response times or durations in red are the ones that are at least 2 times higher than when running without Data Import jobs.

Throughput graphs

Test#1

Memory Utilization

Memory utilization of the most memory-consuming modules :

mod-inventory - 98%
mod-orders - 73%
mod-calendar - 71%
mod-notes - 68%
mod-tags - 66%
mod-data-export-worker - 64%
mod-data-export - 62%
mod-invoice - 61%
mod-finance-storage- 51%
others - usage less than 50%

This graph represents memory usage and shows that no memory leak is suspected for all of the modules.

Service CPU Utilization

CPU usage did not exceed 44 % for all modules. We can observe spikes in CPU usage of DI modules at the beginning of the Data Import jobs. CPU usage of mod-inventory-storage grows up to 31% but decreased automatically to 6% (a possible reason is garbage collection process work). Without DI spikes average CPU usage was up to 13% for all other modules.

Test# 1 DI duration - 17 min.

Most CPU-consuming modules:

mod-inventory-storage - from 6% to 44%
mod-authtoken -13%
mod-inventory - 11%
mod-configuration -11%
mod-quick-marc - 11%
mod-finance-storage - 10%
others - usage less than 10%

Instance CPU Utilization

Average instance CPU utilization was up to 8.5% and spiked at the beginning of Data Import up to 15%

RDS CPU Utilization

Test# 1

Approximately DB CPU usage is up to 55%

RDS Database Connections

Test# 1 for part of the test with DI job- 620 connections count. 395 connections - average usage during the test.

Appendix

Infrastructure

Records count :

mod_source_record_storage.marc_records_lb = 7300919
mod_source_record_storage.raw_records_lb = 7300919
mod_source_record_storage.records_lb = 7300919
mod_source_record_storage.marc_indexers = 245032159 (all records)
mod_source_record_storage.marc_indexers with field_no 010 = 1008129
mod_source_record_storage.marc_indexers with field_no 035 = 8968420
mod_inventory_storage.authority = 852215
mod_inventory_storage.holdings_record = 6091403
mod_inventory_storage.instance = 5581816
mod_inventory_storage.item = 5705915

PTF -environment ncp3

9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, one reader, and one writer
Name API Name Memory GIB vCPUs max_connections
R6G Extra Large db.r6g.xlarge 32 GiB 4 vCPUs 2731
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Kafka topics partitioning: - 2 partitions for DI topics

Modules memory and CPU parameters

Modules	Version	Task Definition	Running Tasks	CPU	Memory	MemoryReservation	MaxMetaspaceSize	Xmx
mod-inventory-storage	26.0.0	1	2	1024	2208	1952	384	1440
mod-inventory	20.0.4	1	2	1024	2880	2592	512	1814
mod-tags	2.0.1	1	2	128	1024	896	128	768
mod-gobi	2.6.0	1	2	128	1024	896	128	700
mod-remote-storage	2.0.2	1	2	1024	4920	4472	512	3960
mod-invoice-storage	5.6.0	1	2	128	1024	896	128	700
edge-sip2	3.0.0	1	2	128	1024	896	128	768
mod-users-bl	7.5.0	1	2	512	1440	1152	128	922
edge-rtac	2.6.0	1	2	128	1024	896	128	768
mod-feesfines	18.2.1	1	2	128	1024	896	128	768
mod-rtac	3.5.0	1	2	128	1024	896	128	768
mod-erm-usage-harvester	4.3.0	1	2	128	1024	896	128	768
mod-search	2.0.1	1	2	400	2592	2480	1024	1440
mod-service-interaction	2.2.2	1	2	256	2048	1844	512	1290
edge-ncip	1.8.1	1	2	128	1024	896	128	768
mod-authtoken	2.13.0	1	2	512	1440	1152	128	922
mod-permissions	6.3.1	2	2	512	1684	1544	512	1024
mod-circulation-storage	16.0.0	1	2	1024	1536	1440	512	896
mod-ncip	1.13.1	1	2	128	1024	896	128	768
mod-pubsub	2.9.1	1	2	1024	1536	1440	512	922
edge-orders	2.8.1	1	2	1024	1536	1440	512	922
mod-circulation	23.5.4	1	2	1536	2880	2592	128	700
edge-caiasoft	2.0.0	1	2	128	1024	896	-	-
mod-data-export	4.7.1	1	1	1024	1024	896	128	768
mod-organizations-storage	4.5.1	1	2	128	1024	896	128	700
mod-source-record-storage	5.6.5	1	2	2048	5600	5000	512	3600
mod-copycat	1.4.0	1	2	896	1024	896	128	768
mod-bulk-operations	1.0.5	1	2	1024	3072	2600	512	1536
mod-quick-marc	3.0.0	1	1	128	2288	2176	512	1664
mod-audit	2.7.0	1	2	1024	1024	896	128	768
mod-oai-pmh	3.11.3	1	2	1024	2248	2000	512	1440
edge-connexion	1.0.6	1	2	128	1024	896	128	768
mod-kb-ebsco-java	3.13.0	1	2	128	1024	896	128	768
mod-patron	5.5.2	1	2	128	1024	896	128	768
mod-email	1.15.3	1	2	128	1024	896	128	768
mod-password-validator	3.0.0	1	2	128	1440	1298	512	768
mod-login	7.9.0	1	2	1024	1440	1298	512	768
mod-data-export-worker	3.0.12	1	2	1024	3072	2600	512	2048
mod-agreements	5.5.2	1	2	128	3096	2580	512	2048
edge-oai-pmh	2.6.1	1	2	1024	1512	1360	512	1440
mod-eusage-reports	1.3.0	1	2	128	1024	896	128	768
mod-orders-storage	13.5.0	1	2	512	1024	896	128	700
mod-notify	3.0.0	1	2	128	1024	896	128	768
mod-source-record-manager	3.6.2	1	2	2048	5600	5000	512	3600
mod-di-converter-storage	2.0.2	2	2	128	1024	896	128	768
mod-template-engine	1.18.0	1	2	128	1024	896	128	768
mod-user-import	3.7.2	1	2	128	1024	896	128	768
mod-finance-storage	8.4.1	1	2	128	1024	896	128	700
mod-users	19.1.1	1	2	128	1024	896	128	768
mod-sender	1.10.0	1	2	128	1024	896	128	768
mod-graphql	1.11.0	1	2	128	1024	896	128	768
mod-licenses	4.3.1	1	2	128	2480	2312	512	1792
mod-invoice-b	5.6.2	1	2	512	1440	1152	128	922
mod-event-config	2.5.0	1	2	128	1024	896	128	768
mod-calendar	2.4.2	1	2	128	1024	896	128	768
mod-erm-usage	4.5.2	1	2	128	1024	896	128	768
mod-patron-blocks	1.8.0	1	2	1024	1024	896	128	768
mod-data-import	2.7.1	1	1	256	2048	1844	512	1292
mod-ebsconet	2.0.0	1	2	128	1248	1024	256	700
edge-dematic	2.0.0	1	2	128	1024	896	-	-
mod-task-list	5.0.1	1	1	128	1024	896	128	768
mod-courses	1.4.7	1	2	128	1024	896	128	768
mod-inventory-update	3.0.1	1	2	128	1024	896	128	768
mod-login-saml	2.6.1	1	2	128	1024	896	128	768
mod-orders	12.6.6	1	2	1024	2048	1440 (Recommended to change to 1544)	512	1024
mod-configuration	5.9.1	1	2	128	1024	896	128	768
mod-organizations	1.7.0	1	2	128	1024	896	128	700
mod-notes	5.0.1	1	2	128	1024	896	128	322
mod-finance	4.7.1	1	2	128	1024	896	128	700
mod-data-export-spring	2.0.1	1	1	256	2048	1844	256	1292
edge-patron	4.11.0	1	2	256	1024	896	128	768
okapi	5.0.1	2	3	1024	1684	1440	512	922
nginx-okapi	2022.03.02	1	2	128	1024	896	-	-
pub-okapi	2022.03.02	1	2	128	1024	896	-	768

Methodology/Approach

To test Baseline for normal NLA library usage the JMeter scripts were used.

Tested with different DI delays:

From test start
1 min delay
20 min delay
without DI

Data was gathered from 2 periods with and without data import.

DI - data import
FYR - Fiscal close - end of FY rollover

Folio Development Teams

Master Script longevity test - NLA report