Overview

The purpose of this test is to define system behavior under stress load. Define (if exists) any anomalies, memory leaks, errors.

Summary

Few test sets was performed:
- 2,3,6,8 threads tests with xlarge (default) DB;
- 2,3,6,8 threads tests with 2xlarge DB;
- test with longer (4 hr) duration for xlarge DB with 8 threads
- 3 threads test with 4xlarge DB.
During mos of a test we've observed that DB CPU usage reached 100% almost from the beginning of a test. For 2xlarge database size CPU usage decreased from 99,4% to 86.9% for 2 threads.
There is performance degradation with increasing load (from 2 threads to 3,6,8). See corresponding charts in Results and ComparisonS section
With increasing DB to 2xlarge performance degradation is not so rapid as it's for standard xlarge DB. However degradation pattern is the same for both test sets
For 90% of the workflows, degradation is above 50% with stress load compared to normal load. The top 10 most affected are (with response time degradation in %):
1. ILR_TC: Get ItemId - 709%
2. VIR_TC: inventory record monitoring - 579%
3. VTT_TC: Inventory View Instance Source - 531%
4. VRO_TC: View Vendor - 420%
5. LO_TC: View Ledgers - 335%
6. VB_TC: Inventory Choose Instance - 326%
7. VH_TC: Inventory View Holdings - 322%
8. VH_TC: Inventory Choose Holdings -308%
9. ETT_TC: Edit Bib RecordTagTable Group - 304%
10. VRO_TC: Edit Vendor - 289%

* - Stress load of 2 threads is 5-10 times (it depends on workflow) bigger than the average load

Recommendations & Jiras (Optional)

PERF-526 - NLA stress test

It is recommended to increase database size to 2x large: General response time for all workflows is about 24% improved for 2xlarge database size compared to xlarge (for stress test) and improved about 23% for a normal load test. Tested with default services configuration and without Data Import.

Changes in configurations for services do not improve response time for workflows but for mod-circulation, it should be revised due to configuration issues found in the task definition: excessive parameters
"-XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m"

{
                    "name": "JAVA_OPTS",
                    "value": "-Dport=8082 -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-circulation.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=512m -Xmx1814m"
                },

If release versions for mod-source-record-storage will be 5.6.5 and for mod-source-record-manager - 3.6.2 then changes of parameters DB_MAXPOOLSIZE for both modules, and DB_CONNECTION_TIMEOUT for mod-source-record-storage needed to avoid PERF-582 - Response 500 for NLA normal load test with Data Import job (2 SP) BLOCKED :

{
    "name": "DB_MAXPOOLSIZE",
    "value": "500"
},
{
    "name": "DB_CONNECTION_TIMEOUT",
    "value": "60"
},

Test Runs

Test #	Test Conditions	Duration	Load generator size (recommended)	Load generator Memory(GiB) (recommended)	Notes (Optional)
1.	2 threads Master script	60 mins	t3.xlarge	14	xlarge DB
2.	3 threads Master script	60 mins	t3.xlarge	14	xlarge DB
3	6 threads Master script	60 mins	t3.xlarge	14	xlarge DB
4	8 threads Master script	60 mins	t3.xlarge	14	xlarge DB
5	2 threads Master script	60 mins	t3.xlarge	14	2xlarge DB
6	3 threads Master script	60 mins	t3.xlarge	14	2xlarge DB
7	6 threads Master script	60 mins	t3.xlarge	14	2xlarge DB
8	8 threads Master script	60 mins	t3.xlarge	14	2xlarge DB
9	8 threads Master script	4 hr	t3.xlarge	14	xlarge DB
10	3 threads Master script	60 mins	t3.xlarge	14	4xlarge DB

Results and Comparisons

Performance dependency with increasing load with xlarge DB

Workflow	2 threads response times	3 threads response times	delta 2 vs 3 threads	6 threads response times	delta 3 vs 6 threads	8 threads response times	delta 6 vs 8 threads
AIE_TC: View Invoices Workflow	7,191	6,991	-0,2	11,612	4,621	14,218	2,606
BLS_TC: Blacklight simulation	3,932	5,073	1,141	7,276	2,203	9,347	2,071
ETT_TC: Edit Bib RecordTagTable Group	17,057	40,933	23,876	58,72	17,787	97,45	38,73
FYR_TC: Load Finance Ledgers Rollover	0,39	0,522	0,132	0,536	0,014	0,773	0,237
ILR_TC: Create ILR	1,482	2,414	0,932	2,446	0,032	3,072	0,626
ILR_TC: Get ItemId	1,731	4,232	2,501	5,574	1,342	9,698	4,124
ILR_TC: Get RequesterId	0,045	0,042	-0,003	0,058	0,016	0,062	0,004
IO_TC: Create Invoices Workflow	3,745	3,754	0,009	5,672	1,918	7,375	1,703
IO_TC: Delete Invoices Workflow	3,158	3,151	-0,007	5,2	2,049	6,682	1,482
IO_TC: Edit Invoices Workflow	5,705	5,723	0,018	9,602	3,879	12,208	2,606
IO_TC: View Invoices Workflow	2,53	2,54	0,01	4,07	1,53	5,067	0,997
IRO_TC: Delete Item	6,702	15,339	8,637	19,647	4,308	32,794	13,147
IRO_TC: Edit Item	6,69	15,567	8,877	18,724	3,157	29,134	10,41
IRO_TC: View Item	11,602	31,375	19,773	39,99	8,615	62,725	22,735
LO_TC: Create Ledgers	1,035	1,023	-0,012	1,381	0,358	1,476	0,095
LO_TC: Delete Ledgers	0,426	0,503	0,077	0,785	0,282	0,974	0,189
LO_TC: Edit Ledgers	0,43	0,497	0,067	0,794	0,297	0,976	0,182
LO_TC: View Ledgers	0,431	0,498	0,067	0,78	0,282	0,99	0,21
POO_TC Create Order	3,047	2,779	-0,268	3,668	0,889	4,369	0,701
POO_TC Delete Order	2,252	2,182	-0,07	3,034	0,852	4,458	1,424
POO_TC Edit Order	3,375	3,365	-0,01	4,524	1,159	6,428	1,904
POO_TC View Order	1,987	2,03	0,043	2,658	0,628	3,915	1,257
PRO_TC: Create User	2,257	2,356	0,099	2,84	0,484	3,009	0,169
PRO_TC: Delete User	1,202	1,296	0,094	1,51	0,214	1,713	0,203
PRO_TC: Edit User	2,447	2,613	0,166	3,316	0,703	3,664	0,348
PRO_TC: View User	1,207	1,242	0,035	1,529	0,287	1,69	0,161
PRV_TC: View Patron record Group	0,128	0,127	-0,001	0,16	0,033	0,184	0,024
VAR_TC: GoTo Marc Authority	0,084	0,139	0,055	0,123	-0,016	0,154	0,031
VAR_TC: Search Marc Authority	0,086	0,08	-0,006	0,089	0,009	0,096	0,007
VB_TC: Inventory Choose Instance	3,587	8,22	4,633	11,939	3,719	19,322	7,383
VB_TC: Inventory Search Instance	0,047	0,05	0,003	0,052	0,002	0,055	0,003
VB_TC: Load Inventory	1,238	2,446	1,208	2,108	-0,338	2,577	0,469
VH_TC: Inventory Choose Holdings	3,218	7,653	4,435	10,726	3,073	18,146	7,42
VH_TC: Inventory Search Holdings	0,026	0,026	0	0,029	0,003	0,032	0,003
VH_TC: Inventory View Holdings	3,846	9,764	5,918	14,365	4,601	24,426	10,061
VH_TC: Load Inventory	1,296	2,563	1,267	2,141	-0,422	2,208	0,067
VIH_TC: View instance holdings details	6,055	12,943	6,888	14,968	2,025	22,904	7,936
VIR_TC: inventory record monitoring	8,112	13,172	5,06	22,711	9,539	33,066	10,355
VRO_TC: Create Vendor	4,864	4,722	-0,142	7,992	3,27	10,481	2,489
VRO_TC: Delete Vendor	1,049	0,958	-0,091	1,111	0,153	1,434	0,323
VRO_TC: Edit Vendor	28,653	29,276	0,623	54,427	25,151	66,964	12,537
VRO_TC: View Vendor	3,932	3,889	-0,043	5,63	1,741	6,748	1,118
VTT_TC: Inventory Choose Instance	3,087	7,201	4,114	9,968	2,767	16,284	6,316
VTT_TC: Inventory Search Instance	0,042	0,044	0,002	0,051	0,007	0,051	0
VTT_TC: Inventory View Instance Source	2,463	6,024	3,561	8,642	2,618	15,141	6,499
VTT_TC: Load Inventory	1,235	2,414	1,179	2,188	-0,226	2,607	0,419
Check-In Controller	4,012	8,042	4,03	9,428	1,386	14,313	4,885
Check-Out Controller	5,359	9,719	4,36	11,119	1,4	15,886	4,767

Performance dependency with increasing load with 2xlarge DB

Workflow	2 threads response times	3 threads response times	delta 2 vs 3 threads	6 threads response times	delta 3 vs 6 threads	8 threads response times	delta 6 vs 8 threads
AIE_TC: View Invoices Workflow	3,422	3,896	0,474	5,066	1,17	6,6	1,534
BLS_TC: Blacklight simulation	2,583	3,148	0,565	3,978	0,83	5,151	1,173
ETT_TC: Edit Bib RecordTagTable Group	22,16	36,338	14,178	60,319	23,981	99,734	39,415
FYR_TC: Load Finance Ledgers Rollover	0,401	0,552	0,151	0,573	0,021		0
FYR_TC:FYR_Rollover Ledgers	3126,796	3347,388	220,592	-	-	-	-
ILR_TC: Create ILR	1,658	2,115	0,457	2,57	0,455	3,386	0,816
ILR_TC: Get ItemId	2,267	3,93	1,663	6,418	2,488	11,449	5,031
ILR_TC: Get RequesterId	0,046	0,053	0,007	0,058	0,005	0,068	0,01
IO_TC: Create Invoices Workflow	2,608	2,718	0,11	3,389	0,671	4,169	0,78
IO_TC: Delete Invoices Workflow	1,728	1,919	0,191	2,583	0,664	3,167	0,584
IO_TC: Edit Invoices Workflow	3,494	3,72	0,226	4,943	1,223	6,163	1,22
IO_TC: View Invoices Workflow	1,533	1,631	0,098	2,174	0,543	2,61	0,436
IRO_TC: Delete Item	8,487	13,132	4,645	20,902	7,77	34,156	13,254
IRO_TC: Edit Item	8,972	14	5,028	22,802	8,802	32,226	9,424
IRO_TC: View Item	15,542	24,431	8,889	37,73	13,299	63,621	25,891
LO_TC: Create Ledgers	0,83	0,911	0,081	1,146	0,235	1,221	0,075
LO_TC: Delete Ledgers	0,196	0,246	0,05	0,37	0,124	0,48	0,11
LO_TC: Edit Ledgers	0,191	0,247	0,056	0,376	0,129	0,478	0,102
LO_TC: View Ledgers	0,195	0,245	0,05	0,37	0,125	0,474	0,104
POO_TC Create Order	2,773	2,676	-0,097	3,101	0,425	3,248	0,147
POO_TC Delete Order	1,994	2,106	0,112	2,518	0,412	2,687	0,169
POO_TC Edit Order	3,135	3,114	-0,021	3,935	0,821	4,233	0,298
POO_TC View Order	1,791	1,835	0,044	2,229	0,394	2,305	0,076
PRO_TC: Create User	2,193	2,449	0,256	2,71	0,261	3,133	0,423
PRO_TC: Delete User	1,119	1,255	0,136	1,394	0,139	1,632	0,238
PRO_TC: Edit User	2,073	2,207	0,134	2,515	0,308	2,926	0,411
PRO_TC: View User	1,108	1,263	0,155	1,364	0,101	1,594	0,23
PRV_TC: View Patron record Group	0,099	0,116	0,017	0,138	0,022	0,164	0,026
VAR_TC: GoTo Marc Authority	0,121	0,149	0,028	0,178	0,029	0,225	0,047
VAR_TC: Search Marc Authority	0,1	0,099	-0,001	0,117	0,018	0,109	-0,008
VB_TC: Inventory Choose Instance	4,576	7,04	2,464	12,04	5	21,352	9,312
VB_TC: Inventory Search Instance	0,053	0,052	-0,001	0,06	0,008	0,057	-0,003
VB_TC: Load Inventory	1,716	2,248	0,532	2,607	0,359	3,38	0,773
VH_TC: Inventory Choose Holdings	3,849	6,654	2,805	10,849	4,195	19,004	8,155
VH_TC: Inventory Search Holdings	0,032	0,032	0	0,038	0,006	0,037	-0,001
VH_TC: Inventory View Holdings	5,088	9,144	4,056	14,993	5,849	25,389	10,396
VH_TC: Load Inventory	1,826	2,407	0,581	2,907	0,5	3,838	0,931
VIH_TC: View instance holdings details	7,013	10,722	3,709	16,549	5,827	26,942	10,393
VIR_TC: inventory record monitoring	5,186	8,122	2,936	12,979	4,857	21,812	8,833
VRO_TC: Create Vendor	1,734	2,009	0,275	2,765	0,756	3,265	0,5
VRO_TC: Delete Vendor	0,718	0,79	0,072	0,956	0,166	1,043	0,087
VRO_TC: Edit Vendor	8,098	9,579	1,481	13,871	4,292	18,568	4,697
VRO_TC: View Vendor	1,251	1,423	0,172	2,066	0,643	2,368	0,302
VTT_TC: Inventory Choose Instance	3,786	6,042	2,256	10,044	4,002	17,29	7,246
VTT_TC: Inventory Search Instance	0,047	0,046	-0,001	0,053	0,007	0,052	-0,001
VTT_TC: Inventory View Instance Source	3,022	5,349	2,327	8,925	3,576	15,993	7,068
VTT_TC: Load Inventory	1,723	2,23	0,507	2,511	0,281	3,283	0,772
Check-In Controller	4,891	6,845	1,954	12,37	5,525	17,733	5,363
Check-Out Controller	6,22	9,341	3,121	13,284	3,943	21,964	8,68

Comparison xlarge tests with 2xlarge tests

Workflow	2 threads response times (xlarge)	2 threads response times 2xlarge	delta	3 threads response times (xlarge)	3 threads response times 2xlarge	delta	6 threads response times (xlarge)	6 threads response times (2xlarge)	delta	8 threads response times (xlarge)	8 threads response times (2large)	delta
AIE_TC: View Invoices Workflow	7,191	3,422	-3,769	6,991	3,896	-3,095	11,612	5,066	-6,546	14,218	6,6	-7,618
BLS_TC: Blacklight simulation	3,932	2,583	-1,349	5,073	3,148	-1,925	7,276	3,978	-3,298	9,347	5,151	-4,196
ETT_TC: Edit Bib RecordTagTable Group	17,057	22,16	5,103	40,933	36,338	-4,595	58,72	60,319	1,599	97,45	99,734	2,284
FYR_TC: Load Finance Ledgers Rollover	0,39	0,401	0,011	0,522	0,552	0,03	0,536	0,573	0,037	0,773	0,784	0,011
FYR_TC:FYR_Rollover Ledgers	-	3126,796	3126,796	-	3347,388	3347,388	-	-	-			0
ILR_TC: Create ILR	1,482	1,658	0,176	2,414	2,115	-0,299	2,446	2,57	0,124	3,072	3,386	0,314
ILR_TC: Get ItemId	1,731	2,267	0,536	4,232	3,93	-0,302	5,574	6,418	0,844	9,698	11,449	1,751
ILR_TC: Get RequesterId	0,045	0,046	0,001	0,042	0,053	0,011	0,058	0,058	0	0,062	0,068	0,006
IO_TC: Create Invoices Workflow	3,745	2,608	-1,137	3,754	2,718	-1,036	5,672	3,389	-2,283	7,375	4,169	-3,206
IO_TC: Delete Invoices Workflow	3,158	1,728	-1,43	3,151	1,919	-1,232	5,2	2,583	-2,617	6,682	3,167	-3,515
IO_TC: Edit Invoices Workflow	5,705	3,494	-2,211	5,723	3,72	-2,003	9,602	4,943	-4,659	12,208	6,163	-6,045
IO_TC: View Invoices Workflow	2,53	1,533	-0,997	2,54	1,631	-0,909	4,07	2,174	-1,896	5,067	2,61	-2,457
IRO_TC: Delete Item	6,702	8,487	1,785	15,339	13,132	-2,207	19,647	20,902	1,255	32,794	34,156	1,362
IRO_TC: Edit Item	6,69	8,972	2,282	15,567	14	-1,567	18,724	22,802	4,078	29,134	32,226	3,092
IRO_TC: View Item	11,602	15,542	3,94	31,375	24,431	-6,944	39,99	37,73	-2,26	62,725	63,621	0,896
LO_TC: Create Ledgers	1,035	0,83	-0,205	1,023	0,911	-0,112	1,381	1,146	-0,235	1,476	1,221	-0,255
LO_TC: Delete Ledgers	0,426	0,196	-0,23	0,503	0,246	-0,257	0,785	0,37	-0,415	0,974	0,48	-0,494
LO_TC: Edit Ledgers	0,43	0,191	-0,239	0,497	0,247	-0,25	0,794	0,376	-0,418	0,976	0,478	-0,498
LO_TC: View Ledgers	0,431	0,195	-0,236	0,498	0,245	-0,253	0,78	0,37	-0,41	0,99	0,474	-0,516
POO_TC Create Order	3,047	2,773	-0,274	2,779	2,676	-0,103	3,668	3,101	-0,567	4,369	3,248	-1,121
POO_TC Delete Order	2,252	1,994	-0,258	2,182	2,106	-0,076	3,034	2,518	-0,516	4,458	2,687	-1,771
POO_TC Edit Order	3,375	3,135	-0,24	3,365	3,114	-0,251	4,524	3,935	-0,589	6,428	4,233	-2,195
POO_TC View Order	1,987	1,791	-0,196	2,03	1,835	-0,195	2,658	2,229	-0,429	3,915	2,305	-1,61
PRO_TC: Create User	2,257	2,193	-0,064	2,356	2,449	0,093	2,84	2,71	-0,13	3,009	3,133	0,124
PRO_TC: Delete User	1,202	1,119	-0,083	1,296	1,255	-0,041	1,51	1,394	-0,116	1,713	1,632	-0,081
PRO_TC: Edit User	2,447	2,073	-0,374	2,613	2,207	-0,406	3,316	2,515	-0,801	3,664	2,926	-0,738
PRO_TC: View User	1,207	1,108	-0,099	1,242	1,263	0,021	1,529	1,364	-0,165	1,69	1,594	-0,096
PRV_TC: View Patron record Group	0,128	0,099	-0,029	0,127	0,116	-0,011	0,16	0,138	-0,022	0,184	0,164	-0,02
VAR_TC: GoTo Marc Authority	0,084	0,121	0,037	0,139	0,149	0,01	0,123	0,178	0,055	0,154	0,225	0,071
VAR_TC: Search Marc Authority	0,086	0,1	0,014	0,08	0,099	0,019	0,089	0,117	0,028	0,096	0,109	0,013
VB_TC: Inventory Choose Instance	3,587	4,576	0,989	8,22	7,04	-1,18	11,939	12,04	0,101	19,322	21,352	2,03
VB_TC: Inventory Search Instance	0,047	0,053	0,006	0,05	0,052	0,002	0,052	0,06	0,008	0,055	0,057	0,002
VB_TC: Load Inventory	1,238	1,716	0,478	2,446	2,248	-0,198	2,108	2,607	0,499	2,577	3,38	0,803
VH_TC: Inventory Choose Holdings	3,218	3,849	0,631	7,653	6,654	-0,999	10,726	10,849	0,123	18,146	19,004	0,858
VH_TC: Inventory Search Holdings	0,026	0,032	0,006	0,026	0,032	0,006	0,029	0,038	0,009	0,032	0,037	0,005
VH_TC: Inventory View Holdings	3,846	5,088	1,242	9,764	9,144	-0,62	14,365	14,993	0,628	24,426	25,389	0,963
VH_TC: Load Inventory	1,296	1,826	0,53	2,563	2,407	-0,156	2,141	2,907	0,766	2,208	3,838	1,63
VIH_TC: View instance holdings details	6,055	7,013	0,958	12,943	10,722	-2,221	14,968	16,549	1,581	22,904	26,942	4,038
VIR_TC: inventory record monitoring	8,112	5,186	-2,926	13,172	8,122	-5,05	22,711	12,979	-9,732	33,066	21,812	-11,254
VRO_TC: Create Vendor	4,864	1,734	-3,13	4,722	2,009	-2,713	7,992	2,765	-5,227	10,481	3,265	-7,216
VRO_TC: Delete Vendor	1,049	0,718	-0,331	0,958	0,79	-0,168	1,111	0,956	-0,155	1,434	1,043	-0,391
VRO_TC: Edit Vendor	28,653	8,098	-20,555	29,276	9,579	-19,697	54,427	13,871	-40,556	66,964	18,568	-48,396
VRO_TC: View Vendor	3,932	1,251	-2,681	3,889	1,423	-2,466	5,63	2,066	-3,564	6,748	2,368	-4,38
VTT_TC: Inventory Choose Instance	3,087	3,786	0,699	7,201	6,042	-1,159	9,968	10,044	0,076	16,284	17,29	1,006
VTT_TC: Inventory Search Instance	0,042	0,047	0,005	0,044	0,046	0,002	0,051	0,053	0,002	0,051	0,052	0,001
VTT_TC: Inventory View Instance Source	2,463	3,022	0,559	6,024	5,349	-0,675	8,642	8,925	0,283	15,141	15,993	0,852
VTT_TC: Load Inventory	1,235	1,723	0,488	2,414	2,23	-0,184	2,188	2,511	0,323	2,607	3,283	0,676
Check-In Controller	4,012	4,891	0,879	8,042	6,845	-1,197	9,428	12,37	2,942	14,313	17,733	3,42
Check-Out Controller	5,359	6,22	0,861	9,719	9,341	-0,378	11,119	13,284	2,165	15,886	21,964	6,078

Comparison to separate workflows stress tests run

* - with xlarge DB

CICO: Check-In/Check-Out

Workflow

2 threads response times

master script

2 threads response times separate workflow

delta

3 threads response times

master script

3 threads response times separate workflow

delta

6 threads response times

master script

6 threads response times separate workflow

delta

8 threads response times

master script

8 threads response times separate workflow

delta

CICO_TC: Check-In Controller

4,012

0.653

-3.359

8,042

0.610

-7.432

9,428

0.594

-8.834

14,313

0.580

-13.733

CICO_TC: Check-Out Controller

5,359

1.124

-4.235

9,719

1.069

-8.650

11,119

1.034

-10.085

15,886

1.022

-14.864

IRO: Item record operation

Workflow

2 threads response times

master script

2 threads response times separate workflow

delta

3 threads response times

master script

3 threads response times separate workflow

delta

6 threads response times

master script

6 threads response times separate workflow

delta

8 threads response times

master script

8 threads response times separate workflow

delta

IRO_TC: Delete Item

6,702

0.916

-5.786

15,339

0.987

-14.352

19,647

0.760

-18.887

32,794

0.790

-32.004

IRO_TC: Edit Item

6,69

0.923

-5.767

15,567

0.990

-14.577

18,724

0.764

-17.960

29,134

0.787

-28.347

IRO_TC: View Item

11,602

1.218

-10.384

31,375

1.321

-30.054

39,99

1.035

-38.955

62,725

1.063

-61.662

mod-inventory and mod-inventory-storage are the most loaded by this workflow.

ETT: Edit Bib RecordTagTable

Workflow

2 threads response times

master script

2 threads response times separate workflow

delta

3 threads response times

master script

3 threads response times separate workflow

delta

6 threads response times

master script

6 threads response times separate workflow

delta

8 threads response times

master script

8 threads response times separate workflow

delta

ETT_TC: Edit Bib RecordTagTable Group

17,057

2.201

-14.856

40,933

1.996

-38.937

58,72

2.441

-56.279

97,45

3.091

-94.359

mod-quick-marc and mod-inventory-storage are the most loaded by this workflow.

Xlarge DB stress test

Memory Utilization

No memory leaks were defined. All of a modules has jumps in memory however those are not significant.

One drop of memory usage can be seen on a screen - it's mod-source-record-manager and mod-source-record-storage. this drop happened due to redeploy to newer versions.

CPU Utilization

most loaded modules:

nginx-okapi - spikes up to 373%
mod-configuration - spikes up to 272%
mod-users - spikes to 250%
mod-inventory-storage up to 85%
okapi - up to 85%
mod-finance-storage up to 85%
mod-finance up to 75%

RDS CPU Utilization

2Xlarge DB stress test

Memory Utilization

CPU Utilization

most loaded modules:

nginx-okapi - spikes up to 500%
mod-users - spikes to 450%
mod-configuration - spikes up to 180%
mod-finance-storage up to 150%
mod-finance up to 120%
mod-inventory-storage up to 97%
okapi - up to 90%

RDS CPU Utilization

4 Hours stress test (xlarge DB)

During previous two testing sets we've found that RDS CPU reached ±100% almost from the beginning of a test. So the purpose of this test is to check if system can handle such high load during longer duration without breaking of DB or other components of a system.

This time DB CPU usage reached 100% from the beginning of a test and stays at this point up to the end. System didn't fail and was stable after test.

Memory Utilization

CPU Utilization

most loaded modules:

nginx-okapi - spikes up to 401%
mod-users - spikes to 230%
mod-configuration - spikes up to 120%
mod-finance-storage up to 60%
mod-finance up to 60%
mod-inventory-storage up to 88%
okapi - up to 77%

RDS CPU Utilization

4Xlarge DB stress test

Memory Utilization

CPU Utilization

most loaded modules:

nginx-okapi - spikes up to 401%
mod-users - spikes to 326%
mod-configuration - spikes up to 116%
mod-finance-storage up to 165%
mod-finance up to 111%
mod-inventory-storage up to 78%
okapi - up to 77%

RDS CPU Utilization

Database analysis

AAS or an active session is a database connection that has submitted a request to the database but has not yet received the response. Measuring the average number of active concurrent sessions over time provides a clear picture of the load on the database. https://aws.amazon.com/blogs/database/analyzing-amazon-rds-database-workload-with-performance-insights/

Slow queries:

a lot of (this queries taking more than 1-2 seconds to execute)

SELECT COUNT(*) FROM fs00001069_mod_inventory_storage Open question - which workflow is calling that taking

long one

2023-07-06 23:19:28 UTC:10.23.10.189(47150):fs00001069_mod_finance_storage@folio:[16764]:LOG: duration: 1808067.854 ms statement: DO
$$
begin
PERFORM fs00001069_mod_finance_storage.budget_encumbrances_rollover('
{ "id": "f2803540-7d7c-4b2a-8068-40dcd0725d16", "ledgerId": "cb8ba034-a78c-46f6-b41c-c0faec101c39", "rolloverType": "Preview", "fromFiscalYearId": "2284fa2a-926c-49bb-b949-0a791fd8353b", "toFiscalYearId": "bad9ec7d-186c-48c4-a64d-a23676044eab", "restrictEncumbrance": true, "restrictExpenditures": true, "needCloseBudgets": true, "currencyFactor": 2, "budgetsRollover": [ { "rolloverAllocation": true, "rolloverBudgetValue": "Available", "setAllowances": false, "adjustAllocation": 0, "addAvailableTo": "Available" } ], "encumbrancesRollover": [ { "orderType": "Ongoing", "basedOn": "InitialAmount", "increaseBy": 0 }, { "orderType": "Ongoing-Subscription", "basedOn": "InitialAmount", "increaseBy": 0 }, { "orderType": "One-time", "basedOn": "InitialAmount", "increaseBy": 0 } ], "metadata": { "createdDate": "2023-07-06T22:49:20.492+00:00", "createdByUserId": "5f31c651-e0cd-4b1a-9acc-b5d1a0d160ce", "updatedDate": "2023-07-06T22:49:20.492+00:00", "updatedByUserId": "5f31c651-e0cd-4b1a-9acc-b5d1a0d160ce" } }');
end;
$$ LANGUAGE plpgsql;

Appendix

Infrastructure

PTF -environment nptf

8 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6.xlarge,db.r6.2xlarge,db.r6.4xlarge database (depending on a test)
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
mod-source-record-storage DB_CONNECTION_TIMEOUT =40;
mod-source-record-manager DB_RECONNECTATTEMPTS = 3, DB_RECONNECTINTERVAL=1000

Backend:

Module nptf-pvt Wed Jul 12 11:47:40 UTC 2023	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mod-remote-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:2.0.2	2	4920	4472	1024	3960	512	512	false
mod-finance-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.4.1	2	1024	896	128	700	88	128	false
mod-ncip	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.13.1	2	1024	896	128	768	88	128	false
mod-agreements	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:5.5.2	2	3096	2580	128	2048	384	512	false
mod-ebsconet	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.0.0	2	1248	1024	128	700	128	256	false
edge-sip2	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-sip2:3.0.0	2	1024	896	128	768	88	128	false
mod-organizations	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:1.7.0	2	1024	896	128	700	88	128	false
mod-settings	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.0.0	2	1024	896	200	0	0	0	false
mod-data-import	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:2.7.1	1	2048	1844	256	1292	384	512	false
edge-dematic	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematic:2.0.0	2	1024	896	128	0	0	0	false
mod-search	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:2.0.1	2	2592	2480	400	1440	512	1024	false
mod-tags	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tags:2.0.1	2	1024	896	128	768	88	128	false
mod-authtoken	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.13.0	2	1440	1152	512	922	88	128	false
mod-inventory-update	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-update:3.0.1	2	1024	896	128	768	88	128	false
mod-notify	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notify:3.0.0	2	1024	896	128	768	88	128	false
mod-configuration	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.9.1	2	1024	896	128	768	88	128	false
mod-orders-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-storage:13.5.0	2	1024	896	512	700	88	128	false
edge-caiasoft	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-caiasoft:2.0.0	2	1024	896	128	0	0	0	false
mod-login-saml	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-saml:2.6.1	2	1024	896	128	768	88	128	false
mod-erm-usage-harvester	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage-harvester:4.3.0	2	1024	896	128	768	88	128	false
mod-gobi	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-gobi:2.6.0	2	1024	896	128	700	88	128	false
mod-licenses	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-licenses:4.3.1	2	2480	2312	128	1792	384	512	false
mod-password-validator	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-password-validator:3.0.0	2	1440	1298	128	768	384	512	false
mod-bulk-operations	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operations:1.0.5	2	3072	2600	1024	1536	384	512	false
mod-graphql	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-graphql:1.11.0	2	1024	896	128	768	88	128	false
mod-finance	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance:4.7.1	2	1024	896	128	700	88	128	false
mod-erm-usage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage:4.5.2	2	1024	896	128	768	88	128	false
mod-copycat	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycat:1.4.0	2	1024	896	128	768	88	128	false
mod-permissions	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissions:6.3.1	2	1684	1544	512	1024	384	512	false
mod-entities-links	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entities-links:1.0.2	2	2592	2480	400	1440	0	1024	false
pub-edge	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge:2022.03.02	2	1024	896	128	768	0	0	false
mod-orders	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders:12.6.6	2	2048	1440	1024	1024	384	512	false
edge-patron	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-patron:4.11.0	2	1024	896	256	768	88	128	false
edge-ncip	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ncip:1.8.1	2	1024	896	128	768	88	128	false
mod-users-bl	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.5.0	2	1440	1152	512	922	88	128	false
mod-inventory-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:26.0.0	2	2208	1952	1024	1440	384	512	false
mod-invoice	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice:5.6.2	2	1440	1152	512	922	88	128	false
edge-ea-data-export	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ea-data-export:3.9.0	2	1024	896	128	768	88	128	false
mod-user-import	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-user-import:3.7.2	2	1024	896	128	768	88	128	false
mod-sender	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sender:1.10.0	2	1024	896	128	768	88	128	false
edge-oai-pmh	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-oai-pmh:2.6.1	2	1512	1360	1024	1440	384	512	false
mod-data-export-worker	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-worker:3.0.12	2	3072	2600	1024	2048	384	512	false
mod-rtac	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtac:3.5.0	2	1024	896	128	768	88	128	false
mod-circulation-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:16.0.0	2	1536	1440	1024	896	384	512	false
mod-source-record-storage	5	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.6.6	2	4096	3688	1024	3076	384	512	false
mod-calendar	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendar:2.4.2	2	1024	896	128	768	88	128	false
mod-event-config	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-config:2.5.0	2	1024	896	128	768	88	128	false
mod-courses	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-courses:1.4.7	2	1024	896	128	768	88	128	false
mod-inventory	3	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.0.4	2	2880	2592	1024	1814	384	512	false
mod-email	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-email:1.15.3	2	1024	896	128	768	88	128	false
mod-circulation	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:23.5.4	2	2880	2592	1536	768	88	128	false
mod-pubsub	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.9.1	2	1536	1440	1024	922	384	512	false
mod-di-converter-storage	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.0.2	2	1024	896	128	768	88	128	false
edge-orders	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-orders:2.8.1	2	1024	896	128	700	88	128	false
edge-rtac	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-rtac:2.6.0	2	1024	896	128	768	88	128	false
mod-template-engine	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-template-engine:1.18.0	2	1024	896	128	768	88	128	false
mod-users	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.1.1	2	1024	896	128	768	88	128	false
mod-patron-blocks	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.8.0	2	1024	896	1024	768	88	128	false
mod-audit	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-audit:2.7.0	2	1024	896	128	768	88	128	false
mod-source-record-manager	5	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.6.3	2	4096	3688	1024	3076	384	512	false
nginx-edge	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-edge:2022.03.02	2	1024	896	128	0	0	0	false
mod-quick-marc	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:3.0.0	1	2288	2176	128	1664	384	512	false
nginx-okapi	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2022.03.02	2	1024	896	128	0	0	0	false
okapi-b	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.0.1	3	1684	1440	1024	922	384	512	false
mod-feesfines	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:18.2.1	2	1024	896	128	768	88	128	false
mod-invoice-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-storage:5.6.0	2	1024	896	128	700	88	128	false
mod-service-interaction	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:2.2.2	2	2048	1844	256	1290	384	512	false
mod-patron	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron:5.5.2	2	1024	896	128	768	88	128	false
mod-data-export	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export:4.7.1	1	1024	896	1024	768	88	128	false
mod-oai-pmh	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmh:3.11.3	2	2248	2000	1024	1440	384	512	false
edge-connexion	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-connexion:1.0.6	2	1024	896	128	768	88	128	false
mod-notes	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:5.0.1	2	1024	896	128	322	128	128	false
mod-kb-ebsco-java	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-ebsco-java:3.13.0	2	1024	896	128	768	88	128	false
mod-data-export-spring	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:2.0.1	1	2048	1844	256	1292	200	256	false
mod-login	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login:7.9.0	2	1440	1298	1024	768	384	512	false
mod-organizations-storage	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storage:4.5.1	2	1024	896	128	700	88	128	false
pub-okapi	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2022.03.02	2	1024	896	128	768	0	0	false
mod-data-import-converter-storage	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import-converter-storage:1.15.2	2	1024	896	128	768	88	128	false
mod-eusage-reports	1	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:1.3.0	2	1024	896	128	768	88	128	false

Changes of configurations

Module nptf-pvt	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mod-configuration	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.9.1	2	1024	896	512	768	88	128	false
mod-inventory-storage	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:26.0.0	2	2208	1952	1536	1440	384	512	false
mod-source-record-storage	6	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.6.6	2	4096	3688	1536	3076	384	512	false
mod-users	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.1.1	2	1024	896	512	768	88	128	false
mod-source-record-manager	6	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.6.3	2	4096	3688	1536	3076	384	512	false
mod-quick-marc	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:3.0.0	2	2288	2176	512	1664	384	512	false
nginx-okapi	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2022.03.02	2	1024	896	1024	0	0	0	false
okapi-b	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.0.1	3	1684	1440	1536	922	384	512	false
mod-circulation-storage	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:16.0.0	2	1536	1440	1536	896	384	512	false
mod-circulation	2	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:23.5.4	2	2880	2592	1536	2048	384	512	false

Different configurations testing

Summary

General response time for all workflows is about 24% improved for 2xlarge database size compared to xlarge (for stress test) and improved about 23% for a normal load test. Tested with default services configuration and without Data Import.

Changes in configurations for services do not improve response time for workflows but for mod-circulation, it should be revised due to configuration issues found in the task definition: excessive parameters
"-XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m"

{
                    "name": "JAVA_OPTS",
                    "value": "-Dport=8082 -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-circulation.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=512m -Xmx1814m"
                },

If release versions for mod-source-record-storage will be 5.6.5 and for mod-source-record-manager - 3.6.2 then changes of parameters DB_MAXPOOLSIZE for both modules, and DB_CONNECTION_TIMEOUT for mod-source-record-storage needed to avoid PERF-582 - Getting issue details... STATUS :

                {
                    "name": "DB_MAXPOOLSIZE",
                    "value": "500"
                },
                {
                    "name": "DB_CONNECTION_TIMEOUT",
                    "value": "60"
                },

1# Normal load test
Workflow	NO DI	NO DI 2xlarge db	Comparison of difference between DB size with the same service changes (%)	NO DI Updated 2xlarge db mod-users, mod-circulation, mod-inventory-storage, mod-inventory, nginx-okapi, mod-srm, mod-srs	NO DI Updated mod-users, mod-circulation, mod-inventory-storage, mod-inventory, nginx-okapi, mod-srm, mod-srs	Comparison of difference between DB size with the same service configurations (%)
AIE_TC: View Invoices Workflow	3.32	2.463	25.81	2.54	2.761	8
BLS_TC: Blacklight simulation	1.526	0.966	36.69	1.061	1.116	4.92
CICO_TC_Check-In Controller	1.247	0.978	21.57	1.081	1.117	3.22
CICO_TC_Check-Out Controller	2.037	1.622	20.37	1.702	1.775	4.11
ETT_TC: Edit Bib RecordTagTable Group	2.534	2.481	2.09	2.841	2.137	-32.94
ILR_TC: Create ILR	0.669	0.509	23.91	0.543	0.58	6.37
ILR_TC: Get ItemId	0.106	0.076	28.3	0.086	0.078	-10.25
ILR_TC: Get RequesterId	0.028	0.025	10.71	0.028	0.032	12.5
IO_TC: Create Invoices Workflow	2.083	1.616	22.41	1.681	1.727	2.66
IO_TC: Delete Invoices Workflow	1.405	1.136	19.14	1.141	1.251	8.79
IO_TC: Edit Invoices Workflow	2.7	2.105	22.03	2.152	2.313	6.96
IO_TC: View Invoices Workflow	1.106	0.934	15.55	1.072	1	-7.2
IRO_TC: Delete Item	1.183	0.902	23.75	0.939	0.985	4.67
IRO_TC: Edit Item	1.316	0.897	31.83	0.977	1.002	2.49
IRO_TC: View Item	1.322	0.992	24.96	1.152	1.163	0.94
LO_TC: Create Ledgers	0.722	0.572	20.77	0.615	0.621	0.96
LO_TC: Delete Ledgers	0.06	0.043	28.33	0.058	0.062	6.45
LO_TC: Edit Ledgers	0.082	0.088	-7.31	0.058	0.046	-26.08
LO_TC: View Ledgers	0.053	0.04	24.52	0.044	0.046	4.34
POO_TC Create Order	2.038	1.576	22.66	1.706	1.785	4.42
POO_TC Delete Order	1.049	0.976	6.95	1.064	0.84	-26.66
POO_TC Edit Order	1.633	1.42	13.04	1.054	1.282	17.78
POO_TC View Order	1.024	0.856	16.4	0.799	0.858	6.87
PRO_TC: Create User	1.49	1.036	30.46	1.08	1.226	11.9
PRO_TC: Delete User	0.813	0.659	18.94	0.631	0.739	14.61
PRO_TC: Edit User	1.312	1.158	11.73	1.106	1.093	-1.18
PRO_TC: View User	0.755	0.511	32.31	0.687	0.597	-15.07
PRV_TC: View Patron record Group	0.097	0.062	36.08	0.084	0.072	-16.66
ULR_TC: Users loan Renewal Transaction	1.71	1.36	20.46	1.431	1.509	5.16
VAR_TC: GoTo Marc Authority	0.061	0.083	-36.06	0.1	0.073	-36.98
VAR_TC: Search Marc Authority	0.127	0.136	-7.08	0.163	0.144	-13.19
VB_TC: Inventory Choose Instance	0.449	0.315	29.84	0.345	0.363	4.95
VB_TC: Inventory Search Instance	0.063	0.066	-4.76	0.084	0.052	-61.53
VB_TC: Load Inventory	0.526	0.438	16.73	0.446	0.468	4.7
VH_TC: Inventory Choose Holdings	0.434	0.311	28.34	0.34	0.358	5.02
VH_TC: Inventory Search Holdings	0.046	0.067	-45.65	0.1	0.048	-108.33
VH_TC: Inventory View Holdings	0.449	0.334	25.61	0.357	0.361	1.1
VH_TC: Load Inventory	0.532	0.47	11.65	0.487	0.51	4.5
VIH_TC: View instance holdings details	1.718	1.405	18.21	1.529	1.494	-2.34
VIR_TC: inventory record monitoring	0.946	0.773	18.28	0.772	0.827	6.65
VRO_TC: Create Vendor	1.527	0.965	36.8	1.005	1.138	11.68
VRO_TC: Delete Vendor	0.565	0.431	23.71	0.424	0.474	10.54
VRO_TC: Edit Vendor	7.297	4.647	36.31	4.422	5.804	23.81
VRO_TC: View Vendor	0.886	0.675	23.81	0.697	0.879	20.7
VTT_TC: Inventory Choose Instance	0.438	0.286	34.7	0.322	0.337	4.45
VTT_TC: Inventory Search Instance	0.038	0.04	-5.26	0.053	0.037	-43.24
VTT_TC: Inventory View Instance Source	0.174	0.118	32.18	0.127	0.137	7.29
Total response time	52.2	40.2	22.98	41.7	43.8	4.79

2# Stress test load 2 threads
Workflow	Before services adjustment with DI	After adjustments with DI	NO DI	NO DI Updated DB 2x large DB CPU usage decreased from 99.4 to 86.9 %	Comparison of difference between DB size with the same service changes (%)	NO DI Updated DB 2x large mod-users, mod-circulation, mod-inventory-storage, mod-inventory, nginx-okapi, mod-srm, mod-srs
AIE_TC: View Invoices Workflow	12.409	10.936	7.134	4.137	42.01	4.302
BLS_TC: Blacklight simulation	5.742	4.684	3.321	1.791	46.07	1.976
Check-In Controller	4.504	3.394	2.299	2.546	-10.74	2.621
Check-Out Controller	5.926	5.472	3.281	3.794	-15.63	3.669
ETT_TC: Edit Bib RecordTagTable Group	19.905	45.347	10.24	11.7	-14.25	11.801
ILR_TC: Create ILR	1.811	1.468	0.992	1.029	-3.72	1.105
ILR_TC: Get ItemId	1.718	1.414	0.858	0.947	-10.37	0.997
ILR_TC: Get RequesterId	0.091	0.071	0.05	0.06	-20	0.044
IO_TC: Create Invoices Workflow	6.133	5.74	3.725	2.886	22.52	2.854
IO_TC: Delete Invoices Workflow	5.181	4.804	3.168	2.02	36.23	1.968
IO_TC: Edit Invoices Workflow	9.872	8.318	5.9	4.004	32.13	3.912
IO_TC: View Invoices Workflow	4.269	3.821	2.533	1.772	30.04	1.73
IRO_TC: Delete Item	6.936	6.551	3.732	4.598	-23.2	4.164
IRO_TC: Edit Item	7.092	6.498	3.531	4.072	-15.32	4.013
IRO_TC: View Item	10.034	7.61	4.781	6.36	-33.02	6.039
LO_TC: Create Ledgers	1.615	1.329	0.926	0.901	2.69	0.898
LO_TC: Delete Ledgers	0.509	0.394	0.233	0.116	50.21	0.12
LO_TC: Edit Ledgers	0.513	0.395	0.231	0.12	48.05	0.132
LO_TC: View Ledgers	0.518	0.373	0.231	0.11	52.38	0.119
POO_TC Create Order	5.877	4.722	2.941	3.231	-9.86	3.295
POO_TC Delete Order	4.055	3.533	2.07	1.94	6.28	1.915
POO_TC Edit Order	6.075	4.93	2.471	2.563	-3.72	2.071
POO_TC View Order	3.864	3.103	1.867	1.655	11.35	1.545
PRO_TC: Create User	3.675	2.922	1.993	2.198	-10.28	1.938
PRO_TC: Delete User	1.939	1.692	1.062	1.161	-9.32	1.062
PRO_TC: Edit User	3.969	3.431	2.175	2.022	7.03	1.853
PRO_TC: View User	1.969	1.685	1.116	1.139	-2.06	1.049
PRV_TC: View Patron record Group	0.241	0.194	0.128	0.117	8.59	0.103
ULR_TC: Users loan Renewal Transaction	4.835	4.969	3.133	2.221	29.1	2.292
VAR_TC: Click Authorized Record	0.251	0.184	0.114	0.167	-46.49	0.155
VAR_TC: GoTo Marc Authority	0.073	0.068	0.066	0.093	-40.9	0.081
VAR_TC: Search Marc Authority	0.107	0.09	0.081	0.136	-67.9	0.102
VB_TC: Inventory Choose Instance	3.776	3.482	1.913	2.149	-12.33	2.11
VB_TC: Inventory Search Instance	0.05	0.049	0.046	0.074	-60.86	0.058
VB_TC: Load Inventory	1.33	1.18	0.877	1.067	-21.66	1.01
VH_TC: Inventory Choose Holdings	3.487	3.098	1.772	2.168	-22.34	2.098
VH_TC: Inventory Search Holdings	0.026	0.026	0.028	0.048	-71.42	0.035
VH_TC: Inventory View Holdings	3.797	3.159	1.895	2.495	-31.66	2.522
VH_TC: Load Inventory	1.395	1.192	0.863	1.024	-18.65	0.977
VIH_TC: View instance holdings details	7.692	6.609	3.833	3.689	3.75	3.66
VIR_TC: inventory record monitoring	10.985	9.359	6.432	2.932	54.41	3.238
VRO_TC: Create Vendor	8.856	7.458	4.735	1.913	59.59	2.056
VRO_TC: Delete Vendor	2.728	2.201	1.086	0.822	24.3	0.917
VRO_TC: Edit Vendor	44.228	36.88	28.451	8.424	70.39	9.557
VRO_TC: View Vendor	7.04	5.277	4.611	1.371	70.26	1.332
VTT_TC: Inventory Choose Instance	3.255	2.769	1.663	1.888	-13.52	1.896
VTT_TC: Inventory Search Instance	0.046	0.047	0.044	0.063	-43.18	0.056
VTT_TC: Inventory View Instance Source	2.37	1.84	1.098	1.347	-22.67	1.323
VTT_TC: Load Inventory	1.356	1.221	0.897	1.073	-19.62	1.036
Total response time			136.6	104.1	23.79%	103.8

Methodology/Approach

As a base to generate stress load we've used MasterScript with excluded controllers on transactions per minute. So in this case threads is not equal to Virtual users as threads will generate more load than users.

Master Script stress test - NLA report

Overview

Summary

Recommendations & Jiras (Optional)

Test Runs

Results and Comparisons

Performance dependency with increasing load with xlarge DB

Performance dependency with increasing load with 2xlarge DB

Comparison xlarge tests with 2xlarge tests

Comparison to separate workflows stress tests run

CICO: Check-In/Check-Out

IRO: Item record operation

ETT: Edit Bib RecordTagTable

Xlarge DB stress test

Memory Utilization

CPU Utilization

RDS CPU Utilization

2Xlarge DB stress test

Memory Utilization

CPU Utilization

RDS CPU Utilization

4 Hours stress test (xlarge DB)

Memory Utilization

CPU Utilization

RDS CPU Utilization

4Xlarge DB stress test

Memory Utilization

CPU Utilization

RDS CPU Utilization

Database analysis

Appendix

Infrastructure

Changes of configurations

Different configurations testing

Summary

Methodology/Approach