Overview

Test goal is to assess performance impact and dependency between Check-in/Check-out and Bulk Edit functionality.

Ticket: PERF-402 - Getting issue details... STATUS

Summary

Test results showed next influence on performance of two scenarios being run in parallel:

Degradation for Bulk Edit is between -2.1% and +14.9%. Degradation for Check-in is between -0.6% and +26%. Degradation for Check-out is between -1% and +6%.
The highest degradation (absolute numbers) was observed for Bulk Edit operation for 100k Items. Degradation was 10 minutes (+7.8%).
Average Service CPU consumption increase (for parallel tests) is about 10%, DB CPU consumption increase is about 5%.
Overall, there is no specific tendency in results comparison. Additional tests may be required for further investigation.

There are some errors observed during testing of Bulk Edit operation for 100k Items. Errors are connected to the items being processed, about 0.03% of the whole amount. Possible cause of these errors is that some items were duplicated in CICO script and were reused there. It didn't have any impact on the performance.

Test Runs

Test number	Test	Duration of CI/CO	Timeframe Bulk Edit	Timeframe CICO
1.	Bulk edit 10k Items	-	3/30/2023, 7:44 AM - 3/30/2023, 7:59 AM	-
2.	Bulk edit 100k Items	-	3/31/2023, 4:08 PM - 3/31/2023, 6:29 PM	-
3.	Bulk edit 10k Users	-	3/30/2023, 8:33 AM - 3/30/2023, 8:43 AM	-
4.	Bulk edit 10k Holdings	-	3/30/2023, 9:11 AM - 3/30/2023, 9:26 AM	-
5.	Bulk edit 10k Items + 10k Users + 10k Holdings	-	4/3/2023, 12:02 PM - 4/3/2023, 12:18 PM Items 4/3/2023, 12:03 PM - 4/3/2023, 12:13 PM Users 4/3/2023, 12:03 PM - 4/3/2023, 12:18 PM Holdings	-
6.	CI/CO 8 users	30 mins	-	3/29/2023, 09:42 AM - 3/29/2023, 10:12 AM
7.	CI/CO 8 users + Bulk edit 10k Items	30 mins	3/29/2023, 10:44 AM - 3/29/2023, 11:00 AM	3/29/2023, 10:41 AM - 3/29/2023, 11:12 AM
8.	CI/CO 8 users + Bulk edit 100k Items	180 mins	4/2/2023, 7:15 PM - 4/2/2023, 9:56 PM	4/2/2023, 7:08 PM - 4/2/2023, 10:09 PM
9.	CI/CO 8 users + Bulk edit 10k Users	30 mins	3/29/2023, 11:39 AM - 3/29/2023, 11:49 AM	3/29/2023, 11:36 AM - 3/29/2023, 12:06 PM
10.	CI/CO 8 users + Bulk edit 10k Holdings	30 mins	3/29/2023, 1:01 PM - 3/29/2023, 1:18 PM	3/29/2023, 12:56 PM - 3/29/2023, 1:27 PM
11.	CI/CO 8 users + Bulk edit 10k Items + 10k Users + 10k Holdings	30 mins	4/3/2023, 1:16 PM - 4/3/2023, 1:31 PM Items 4/3/2023, 1:16 PM - 4/3/2023, 1:27 PM Users 4/3/2023, 1:16 PM - 4/3/2023, 1:30 PM Holdings	4/3/2023, 1:12 PM - 4/3/2023, 1:42 PM

Response Times

Test	Bulk Edit response time, min			CI/CO respoinse time, sec
Test	Items	Users	Holdings	Check-in	Check
Baseline tests (CI/CO and BE run separately)
Bulk edit 10k Items	15:30	-		-	-
Bulk edit 100k Items	02:20:29	-		-	-
Bulk edit 10k Users	-	10:22		-	-
Bulk edit 10k Holdings	-	-	14:59	-	-
Bulk edit 10k Items + 10k Users + 10k Holdings	15:56	10:35	14:49	-	-
CI/CO 8 users	-	-	-	0.448	0.831
Verification tests (CI/CO and BE run in parallel)
Bulk edit 10k Items + CI/CO 8 users	15:47 (+1.8%)	-		0.445 (-0.6%)	0.849 (+2%)
Bulk edit 100k Items + CI/CO 8 users	02:41:29 (+14.9%) 02:31:34 (+7.8%) - retest	-		0.445 (-1%) 0.427 (-4.6%) - retest	0.857 (+3%) 0.823 (-1%) - retest
Bulk edit 10k Users+ CI/CO 8 users	-	10:28 (+0.9%)		0.543 (+21%)	0.858 (+3%)
Bulk edit 10k Holdings+ CI/CO 8 users	-	-	16:38 (+11%)	0.535 (+19%)	0.856 (+3%)
Bulk edit 10k Items + 10k Users + 10k Holdings+ CI/CO 8 users	15:38 (-1.8%)	10:59 (+3.7%)	14:30 (-2.1%)	0.566 (+26%)	0.884 (+6%)

*Cells are colored differently to indicate which numbers are being compared.

Resource monitoring: CI/CO 8 users

Instance CPU Utilization

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

Resource monitoring: Bulk edit 10k Items + CI/CO 8 users

Instance CPU Utilization

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

Service CPU Utilization

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

Memory Utilization

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

DB CPU Utilization

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

DB Connections

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

Resource monitoring: Bulk edit 100k Items + CI/CO 8 users

Instance CPU Utilization

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

Service CPU Utilization

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

Unusually high CPU utilization was observed on mod-users module during retest - about 33%.

Memory Utilization

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

DB CPU Utilization

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

DB Connections

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

Resource monitoring: Bulk edit 10k Users + CI/CO 8 users