/
[Orchid] Combined Bulk edit + Check-in-check-out

[Orchid] Combined Bulk edit + Check-in-check-out

Apr 14, 2023

Overview

Test goal is to assess performance impact and dependency between Check-in/Check-out and Bulk Edit functionality.

Ticket: PERF-402: [Orchid] Performance dependencies between CI/CO and bulk editClosed

Summary

Test results showed next influence on performance of two scenarios being run in parallel:

  • Degradation for Bulk Edit is between -2.1% and +14.9%. Degradation for Check-in is between -0.6% and +26%. Degradation for Check-out is between -1% and +6%. 

  • The highest degradation (absolute numbers) was observed for Bulk Edit operation for 100k Items. Degradation was 10 minutes (+7.8%).

  • Average Service CPU consumption increase (for parallel tests) is about 10%, DB CPU consumption increase is about 5%.

  • Overall, there is no specific tendency in results comparison. Additional tests may be required for further investigation.

There are some errors observed during testing of Bulk Edit operation for 100k Items. Errors are connected to the items being processed, about 0.03% of the whole amount. Possible cause of these errors is that some items were duplicated in CICO script and were reused there. It didn't have any impact on the performance.

Test Runs 

Test number

Test

Duration of CI/CO

Timeframe Bulk Edit

Timeframe CICO

Test number

Test

Duration of CI/CO

Timeframe Bulk Edit

Timeframe CICO

1.

Bulk edit 10k Items 

-

3/30/2023, 7:44 AM - 3/30/2023, 7:59 AM

-

2.

Bulk edit 100k Items 

-

3/31/2023, 4:08 PM - 3/31/2023, 6:29 PM

-

3.

Bulk edit 10k Users

-

3/30/2023, 8:33 AM - 3/30/2023, 8:43 AM

-

4.

Bulk edit 10k Holdings

-

3/30/2023, 9:11 AM - 3/30/2023, 9:26 AM

-

5.

Bulk edit 10k Items + 10k Users + 10k Holdings

-

4/3/2023, 12:02 PM - 4/3/2023, 12:18 PM Items

 4/3/2023, 12:03 PM - 4/3/2023, 12:13 PM Users 

4/3/2023, 12:03 PM - 4/3/2023, 12:18 PM Holdings

-

6.

CI/CO 8 users

30 mins

-

3/29/2023, 09:42 AM - 3/29/2023, 10:12 AM

7.

CI/CO 8 users + Bulk edit 10k Items 

30 mins

3/29/2023, 10:44 AM - 3/29/2023, 11:00 AM

3/29/2023, 10:41 AM - 3/29/2023, 11:12 AM

8.

CI/CO 8 users + Bulk edit 100k Items 

180 mins

4/2/2023, 7:15 PM -  4/2/2023, 9:56 PM

4/2/2023, 7:08 PM -  4/2/2023, 10:09 PM

9.

CI/CO 8 users + Bulk edit 10k Users

30 mins

3/29/2023, 11:39 AM - 3/29/2023, 11:49 AM

3/29/2023, 11:36 AM - 3/29/2023, 12:06 PM

10.

CI/CO 8 users + Bulk edit 10k Holdings

30 mins

3/29/2023, 1:01 PM - 3/29/2023, 1:18 PM

3/29/2023, 12:56 PM - 3/29/2023, 1:27 PM

11.

CI/CO 8 users + Bulk edit 10k Items + 10k Users + 10k Holdings

30 mins

4/3/2023, 1:16 PM - 4/3/2023, 1:31 PM Items

4/3/2023, 1:16 PM - 4/3/2023, 1:27 PM Users 

4/3/2023, 1:16 PM - 4/3/2023, 1:30 PM Holdings

 

4/3/2023, 1:12 PM - 4/3/2023, 1:42 PM

Response Times

Test

Bulk Edit response time, min

CI/CO respoinse time, sec

Items

Users

Holdings

Check-in

Check

Baseline tests (CI/CO and BE run separately)

Bulk edit 10k Items 

15:30

-

 

-

-

Bulk edit 100k Items 

02:20:29

-

 

-

-

Bulk edit 10k Users

-

10:22

 

-

-

Bulk edit 10k Holdings

-

-

14:59

-

-

Bulk edit 10k Items + 10k Users + 10k Holdings

15:56

10:35

14:49

-

-

CI/CO 8 users

-

-

-

0.448

0.831

Verification tests (CI/CO and BE run in parallel)

Bulk edit 10k Items + CI/CO 8 users 

15:47 (+1.8%)

-

 

0.445 (-0.6%)

0.849 (+2%)

Bulk edit 100k Items + CI/CO 8 users 

02:41:29 (+14.9%)

02:31:34 (+7.8%) - retest

-

 

0.445 (-1%)

0.427 (-4.6%) - retest

0.857 (+3%)

0.823 (-1%) - retest

Bulk edit 10k Users+ CI/CO 8 users 

-

10:28 (+0.9%)

 

0.543 (+21%)

0.858 (+3%)

Bulk edit 10k Holdings+ CI/CO 8 users 

-

-

16:38 (+11%)

0.535 (+19%)

0.856 (+3%)

Bulk edit 10k Items + 10k Users + 10k Holdings+ CI/CO 8 users 

15:38 (-1.8%)

10:59 (+3.7%)

14:30 (-2.1%)

0.566 (+26%)

 

0.884 (+6%)

*Cells are colored differently to indicate which numbers are being compared.

Resource monitoring: CI/CO 8 users

Instance CPU Utilization

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

 

Resource monitoring: Bulk edit 10k Items + CI/CO 8 users

Instance CPU Utilization

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

Service CPU Utilization

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

Memory Utilization

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

DB CPU Utilization

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

DB Connections

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

Resource monitoring: Bulk edit 100k Items + CI/CO 8 users

Instance CPU Utilization

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

Service CPU Utilization

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

Unusually high CPU utilization was observed on mod-users module during retest - about 33%.

Memory Utilization

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

DB CPU Utilization

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

DB Connections

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

Resource monitoring: Bulk edit 10k Users + CI/CO 8 users

Instance CPU Utilization

Bulk edit 10k Users

Bulk edit 10k Users + CI/CO 8 users

Service CPU Utilization

Bulk edit 10k Users

Bulk edit 10k Users + CI/CO 8 users

Memory Utilization

Bulk edit 10k Users

 

Bulk edit 10k Users + CI/CO 8 users

DB CPU Utilization

Bulk edit 10k Users

Bulk edit 10k Users + CI/CO 8 users

DB Connections

Bulk edit 10k Users

Bulk edit 10k Users + CI/CO 8 users

 

Resource monitoring: Bulk edit 10k Holdings + CI/CO 8 users

Instance CPU Utilization

Bulk edit 10k Holdings

Bulk edit 10k Holdings+ CI/CO 8 users

Service CPU Utilization

Bulk edit 10k Holdings

Bulk edit 10k Holdings+ CI/CO 8 users

Memory Utilization

Bulk edit 10k Holdings

Bulk edit 10k Holdings+ CI/CO 8 users

DB CPU Utilization

Bulk edit 10k Holdings

Bulk edit 10k Holdings+ CI/CO 8 users

DB Connections

Bulk edit 10k Holdings

Bulk edit 10k Holdings+ CI/CO 8 users

 

 

Resource monitoring: Bulk edit 10k Items + 10k Users + 10k Holdings + CI/CO 8 users 

Instance CPU Utilization

Bulk edit 10k Items + 10k Users + 10k Holdings

Bulk edit 10k Items + 10k Users + 10k Holdings + CI/CO 8 users

Service CPU Utilization

Bulk edit 10k Items + 10k Users + 10k Holdings

Bulk edit 10k Items + 10k Users + 10k Holdings + CI/CO 8 users

Memory Utilization

Bulk edit 10k Items + 10k Users + 10k Holdings

Bulk edit 10k Items + 10k Users + 10k Holdings + CI/CO 8 users

DB CPU Utilization

Bulk edit 10k Items + 10k Users + 10k Holdings

Bulk edit 10k Items + 10k Users + 10k Holdings + CI/CO 8 users

DB Connections

Bulk edit 10k Items + 10k Users + 10k Holdings

Bulk edit 10k Items + 10k Users + 10k Holdings + CI/CO 8 users

Appendix

Infrastructure

 

PTF -environment ncp5

  • 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

  • 2 instances of db.r6.xlarge database instances: Writer & reader instances

  • MSK ptf-kakfa-3 [ kafka configurations]

    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

 

Modules memory and CPU parameters:

Module

SoftLimit

XMX

Revision

Version

desiredCount

CPUUnits

RWSplitEnabled

HardLimit

Metaspace

MaxMetaspaceSize

Module

SoftLimit

XMX

Revision

Version

desiredCount

CPUUnits

RWSplitEnabled

HardLimit

Metaspace

MaxMetaspaceSize

mod-inventory-storage-b

1952

1440

3

579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:26.0.0

2

1024

False

2208

384

512

mod-inventory-b

2592

1814

7

579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.0.3

2

1024

False

2880

384

512

okapi-b

1440

922

1

579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.0.1

3

1024

False

1684

384

512

mod-users-b

896

768

4