[Orchid] Combined Bulk edit + Check-in-check-out

Owned by Hanna Tuhuzbayeva
Last updated: Apr 14, 2023
10 min read

Overview

Test goal is to assess performance impact and dependency between Check-in/Check-out and Bulk Edit functionality.

Ticket: PERF-402 - Getting issue details... STATUS

Summary

Test results showed next influence on performance of two scenarios being run in parallel:

  • Degradation for Bulk Edit is between -2.1% and +14.9%. Degradation for Check-in is between -0.6% and +26%. Degradation for Check-out is between -1% and +6%. 
  • The highest degradation (absolute numbers) was observed for Bulk Edit operation for 100k Items. Degradation was 10 minutes (+7.8%).
  • Average Service CPU consumption increase (for parallel tests) is about 10%, DB CPU consumption increase is about 5%.
  • Overall, there is no specific tendency in results comparison. Additional tests may be required for further investigation.

There are some errors observed during testing of Bulk Edit operation for 100k Items. Errors are connected to the items being processed, about 0.03% of the whole amount. Possible cause of these errors is that some items were duplicated in CICO script and were reused there. It didn't have any impact on the performance.

Test Runs 

Test number

Test

Duration of CI/CO

Timeframe Bulk EditTimeframe CICO

1.

Bulk edit 10k Items -3/30/2023, 7:44 AM - 3/30/2023, 7:59 AM-

2.

Bulk edit 100k Items -3/31/2023, 4:08 PM - 3/31/2023, 6:29 PM-

3.

Bulk edit 10k Users-3/30/2023, 8:33 AM - 3/30/2023, 8:43 AM

-

4.Bulk edit 10k Holdings-3/30/2023, 9:11 AM - 3/30/2023, 9:26 AM-
5.Bulk edit 10k Items + 10k Users + 10k Holdings-

4/3/2023, 12:02 PM - 4/3/2023, 12:18 PM Items

 4/3/2023, 12:03 PM - 4/3/2023, 12:13 PM Users 

4/3/2023, 12:03 PM - 4/3/2023, 12:18 PM Holdings

-
6.CI/CO 8 users30 mins-

3/29/2023, 09:42 AM - 3/29/2023, 10:12 AM

7.CI/CO 8 users + Bulk edit 10k Items 30 mins3/29/2023, 10:44 AM - 3/29/2023, 11:00 AM

3/29/2023, 10:41 AM - 3/29/2023, 11:12 AM

8.CI/CO 8 users + Bulk edit 100k Items 180 mins

4/2/2023, 7:15 PM -  4/2/2023, 9:56 PM

4/2/2023, 7:08 PM -  4/2/2023, 10:09 PM
9.CI/CO 8 users + Bulk edit 10k Users30 mins3/29/2023, 11:39 AM - 3/29/2023, 11:49 AM

3/29/2023, 11:36 AM - 3/29/2023, 12:06 PM

10.CI/CO 8 users + Bulk edit 10k Holdings30 mins3/29/2023, 1:01 PM - 3/29/2023, 1:18 PM

3/29/2023, 12:56 PM - 3/29/2023, 1:27 PM

11.CI/CO 8 users + Bulk edit 10k Items + 10k Users + 10k Holdings30 mins

4/3/2023, 1:16 PM - 4/3/2023, 1:31 PM Items

4/3/2023, 1:16 PM - 4/3/2023, 1:27 PM Users 

4/3/2023, 1:16 PM - 4/3/2023, 1:30 PM Holdings


4/3/2023, 1:12 PM - 4/3/2023, 1:42 PM

Response Times

Test

Bulk Edit response time, minCI/CO respoinse time, sec
ItemsUsersHoldingsCheck-inCheck
Baseline tests (CI/CO and BE run separately)
Bulk edit 10k Items 15:30-
--
Bulk edit 100k Items 02:20:29-
--
Bulk edit 10k Users-10:22
--
Bulk edit 10k Holdings--14:59--
Bulk edit 10k Items + 10k Users + 10k Holdings15:5610:3514:49--
CI/CO 8 users---0.4480.831
Verification tests (CI/CO and BE run in parallel)
Bulk edit 10k Items + CI/CO 8 users 15:47 (+1.8%)-
0.445 (-0.6%)0.849 (+2%)
Bulk edit 100k Items + CI/CO 8 users 

02:41:29 (+14.9%)

02:31:34 (+7.8%) - retest

-

0.445 (-1%)

0.427 (-4.6%) - retest

0.857 (+3%)

0.823 (-1%) - retest

Bulk edit 10k Users+ CI/CO 8 users -10:28 (+0.9%)
0.543 (+21%)0.858 (+3%)
Bulk edit 10k Holdings+ CI/CO 8 users --16:38 (+11%)0.535 (+19%)0.856 (+3%)
Bulk edit 10k Items + 10k Users + 10k Holdings+ CI/CO 8 users 15:38 (-1.8%)10:59 (+3.7%)14:30 (-2.1%)

0.566 (+26%)


0.884 (+6%)

*Cells are colored differently to indicate which numbers are being compared.

Resource monitoring: CI/CO 8 users

Instance CPU Utilization

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections


Resource monitoring: Bulk edit 10k Items + CI/CO 8 users

Instance CPU Utilization

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

Service CPU Utilization

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

Memory Utilization

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

DB CPU Utilization

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

DB Connections

Bulk edit 10k Items

Bulk edit 10k Items + CI/CO 8 users

Resource monitoring: Bulk edit 100k Items + CI/CO 8 users

Instance CPU Utilization

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

Service CPU Utilization

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

Unusually high CPU utilization was observed on mod-users module during retest - about 33%.

Memory Utilization

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

DB CPU Utilization

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

DB Connections

Bulk edit 100k Items

Bulk edit 100k Items + CI/CO 8 users

Bulk edit 100k Items + CI/CO 8 users (retest)

Resource monitoring: Bulk edit 10k Users + CI/CO 8 users

Instance CPU Utilization

Bulk edit 10k Users

Bulk edit 10k Users + CI/CO 8 users

Service CPU Utilization

Bulk edit 10k Users

Bulk edit 10k Users + CI/CO 8 users

Memory Utilization

Bulk edit 10k Users


Bulk edit 10k Users + CI/CO 8 users

DB CPU Utilization

Bulk edit 10k Users

Bulk edit 10k Users + CI/CO 8 users

DB Connections

Bulk edit 10k Users

Bulk edit 10k Users + CI/CO 8 users


Resource monitoring: Bulk edit 10k Holdings + CI/CO 8 users

Instance CPU Utilization

Bulk edit 10k Holdings

Bulk edit 10k Holdings+ CI/CO 8 users

Service CPU Utilization

Bulk edit 10k Holdings

Bulk edit 10k Holdings+ CI/CO 8 users

Memory Utilization

Bulk edit 10k Holdings

Bulk edit 10k Holdings+ CI/CO 8 users

DB CPU Utilization

Bulk edit 10k Holdings

Bulk edit 10k Holdings+ CI/CO 8 users

DB Connections

Bulk edit 10k Holdings

Bulk edit 10k Holdings+ CI/CO 8 users



Resource monitoring: Bulk edit 10k Items + 10k Users + 10k Holdings + CI/CO 8 users 

Instance CPU Utilization

Bulk edit 10k Items + 10k Users + 10k Holdings

Bulk edit 10k Items + 10k Users + 10k Holdings + CI/CO 8 users

Service CPU Utilization

Bulk edit 10k Items + 10k Users + 10k Holdings

Bulk edit 10k Items + 10k Users + 10k Holdings + CI/CO 8 users

Memory Utilization

Bulk edit 10k Items + 10k Users + 10k Holdings

Bulk edit 10k Items + 10k Users + 10k Holdings + CI/CO 8 users

DB CPU Utilization

Bulk edit 10k Items + 10k Users + 10k Holdings

Bulk edit 10k Items + 10k Users + 10k Holdings + CI/CO 8 users

DB Connections

Bulk edit 10k Items + 10k Users + 10k Holdings

Bulk edit 10k Items + 10k Users + 10k Holdings + CI/CO 8 users

Appendix

Infrastructure


PTF -environment ncp5

  • 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
  • 2 instances of db.r6.xlarge database instances: Writer & reader instances
  • MSK ptf-kakfa-3 [ kafka configurations]
    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


Modules memory and CPU parameters:

Module

SoftLimit

XMX

Revision

Version

desiredCount

CPUUnits

RWSplitEnabled

HardLimit

Metaspace

MaxMetaspaceSize

mod-inventory-storage-b195214403
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:26.0.0
21024False2208384512
mod-inventory-b259218147
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.0.3
21024False2880384512
okapi-b14409221
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.0.1
31024False1684384512
mod-users-b8967684
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.1.0
2128False102488128
mod-data-export-worker
2600
2048
3
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-worker:3.0.9
2
1024
False
3072
384
512
mod-data-export-spring
1844
1292
3
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:2.0.0-SNAPSHOT.67
1
 256
False
2048
200
256
mod-bulk-operations260015362
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operations:1.0.4
21024False3072384512
mod-notes
896
322
3
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:5.1.0-SNAPSHOT.245
2
128
False
1024
128
128
mod-agreements
2580
2048
2
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:5.6.0-SNAPSHOT.117
2
128
False
3096
384
512
nginx-okapi
896
01
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2022.03.02
2
128
False
1024
00
mod-feesfines8967684
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:18.2.1
2128False102488128
mod-patron-blocks8967684
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.8.0
21024False102488128
mod-pubsub
1440
922
4
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.9.1
21024False
1536
384512
mod-authtoken11529223
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.14.0-SNAPSHOT.238
2
512
False114088128
mod-circulation-storage14408963
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:16.0.0
21024False
1536
384512
mod-circulation25927683
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:23.5.2
2
1536False102488128
mod-configuration8967683
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.9.0
2128False288088128
mod-remote-storage4472
3960
3
579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:2.0.2
21024False4920512512

Methodology/Approach

  1. Before each test run necessary commands to return the database to the initial state - both for CICO and BE. Instructions can be found here: Steps for testing process.
  2. Conduct CICO and BE tests separately for baseline.
  3. Conduct CICO and BE tests in parallel for verification.
  4. Compare results and resource consumption.

Note - make sure that items used for CICO and BE items don't have any items in common. Lists should be totally different.

Grafana screenshots

CI/CO 8 users

CI/CO 8 users + Bulk edit 10k Items 

CI/CO 8 users + Bulk edit 100k Items 

CI/CO 8 users + Bulk edit 100k Items (retest)

CI/CO 8 users + Bulk edit 10k Users

CI/CO 8 users + Bulk edit 10k Holdings

CI/CO 8 users + Bulk edit 10k Items + 10k Users + 10k Holdings


Errors

CI/CO 8 users + Bulk edit 100k Items:

{
  "errors": [
    {
      "message": "New status value \"Unknown\" is not allowed",
      "parameters": [
        {
          "key": "IDENTIFIER",
          "value": "084288999"
        }
      ]
    },
    {
      "message": "[409 Conflict] during [PUT] to [http://inventory/items/6d068c88-4721-4a96-918a-5fb53e41e3ee] [ItemClient#updateItem(Item,String)]: [ERROR: Cannot update record 6d068c88-4721-4a96-918a-5fb53e41e3ee because it has been changed (optimistic locking): Stored _version is 1211, _version of request is 1209 (23F09)]",
      "parameters": [
        {
          "key": "IDENTIFIER",
          "value": "083937391"
        }
      ]
    },
    {
      "message": "New status value \"Unknown\" is not allowed",
      "parameters": [
        {
          "key": "IDENTIFIER",
          "value": "085105144"
        }
      ]
    },
    {
      "message": "[409 Conflict] during [PUT] to [http://inventory/items/4a4ef447-d6ef-44b3-a51f-c6326c4fc67d] [ItemClient#updateItem(Item,String)]: [ERROR: Cannot update record 4a4ef447-d6ef-44b3-a51f-c6326c4fc67d because it has been changed (optimistic locking): Stored _version is 1202, _version of request is 1201 (23F09)]",
      "parameters": [
        {
          "key": "IDENTIFIER",
          "value": "083926413"
        }
      ]
    },
    {
      "message": "New status value \"Unknown\" is not allowed",
      "parameters": [
        {
          "key": "IDENTIFIER",
          "value": "084436924"
        }
      ]
    },
    {
      "message": "[409 Conflict] during [PUT] to [http://inventory/items/44bdcabd-78bf-40b0-a28b-d3d70bebaf57] [ItemClient#updateItem(Item,String)]: [ERROR: Cannot update record 44bdcabd-78bf-40b0-a28b-d3d70bebaf57 because it has been changed (optimistic locking): Stored _version is 1259, _version of request is 1258 (23F09)]",
      "parameters": [
        {
          "key": "IDENTIFIER",
          "value": "084583014"
        }
      ]
    },
    {
      "message": "New status value \"Unknown\" is not allowed",
      "parameters": [
        {
          "key": "IDENTIFIER",
          "value": "084994419"
        }
      ]
    },
    {
      "message": "New status value \"Unknown\" is not allowed",
      "parameters": [
        {
          "key": "IDENTIFIER",
          "value": "085189978"
        }
      ]
    },
    {
      "message": "New status value \"Unknown\" is not allowed",
      "parameters": [
        {
          "key": "IDENTIFIER",
          "value": "084644331"
        }
      ]
    },
    {
      "message": "[409 Conflict] during [PUT] to [http://inventory/items/fb03ccf9-4e69-470e-aab4-9ab0bf6dad8e] [ItemClient#updateItem(Item,String)]: [ERROR: Cannot update record fb03ccf9-4e69-470e-aab4-9ab0bf6dad8e because it has been changed (optimistic locking): Stored _version is 1221, _version of request is 1220 (23F09)]",
      "parameters": [
        {
          "key": "IDENTIFIER",
          "value": "083958216"
        }
      ]
    }
  ],
  "total_records": 57
}