/
Data Import MARC BIB + CI/CO (Ramsons) [NON-ECS]

Data Import MARC BIB + CI/CO (Ramsons) [NON-ECS]

Owned by Stanislav Nehrii
Last updated: yesterday at 9:50 AM

Overview In progress

  • This document contains the results of testing Check-in/Check-out and Data Import for MARC Bibliographic records on the Ramsons[ECS] release environment. 

PERF-969 - Getting issue details... STATUS  

Summary

  • Data Import with Check In/Check Out tests finished successfully with PTF - Create 2 and PTF - Updates Success - 2 job profiles with files 5K, 10K, 25K, 50K, 100K records.
  • Comparing with previous testing results Quesnelia and Ramsons releases

Test Runs 

Test №ScenarioTest ConditionsResults

1		DI MARC Bib Create5K, 10K, 25K, 50K, 100K consequentially
Completed
CICO 8 users

2		DI MARC Bib Update5K, 10K, 25K, 50K, 100K consequentially


Completed

CICO8 users

Test Results

This table contains durations for Data Import with Check-in/Check-out. 

ProfileMARC File

DI Duration

Ramsons (hh:mm:ss)

Check In, Check Out Response time (8 users)

Ramsons

CI Average secCO Average sec

DI MARC Bib Create

(PTF - Create 2)

5K.mrc


10K.mrc


25K.mrc


50K.mrc


100K.mrc


DI MARC Bib Create

(PTF - Updates Success - 2)

5K.mrc


10K.mrc


25K.mrc


50K.mrc


100K.mrc


Check-in/Check-out without Data Import

ScenarioLoad levelRequestResponse time, sec
Quesnelia
95 percaverage
Circulation Check-in/Check-out
(without Data import)		8 usersCheck-in

Check-out

Comparison

This table contains DI durations with CICO comparison between Quesnelia and Ramsons releases.

ProfileMARC FileDI Duration

DI Delta  
Quesnelia/Ramsons

with CI/CO
(hh:mm:ss)

Check In, Check Out
Response time
(8 users)		Check In, Check Out
Response time
(8 users)		Delta, %

without CI/CO

with CI/COQuesneliaRamsonsQuesnelia/RamsonsQuesnelia/Ramsons
QuesneliaRamsonsQuesneliaRamsonsCI Average secCO Average secCI Average secCO Average secCICO
DI MARC Bib Create (PTF - Create 2)5K.mrc

00:03:2100:04:1100:00:500.8311.3571.01221.54%47.38%
10K.mrc00:04:1400:05:1000:06:5100:06:3900:00:120.8451.410.951.8812.43%33.33%
25K.mrc00:09:4100:10:3000:12:4100:16:1300:03:320.7191.3331.182.2564.12%68.79%
50K.mrc00:18:1800:15:4300:23:1900:29:4700:06:280.6911.3271.112.2960.64%72.57%
100K.mrc00:38:3600:31:5100:51:2400:56:0000:04:360.6641.3351.62.4140.96%79.78%
DI MARC Bib Update (PTF - Updates Success - 6)5K.mrc

00:04:1200:06:1900:02:070.7641.4580.992.229.58%50.89%
10K.mrc00:05:5900:07:1000:08:1500:12:1000:03:550.7791.3771.12.541.21%81.55%
25K.mrc00:19:5200:19:0300:20:3800:24:3100:03:530.7551.4011.042.137.75%49.89%
50K.mrc00:37:5300:38:5300:43:0600:49:5300:06:470.751.4441.032.237.33%52.35%
100K.mrc01:14:0001:23:0001:29:0901:48:0000:18:510.731.45812.136.99%44.03%

Detailed CICO response time comparison without DI

ScenarioLoad levelRequestResponse time, sec
Poppy		Response time, sec
Quesnelia
95 percaverage95 percaverage
Circulation Check-in/Check-out (without Data import)8 usersCheck-in



Check-out



Test №1

Resource utilization

 Resource utilization table
Create 100k



ModuleCPU
ModuleRAM
mod-inventory-b7.09
mod-data-import-b79.88
mod-inventory-storage-b4.75
mod-oa-b76.71
okapi-b3.78
mod-dcb-b63.23
mod-source-record-storage-b3.64
mod-pubsub-b61.84
mod-pubsub-b3.27
mod-inventory-b61.03
mod-source-record-manager-b1.92
okapi-b46.94
mod-search-b1.61
mod-search-b46.46
mod-quick-marc-b0.76
mod-users-b41.64
nginx-okapi0.68
mod-quick-marc-b39.2
mod-authtoken-b0.67
mod-source-record-storage-b37.03
mod-di-converter-storage-b0.52
mod-feesfines-b35.11
mod-users-b0.52
mod-di-converter-storage-b32.02
mod-circulation-storage-b0.44
mod-entities-links-b31.76
mod-circulation-b0.42
mod-patron-blocks-b31.18
mod-configuration-b0.17
mod-configuration-b30.47
mod-entities-links-b0.14
mod-users-bl-b28.52
mod-feesfines-b0.13
mod-source-record-manager-b28.43
mod-dcb-b0.12
mod-inventory-storage-b28.15
mod-users-bl-b0.07
mod-circulation-b23.55
mod-patron-blocks-b0.07
mod-patron-b22.89
pub-okapi0.05
mod-authtoken-b21.09
mod-data-import-b0.05
mod-circulation-storage-b20.46
mod-oa-b0.05
edge-patron-b16.96
edge-patron-b0.04
nginx-okapi4.85
mod-patron-b0.04
pub-okapi4.69
pub-edge0
pub-edge4.35

Response times

 Response times

Service CPU Utilization


Service Memory Utilization


DB CPU Utilization


DB Connections


DB load

Top SQL-queries


Top applications


Test №2

Resource utilization

 Resource utilization table
Update 100k



ModuleCPU
ModuleRAM
mod-inventory-b7.8
mod-data-import-b89.05
mod-source-record-storage-b6.67
mod-oa-b77.18
mod-inventory-storage-b5.65
mod-dcb-b65.35
okapi-b5.46
mod-inventory-b65.01
mod-pubsub-b3.27
mod-pubsub-b62.44
mod-source-record-manager-b1.59
okapi-b47.29
nginx-okapi0.95
mod-source-record-storage-b45.72
mod-search-b0.82
mod-search-b44.62
mod-di-converter-storage-b0.79
mod-users-b42.13
mod-circulation-storage-b0.7
mod-quick-marc-b39.38
mod-authtoken-b0.64
mod-feesfines-b35.38
mod-quick-marc-b0.53
mod-source-record-manager-b33.99
mod-users-b0.43
mod-di-converter-storage-b32.89
mod-circulation-b0.34
mod-entities-links-b32.09
mod-configuration-b0.16
mod-patron-blocks-b31.25
mod-feesfines-b0.14
mod-configuration-b30.65
mod-dcb-b0.12
mod-users-bl-b29.12
mod-entities-links-b0.11
mod-inventory-storage-b28.55
mod-patron-blocks-b0.07
mod-circulation-b24.71
mod-data-import-b0.06
mod-patron-b23.22
pub-okapi0.05
mod-circulation-storage-b22.25
mod-oa-b0.04
mod-authtoken-b21.35
mod-users-bl-b0.04
edge-patron-b16.91
mod-patron-b0.04
nginx-okapi4.9
edge-patron-b0.03
pub-okapi4.74
pub-edge0.002
pub-edge4.46

Response times

 Response times

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

                                                                                                                     

Top SQL-queries


Appendix

Infrastructure

PTF -environment rcon
  • rcon 12 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 
  • 1 instance of db.r6g.xlarge database instance: Writer instance
  • MSK fse-test
      • 4 kafka.m7g.xlarge brokers in 2 zones (2 brokers per zone)

      • Apache Kafka version 3.7.x, metadata mode - KRaft

      • EBS storage volume per broker 300 GiB

      • auto.create.topics.enable=true
      • log.retention.minutes=480
      • default.replication.factor=2
      • revision - 26
  • OpenSearch 2.13 ptf-test cluster

    • r6g.2xlarge.search 4 data nodes

    • r6g.large.search 3 dedicated master nodes

DB table records size:

  • [tenant]
    • instances -            1163924
    • items -                  2091901
    • holding records - 1348036


Methodology/Approach

DI tests scenario (DI MARC Bib Create and Update) were started from UI with delay.

Test runs:

  • Test 1: Manually tested 5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause) records files, DI (DI MARC Bib Create ) started on College tenant(cs00000int_0001) only, and CICO with 8 users on background.
  • Test 2: Manually tested 5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause) records files, DI (DI MARC Bib Update) started on College tenant(cs00000int_0001) only, and CICO with 8 users on background.