Data Import with Check-ins Check-outs Nolana

Data Import with Check-ins Check-outs Nolana

Infrastructure

PTF -environment ncp3

  • m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 2 instances of db.r6.xlarge database instances, one reader, and one writer

  • MSK ptf-kakfa-3

    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

 

Modules memory and CPU parameters

Modules

Version

Task Definition

Running Tasks 

CPU

Memory

MemoryReservation

MaxMetaspaceSize

Xmx

Modules

Version

Task Definition

Running Tasks 

CPU

Memory

MemoryReservation

MaxMetaspaceSize

Xmx

mod-inventory

19.0.1

1

2

1024

2880

2592

512m

1814m

mod-inventory-storage

25.0.1

1-2

2

1024

2208

(1872 in MG)

1952

(1684 in MG)

512m

1440m

okapi

4.14.7

1-2

3

1024

1684

(1512 in MG)

1440

(1360 in MG)

512m

922m

mod-feesfines

18.1.0

1-2

2

128

1024

896

128

768

mod-patron-blocks

1.7.1

1-2

2

1024

1024

896

128

768

mod-pubsub

2.7.0

1-2

2

1024

1536

(1440 in MG)

1440

(1296 in MG)

512

922

mod-authtoken

2.12.0

1-2

2

512

(128 in MG)

1440

(1024 in MG)

1152

(896 in MG)

128

922

(768 in MG)

mod-circulation-storage

15.0.0

1-2

2

1024

1536

(1152 in MG)

1440

(1024 in MG)

512

896

mod-circulation

23.3.0

1

2

1024

1024

896

128

768

mod-configuration

5.9.0

1-2

2

128

1024

896

128m

768m

mod-users

19.0.0

1-2

2

258

1024

896

128m

768m

mod-remote-storage

1.7.0

1-2

2

128

1872

1684

512m

1178m

mod-data-import-cs

1.15.1

1-2

2

258

1024

896

128m

768m

mod-quick-marc

2.5.0

1-2

1

128

2288

(2098 in MG)

2176

(1920 in MG)

512m

1664m

mod-source-record-storage

5.5.2

1-2

2

1024

1536

(1440 in MG)

1440

(1296 in MG)

512m

908m

mod-data-import

2.6.1

1-2

1

256

2048

1844

512m

1292m

mod-source-record-manager

3.5.4

2-3

2

1024

4096

(2048 in MG)

3688

(1844 in MG)

512m

(800M in MG)

2048m

(1024m in MG)

MG- Morning Glory release

Front End:

  • Item Check-in (folio_checkin-7.2.0)

  • Item Check-out (folio_checkout-8.2.0)

Summary 

  • In Nolana, there is a significant improvement in the performance of data import and CheckIn/CheckOut compared to KIWI and Lotus.

  • The time of Data Import with CI/CO is even better than without CI/CO. Apparently, the reason is for the baseline test the mod-source-record-manager version was 3.5.0 for the test with CI/CO it was 3.5.4.

  • From time to time there are spikes in response time for CI/CO up to 10 seconds.

  • RDS CPU Utilization was high up to 97%

Results

 

MARC File

Profile

DI Duration

Check In (CI), Check Out (CO)

MARC File

Profile

DI Duration

Check In (CI), Check Out (CO)

 

 

 

users

CI Average (seconds)

CO Average (seconds)

5K.mrc

PTF - Create 2

00:01:56.847

8

0.817

1.417

10K.mrc

PTF - Create 2

00:03:57.455

8

0.842

1.574

25K.mrc

PTF - Create 2

00:09:24.249

8

0.882

1.641

50K.mrc

PTF - Create 2

00:19:28.933

8

0.926

1.666

100K.mrc

PTF - Create 2

00:38:44.751

8

1.021

1.862

5K.mrc

PTF - Updates Success - 1

00:02:51.525

8

0.747

1.094

10K.mrc

PTF - Updates Success - 1

00:04:57.6

8

0.541

1.026

25K.mrc

PTF - Updates Success - 1

00:13:12.53

8

0.700

1.248

50K.mrc

PTF - Updates Success - 1

00:27:39.693

8

0.700

1.199

100K.mrc

PTF - Updates Success - 1

00:48:45.335

8

0.556

1.046

Bolded numbers are the best response time of CI/CO and the smallest spikes of response time. For  10k MARC Update response time of CI/CO with data import is even better than for CI/CO with 5k MARC Update. For 100k.mrc response time is almost the same as for 10k.mrc that's why it is 2 sets of bolded numbers here.

Comparisons

The following table compares additional test results to previous release numbers and to the CICO baselines Nolana (of Check In average time 0.456s and Checkout average time 0.698s). Note that Lotus numbers are in red, Nolana numbers are in black, and Kiwi numbers are in blue.

In the Nolana version, there is a significant improvement in the performance of data import and CheckIn/CheckOut.

For the baseline test the mod-source-record-manager version was 3.5.0 for the test with CI/CO it was 3.5.4. Apparently, it is the reason why the time of Data Import with CI/CO is even better than without CI/CO.

 

Profile

Duration

KIWI (Lotus) without CICO

Duration

with CICO 8 users KIWI (Lotus)

Duration

Nolana without CICO

Duration

with CICO 8 users Nolana

CheckIn average (seconds) 

CheckOut average (seconds) 

Deviation From the baseline CICO response times

 

Profile

Duration

KIWI (Lotus) without CICO

Duration

with CICO 8 users KIWI (Lotus)

Duration

Nolana without CICO

Duration

with CICO 8 users Nolana

CheckIn average (seconds) 

CheckOut average (seconds) 

Deviation From the baseline CICO response times

5K MARC BIB Create

PTF - Create 2

5 min, 8 min (05:32.264)

(08:48.556)

 5 min

(05:48.671)

2 min 51 s

00:01:56.847

0.851

0.817

1.388

1.417

CI:  44%

CO:  51%

5K MARC BIB Update

PTF - Updates Success - 1

11 min, 13 min

(10:07.723)

7 min

06:27.143

2 min 27s

00:02:51.525

1.102

0.747

1.867

1.094

CI:  39%

CO:  36

10K MARC BIB Create

PTF - Create 2