/
Data Import MARC Authorities [Poppy]

Data Import MARC Authorities [Poppy]

Nov 28, 2023

Overview

In the scope of PERF-698: [Poppy] [Data import] Create MARC authority RecordsClosed - * Poppy Release | Data import | Create MARC authority Records it's needed to run tests to answer questions: 

  • Determine the time it takes to complete Data-import on PCP1 env

  • Determine the main modules that are involved in the process (if obvious or if known)

  • Test specific settings or items or scenarios:
    Check-in and Checkout (CICO) is in progress and there are 5 concurrent users, and run DI for 1K - 5K- 10K-25K-50K
    Test concurrent DI jobs with multi-tenants in the same cluster.

Summary

  • 1K, 5K, 10K, 50K data import jobs finished successfully without errors or issues. 22K completed with 67 errors, problem in data, because same errors were on 3 tenants.

  • General summary. Data Import MARC Authorities work on Poppy like on the previous releases. DI duration is about the same and resource utilization has the same behavior as in the previous releases. 

  • The environment is stable and can process 5 DI jobs in parallel on 3 tenants.

Test 1. DI Primary tenant 1k-5K-10K-22K-50K

  • DI performance for:

    • 1K is faster by Orchid but on the same level as Morning Glory and Nolana

    • 5K and 10K DI duration is about the same as Orchid, Morning Glory, and Nolana

    • 23K and 50K DI duration is longer on Poppy in comparison to Orchid, but faster than on Morning Glory and Nolana

  • Resource Utilization during data imports was stable during the test. 

    • Maximal CPU consumption was for mod-quick-marc service about 55-60% of CPU and spiked up to 80% on mod-data-import-b, when the 50K records file was uploading;

    • No memory leaks;

    • RDS utilization was about 95%;

Test 2. DI Primary tenant 1k-5K-10K-22K-50K + CI/CO 5VU

  • Average response time is slower from 2 up to 18% for small files(1K, 5K, and 10K ), but for big files (23K, 50K);

  • The duration of DI during CICO is similar to results "without CICO";

  • Memory and CPU utilization were without problems. The most consuming CPU service is mod-inventory-b about 80%. RDS utilization was about 95%;

Test 3. Multitenant DI 50K-22K-10K-5K-1K

All DI finished successfully(except 23K, DI file has 67 broken records), with no memory leaks. 

Test Runs & Results


Job Profile Default - Create SRS MARC Authority

Test #

# of records 

% creates

File

DI duration 
Morning Glory

DI
duration
Nolana

DI
duration 
Orchid

DI
duration 
Poppy

Test #

# of records 

% creates

File

DI duration 
Morning Glory

DI
duration
Nolana

DI
duration 
Orchid

DI
duration 
Poppy

1

1,000

100

1k_marc_authority.mrc?api=v2  

24 s

27 s

41 sec

29 sec

2

5,000

100

 LC_SUBJ_msplit00000000.mrc?api=v2

1 min 21 s

1 min 15 s

1min 21s

1 min 38 sec

3

10,000

100

msplit00000000.mrc?api=v2 

2 min 32 s

2 min 31 s

2min 53s

2 min 53 sec

4

22778

100

 msplit00000013.mrc?api=v2

11 min 14 s

7 min 7 s

5 min 42s

6 min 24 sec

5

50,000

100

50000_authorityrecords.mrc?api=v2

22 min

11 min 24 s

11 min 11s

13 min 48 sec

 Test with CICO 5 concurrent users and DI 1K and 5K and 50K 

 

CICO, Median time without
DI

CICO, 95% time without
DI

 

CICO, Median time without
DI

CICO, 95% time without
DI

Check-In

516 ms

567 ms

Check-Out

910 ms

2094 ms 

Test #

# of records 

DI Duration

with CICO

CI time Avg
without

Baseline CI delta

CI time 95th pct

Baseline CI delta

CO time Avg

Baseline CO Avg

Delta

CO time 95th pct

Baseline CO delta

Test #

# of records 

DI Duration

with CICO

CI time Avg
without

Baseline CI delta

CI time 95th pct

Baseline CI delta

CO time Avg

Baseline CO Avg

Delta

CO time 95th pct

Baseline CO delta

1

1,000

35 sec

0.525

+2%

0.576

+1%

1.078

+18%

1.326

-27%

2

5,000

1 min 41 sec

0.513

-1%

0.612

+8%

0.9

-0.001%

1.019

-52%

3

10,000

3 min 4 sec

0.581

+12%

0.685

+20%

1.016

+11%

1.321

-27%

4

22,778

6 min 32 sec

0.598

+16%

1.542

+171%

1.244

+36%

1.729

-18%

5

50,000

13 min 48 sec

0.671

+30%

1.953

+244%

1.51

+65%

2.09

-0.01%

Multitenant testing

File with 50K, 23K, 10K, 5K, 1K loaded sequences and launched on Tenant1, Tenant2, and Tenant3 

Test

Num of records

Tenant 1
pcp1-00
duration

Tenant 2
pcp1-01
duration

Tenant 3
pcp1-02
duration

Test

Num of records

Tenant 1
pcp1-00
duration

Tenant 2
pcp1-01
duration

Tenant 3
pcp1-02
duration

1.

1000

66 min

75 min

74 min

2.

5000

67 min

76 min

75 min

3.

10,000

67 min

76 min

74 min

4.

22,778

67 min*

73 min*

71 min*

5.

50,000

62 min

56 min

54 min

*Completed with errors

And on all of the tenants test 4 finished with 67 errors, with the same messages



Error message:
io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"JSON parse error: Cannot deserialize value of type `java.util.UUID` from String \"76c74801-afec-45a0-aad7-3ff23591e147 76c74801-afec-45a0-aad7-3ff23591e147\": UUID has to be represented by standard 36-char representation","type":"HttpMessageNotReadableException","code":"validation"}],"total_records":1}

 

 

Resource utilization


Test 1. DI Primary tenant 1k-5K-10K-22K-50K

Data-import 17:43 → 1000 records import; 17:50 → 5000 records import; 17:55-18:00 →10K records import; 18:05- 18:10 → 23K records import; 18:20- 18:35→ 50K records import.

Service CPU Utilization

CPU Utilization during data-imports was stable during the test, mod-quick-marc-b consumed about 55-60% of CPU and Spikes up to 80% on mod-data-import-b, but we saw the same behavior when large files were uploaded for data-import.
CPU utilization after tests returned to its before-test statements.

Memory utilization

 

RDS utilization was about 95%


Database load 

Top SQL

 

Test 2. DI Primary tenant 1k-5K-10K-22K-50K + CI/CO 5VU

CICO Responce time graph 
Data import 1k- 23:53,15 - 23:53,50; 5k- 23:57,17 - 23:59; 10k- 00:04 - 00:07; 23k- 00:09 - 00:15; 3k- 00:18 - 00:32; 

Service CPU Utilization

CPU Utilization during data imports was stable during the test, and after the tests finished returned to its before-test statements.

Memory utilization was stable during the tests.

RDS utilization was about 95%

Number of DB connections

Database load 

Top SQL

Test 3. Multitenant DI 50K-22K-10K-5K-1K.

Service CPU Utilization

CPU Utilization during data imports was stable during the test, except 2 spikes on mod-permission-b services to about 110%, and after the tests finished returned to its before-test statements.

Memory utilization was stable during the tests.

Maximal RDS utilization was about 96%

Number of DB connections

Database Load

Appendix

Infrastructure

PTF -environment pcp1

  • 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 2 instances of db.r6.xlarge database instances, one reader, and one writer

  • MSK tenant

    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

Modules memory and CPU parameters

Module

Task
Def. Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

pcp1-pvt

mod-entities-links

13

mod-entities-links:2.0.3

2

2592

2480

400

1440

0

1024

FALSE

mod-data-import

11

mod-data-import:3.0.3

1

2048

1844

256

1292

384

512

FALSE

mod-permissions

32

mod-permissions:6.4.0

2

1684

1544

512

1024

384

512

FALSE

mod-inventory-storage

12

mod-inventory-storage:27.0.3

2

4096

3690

2048

3076

384

512

FALSE

mod-source-record-storage

15

mod-source-record-storage:5.7.3

2

5600

5000

2048