Data Import MARC Authorities [Poppy]

Overview

In the scope of PERF-698 - Getting issue details... STATUS  - * Poppy Release | Data import | Create MARC authority Records it's needed to run tests to answer questions: 

  • Determine the time it takes to complete Data-import on PCP1 env
  • Determine the main modules that are involved in the process (if obvious or if known)
  • Test specific settings or items or scenarios:
    Check-in and Checkout (CICO) is in progress and there are 5 concurrent users, and run DI for 1K - 5K- 10K-25K-50K
    Test concurrent DI jobs with multi-tenants in the same cluster.

Summary

  • 1K, 5K, 10K, 50K data import jobs finished successfully without errors or issues. 22K completed with 67 errors, problem in data, because same errors were on 3 tenants.
  • General summary. Data Import MARC Authorities work on Poppy like on the previous releases. DI duration is about the same and resource utilization has the same behavior as in the previous releases. 
  • The environment is stable and can process 5 DI jobs in parallel on 3 tenants.

Test 1. DI Primary tenant 1k-5K-10K-22K-50K

  • DI performance for:
    • 1K is faster by Orchid but on the same level as Morning Glory and Nolana
    • 5K and 10K DI duration is about the same as Orchid, Morning Glory, and Nolana
    • 23K and 50K DI duration is longer on Poppy in comparison to Orchid, but faster than on Morning Glory and Nolana
  • Resource Utilization during data imports was stable during the test. 
    • Maximal CPU consumption was for mod-quick-marc service about 55-60% of CPU and spiked up to 80% on mod-data-import-b, when the 50K records file was uploading;
    • No memory leaks;
    • RDS utilization was about 95%;

Test 2. DI Primary tenant 1k-5K-10K-22K-50K + CI/CO 5VU

  • Average response time is slower from 2 up to 18% for small files(1K, 5K, and 10K ), but for big files (23K, 50K);
  • The duration of DI during CICO is similar to results "without CICO";
  • Memory and CPU utilization were without problems. The most consuming CPU service is mod-inventory-b about 80%. RDS utilization was about 95%;

Test 3. Multitenant DI 50K-22K-10K-5K-1K

All DI finished successfully(except 23K, DI file has 67 broken records), with no memory leaks. 

Test Runs & Results

Job Profile Default - Create SRS MARC Authority

Test #

# of records 

% creates

File

DI duration 
Morning Glory

DI
duration
Nolana

DI
duration 
Orchid

DI
duration 
Poppy

11,0001001k_marc_authority.mrc?api=v2  24 s27 s41 sec29 sec
25,000100 LC_SUBJ_msplit00000000.mrc?api=v21 min 21 s1 min 15 s1min 21s1 min 38 sec
310,000100msplit00000000.mrc?api=v2 2 min 32 s2 min 31 s2min 53s2 min 53 sec
422778100 msplit00000013.mrc?api=v211 min 14 s7 min 7 s5 min 42s6 min 24 sec
550,00010050000_authorityrecords.mrc?api=v222 min11 min 24 s11 min 11s13 min 48 sec

 Test with CICO 5 concurrent users and DI 1K and 5K and 50K 


CICO, Median time without
DI
CICO, 95% time without
DI
Check-In516 ms567 ms
Check-Out910 ms2094 ms 

Test #

# of records 

DI Duration

with CICO

CI time Avg
without

Baseline CI delta

CI time 95th pct

Baseline CI delta

CO time Avg

Baseline CO Avg

Delta

CO time 95th pct

Baseline CO delta

11,00035 sec0.525+2%0.576+1%1.078+18%1.326-27%
25,0001 min 41 sec0.513-1%0.612+8%0.9-0.001%1.019-52%
310,0003 min 4 sec0.581+12%0.685+20%1.016+11%1.321-27%
422,7786 min 32 sec0.598+16%1.542+171%1.244+36%1.729-18%
550,00013 min 48 sec0.671+30%1.953+244%1.51+65%2.09-0.01%

Multitenant testing

File with 50K, 23K, 10K, 5K, 1K loaded sequences and launched on Tenant1, Tenant2, and Tenant3 

TestNum of recordsTenant 1
pcp1-00
duration
Tenant 2
pcp1-01
duration
Tenant 3
pcp1-02
duration
1.100066 min75 min74 min
2.500067 min76 min75 min
3.10,00067 min76 min74 min
4.22,77867 min*73 min*71 min*
5.50,00062 min56 min54 min

*Completed with errors

 Error detailes

And on all of the tenants test 4 finished with 67 errors, with the same messages



Error message:
io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"JSON parse error: Cannot deserialize value of type `java.util.UUID` from String \"76c74801-afec-45a0-aad7-3ff23591e147 76c74801-afec-45a0-aad7-3ff23591e147\": UUID has to be represented by standard 36-char representation","type":"HttpMessageNotReadableException","code":"validation"}],"total_records":1}



Resource utilization

Test 1. DI Primary tenant 1k-5K-10K-22K-50K

Data-import 17:43 → 1000 records import; 17:50 → 5000 records import; 17:55-18:00 →10K records import; 18:05- 18:10 → 23K records import; 18:20- 18:35→ 50K records import.

Service CPU Utilization

CPU Utilization during data-imports was stable during the test, mod-quick-marc-b consumed about 55-60% of CPU and Spikes up to 80% on mod-data-import-b, but we saw the same behavior when large files were uploaded for data-import.
CPU utilization after tests returned to its before-test statements.

Memory utilization


RDS utilization was about 95%

Database load 

Top SQL


Test 2. DI Primary tenant 1k-5K-10K-22K-50K + CI/CO 5VU

CICO Responce time graph 
Data import 1k- 23:53,15 - 23:53,50; 5k- 23:57,17 - 23:59; 10k- 00:04 - 00:07; 23k- 00:09 - 00:15; 3k- 00:18 - 00:32; 

Service CPU Utilization

CPU Utilization during data imports was stable during the test, and after the tests finished returned to its before-test statements.

Memory utilization was stable during the tests.

RDS utilization was about 95%