/
Data Import test report QCON (Quesnelia)[ECS]

Data Import test report QCON (Quesnelia)[ECS]

Owned by Stanislav Nehrii
Last updated: Jun 01, 2024

Overview

This document contains the results of testing Data Import for MARC Bibliographic records at Quesnelia release [ECS].
Ticket: https://folio-org.atlassian.net/browse/PERF-858 on QCON environment.

Summary

  • All Data-imports jobs finished successfully without errors.

  • The PTF - Updates Success - 2 profile(based on qcp1: PTF - Updates Success - 6 ) was created for the QCON Quesnelai release on tenant: cs00000int_0001.

  • DI duration growth correlates to the number of records imported. 

  • No memory leak is suspected for DI modules.

  • Approximate DB CPU usage is close to 95% and this number goes for all jobs with files of more than 10k records. 

Comparison with previous testing results Data Import test report (Quesnelia)[non-ECS]

  • Duration for Data-import create better for for files with smaller size and the same for file with 500k records.

  • Duration for Data-import update better for for files with smaller size and slower on 20% for files with 100k and 500k records.

  • Services CPU utilization, Service memory utilization, and DB CPU utilization have the same utilization trend and values as in the Poppy release.

 

Results

Test #

Data-import test

Profile

Duration

Poppy 

Duration

Quesnelia (qcp1)

Duration

Quesnelia (qcon)

Difference, %

Results

Test #

Data-import test

Profile

Duration

Poppy 

Duration

Quesnelia (qcp1)

Duration

Quesnelia (qcon)

Difference, %

Results

1.

1k MARC BIB Create

PTF - Create 2

39 sec

54 sec

31 sec

-42%

Completed

2.

5k MARC BIB Create

PTF - Create 2

2 min 22 sec

3 min 20 sec

 

 

Not tested

3.

10k MARC BIB Create

PTF - Create 2

4 min 29 sec

6 minutes

4 min 14 sec

-29%

Completed

4.

25k MARC BIB Create

PTF - Create 2

10 min 38 sec

13 min 41 sec

9 min 41 sec

-29%

Completed 

5.

50k MARC BIB Create

PTF - Create 2

20 min 26 sec

21 min 59 sec

18 min 18 sec

-16%

Completed 

6.

100k MARC BIB Create

PTF - Create 2

2 hours 46 min

Cancelled

40 min 16 sec

38 min 36 sec

-4%

Completed

7.

500k MARC BIB Create

PTF - Create 2

Not Tested

3 hours 27 min

3 hours 30 min

+1.84%

Completed

8.

1k MARC BIB Update

PTF - Updates Success - 6

34 sec

(PTF - Updates Success - 1)

1 min 59 sec

44 sec

-63%

Completed

9

2k MARC BIB Update

PTF - Updates Success - 6

1 min 09 sec (PTF - Updates Success - 1)

2 min 43 sec

 

 

Not tested

10

5k MARC BIB Update

PTF - Updates Success - 6

2 min 31 sec (PTF - Updates Success - 1)

7 min 10 sec

 

 

Not tested

11

10k MARC BIB Update

PTF - Updates Success - 6

5 min 13 sec (PTF - Updates Success - 1)

10 min 27 sec

5 min 59 sec

-42%

Completed

12

25k MARC BIB Update

PTF - Updates Success - 6

12 min 27 sec (PTF - Updates Success - 1)

23 min 16 sec

19 min 52 sec

-14%

Completed

13

50k MARC BIB Update

PTF - Updates Success - 6

Not tested

40 min 52 sec

37 min 53 sec

-7%

Completed

14

100k MARC BIB Update

PTF - Updates Success - 6

Not tested

1 hrs 2 min

1 hrs 14 min

+19%

Completed

15

500k MARC BIB Update

PTF - Updates Success - 6

Not tested

5 hrs 31 min

6 hrs 39 min

+21%

Completed

Service CPU Utilization 

MARC BIB CREATE

Tests #1-7

1k, 10k, 25k, 50k, 100k, 500k records

CPU utilization for all modules returned by default numbers after all tests. Average for mod-inventory-b - 90%, mod-inventory-storage-b - 27%, mod-source-record-storage-b - 15%, mod-source-record-manager-b - 14%, mod-di-converter-storage-b - 51%, , mod-data-import - 350% spike for 500k job(same behavior on Poppy version).

 

image-20240528-101829.png

 

MARC BIB UPDATE

Tests #8-15

1k, 10k, 25k, 50k, 100k, 500k records

 

image-20240529-114453.png

 

Memory Utilization

No memory leak is suspected for DI modules.

MARC BIB CREATE

Tests #1-7

1k, 10k, 25k, 50k, 100k, 500k records

 

image-20240528-102909.png

 

MARC BIB UPDATE

Tests #8-15

1k, 10k, 25k, 50k, 100k, 500k records

image-20240529-115203.png

 

RDS CPU Utilization 

MARC BIB CREATE

Average 90% for DI jobs with more than 10k records for Create and Update profiles

 

image-20240528-102043.png

 

MARC BIB UPDATE

image-20240529-114729.png

 

RDS Database Connections

MARC BIB CREATE
DB connections was 1400 in average

image-20240528-102207.png

 

MARC BIB Update

DB connections was 1400 in average

image-20240529-114930.png

 

Average active sessions (AAS)

MARC BIB CREATE

image-20240528-103737.png

Top SQL

image-20240528-103927.png

 

MARC BIB UPDATE

 

image-20240529-115801.png

Top SQL

image-20240529-115947.png

 

OpenSearch Service

Cluster status was green during the tests
Master nodes
1. CPU utilization MasterCPUUtilization
https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#metricsV2?graph=~(metrics~(~(~'AWS2fES~'MasterCPUUtilization~'DomainName~'fse~'ClientId~'054267740449))~view~'timeSeries~stacked~false~region~'us-east-1~title~'CPU20utilization2028Percent*29~period~60~stat~'Maximum~yAxis~(left~(showUnits~false)))

MARC BIB Create
CPU utilization was 20% in average

image-20240531-105522.png

MARC BIB Update

CPU utilization was 20% in average

image-20240531-105644.png

 


Data nodes
1. CPU utilization. CPUUtilization
https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#metricsV2?graph=~(metrics~(~(~'AWS2fES~'CPUUtilization~'DomainName~'fse~'ClientId~'054267740449))~view~'timeSeries~stacked~false~region~'us-east-1~title~'CPU20utilization2028Percent*29~period~60~stat~'Maximum~yAxis~(left~(showUnits~false)))

2. Maximum memory utilization (SysMemoryUtilization)
https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#metricsV2?graph=~(metrics~(~(~'AWS2fES~'SysMemoryUtilization~'DomainName~'fse~'ClientId~'054267740449))~view~'timeSeries~stacked~false~region~'us-east-1~title~'Maximum20memory20utilization2028Percent29~period~60~stat~'Maximum~yAxis~(left~(showUnits~false)))


MARC BIB Create
CPU utilization was 99% in average

image-20240528-124808.png

Maximum memory utilization was 92% in average

image-20240531-104655.png

 

MARC BIB Update

CPU utilization was 99% in average

image-20240529-121247.png

Maximum memory utilization was 94% in average

image-20240531-104919.png

 

Managed Streaming for Apache Kafka

CPU (User) usage by broker

MARC BIB Create

 

image-20240528-122500.png

MARC BIB Update

image-20240529-120826.png

 

Appendix

Infrastructure

  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

  • 1 instance of db.r6.xlarge database instance: Writer instance

  • OpenSearch

    • domain: fse

    • Number of nodes: 9

    • Version: OpenSearch_2_7_R20240502

  • MSK - tenat

    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

    • Kafka consolidated topics enabled

     

Methodology

  1. Pregenerated files were used for DI Create job profile

    • 1K, 10K, 25K, 50K, 100K and 500K files.

  2. Run DI Create on a single tenant(cs00000int_0001) one by one with the delay with files using PTF - Create 2 profile.

  3. Prepare files for DI Update with the Data export app, using previously imported items

  4. Run DI Update on a single tenant(cs00000int_0001) one by one with the delay with prepared files using PTF - Update Success 2 profile

  • 1K, 10K, 25K, 50K, 100K and 500K files.

  1. Data-import durations were obtained from DB using SQL query

select file_name,started_date,completed_date, completed_date - started_date as duration ,status from cs00000int_0001_mod_source_record_manager.job_execution order by started_date desc limit 1000;