Overview

This document contains the results of testing concurrent Data Import with file splitting feature for MARC Bibliographic records in the Quesnelia release.

The purpose for this test is to define how concurrent DI affect duration of DI jobs on the central tenant and to check possible issues during smoke test with 50k DI Create job running concurrently on all 3 tenants.

Compared with Data Import Creates + Updates Poppy pcp1

Ticket:  PERF-837 - Getting issue details... STATUS

Summary

• Data Import duration of 10k and 25k jobs multiplies in correspondence with number of concurrent DI jobs. Duration doubles with 2 concurrent jobs (1 job for each tenant) and 3 times longer for 3 concurrent jobs. This trend is consistent across tenants. The same as in Poppy.
• Running smoke test 50k file with DI create job revealed issue with module mod-permission which spiked and led to 4 split jobs got SNAPSHOT_UPDATE_ERROR at the beginning of the test on third fs07000002 tenant. No error messages in UI. Reason - No action.
• Average CPU utilization was for top two modules during DI Create jobs in mod-inventory-b - 121%, mod-quick-marc-b - 92%, Update jobs mod-inventory-b - 150%, mod-quick-marc-b - 92%. Modules used less resources in quesnelia. More information can be found in Service CPU Utilization tab.
• Memory consumption: mod-inventory-b - 87%, mod-data-import-b - 80%, mod-source-record-storage-b - 73%, mod-permissions-b - 71% which is slightly less than in Poppy
• RDS CPU utilization was 88% for all jobs that is 10% less than in Poppy

• Create jobs DB connections for 2 tenants  - 990, for 3 tenants - 1100, Update jobs DB connections for 2 tenants  - 950, for 3 tenants - 1000

• Top long query was SELECT jsonb FROM fs07000002_mod_permissions.permissions - rows 4275 rows/sec, average latency 1579 ms/call

Test Runs 

Test Results

Duration of DI jobs grow proportionally to file size.

Smoke Test finished successfully for 3 concurrent DI Create jobs of 50K each.

Comparison

Compared with Data Import Creates + Updates Poppy pcp1

Service CPU Utilization

Service Memory Utilization

MSK tenant cluster

Disk usage by broker

CPU (User) usage by broker

DB CPU Utilization

RDS CPU utilization was 88% for all jobs that is 10% less than in Poppy

DB Connections

Create jobs DB connections for 2 tenants  - 990, for 3 tenants - 1100 

Update jobs DB connections for 2 tenants  - 950, for 3 tenants - 1000

DB load

Top SQL

SELECT jsonb FROM fs07000002_mod_permissions.permissions 

Errors & Exceptions

During smoke test with 50k on 3 tenants SNAPSHOT_UPDATE_ERROR was observed on third fs07000002 tenant. No error messages in UI. Reason - No action on four split jobs at the beginning.

Infrastructure

PTF -environment qcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	2731

• Number of records in DB:
  •  fs09000000
    • instances - 25901331
    • items - 27074913
    • holdings - 25871735
  • fs07000001
    • nstances - 10100620
    • items - 1484850
    • holdings - 10522266
  • fs07000002
    • nstances - 1161275
    • items - 1153548
    • holdings - 1153548
• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• Open Search ptf-test
  • version OpenSearch_2_7_R20240502 
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Storage type - EBS
    • EBS volume size (GiB) - 500
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3

Methodology/Approach

• DI tests were started from UI concurrently with 1 job on each tenant, fs09000000 first and then on fs07000001 so in total two jobs on two tenants.
• Then 1 job on three tenants concurrently with several seconds delay - started with tenant fs09000000, second tenant -fs07000001 and third tenant - fs07000002. 
• DI Create jobs executed first. Then DI Update jobs.

To get instance ids and job durations used these queries

select id
-- from fs09000000_mod_inventory_storage.instance
-- from fs07000001_mod_inventory_storage.instance
-- from fs07000002_mod_inventory_storage.instance

-- limit 1
-- To get instance ids for 10k Update on main tenant
-- where creation_date > '2024-06-18 06:40:14' and creation_date < '2024-06-18 06:45:15.549'
-- To get instance ids for 25k Update on main tenant
-- where creation_date > '2024-06-18 06:50:00' and creation_date < '2024-06-18 07:00:43.192'


-- To get instance ids for 10k Update on 0701 tenant
-- where creation_date > '2024-06-18 07:07:51' and creation_date < '2024-06-18 07:12:53.209'
-- To get instance ids for 25k Update on 0701 tenant
-- where creation_date > '2024-06-18 07:17:04' and creation_date < '2024-06-18 07:28:19.22'

-- To get instance ids for 10k Update on 0702 tenant
-- where creation_date > '2024-06-18 07:33:07' and creation_date < '2024-06-18 07:38:06.762'
-- To get instance ids for 25k Update on 0702 tenant
-- where creation_date > '2024-06-18 07:42:50' and creation_date < '2024-06-18 07:54:12.09'


select file_name,total_records_in_file,started_date,completed_date, completed_date - started_date as duration ,status,error_status
-- from fs09000000_mod_source_record_manager.job_execution
-- from fs07000002_mod_source_record_manager.job_execution
-- from fs07000002_mod_source_record_manager.job_execution
where subordination_type = 'COMPOSITE_PARENT'
-- where started_date > '2024-06-13 14:47:54' and completed_date < '2024-06-13 19:01:50.832'
order by started_date desc
limit 10