Skip to end of banner
Go to start of banner

PTF - Data Export Test Report (Quesnelia) [non-ECS]

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Overview IN PROGRES

This document contains the results of testing Data Export (MARC BIB) on Quesnelia release with Data Export tests for 1k, 100k, 500k. Three csv files were prepared to run Data Export with Default instances export job profile and srs - holdings and items job profiles.

Ticket: PERF-822 - Getting issue details... STATUS


Summary

  • DE jobs with files 1k, 100k, 500k durations has no significant changes compared with Orchid. No issues with token. DE jobs with 500k were completed with FAIL statuses in a both profiles.
  • FAIL status for DE srs - holdings and items job profile with 500k file is likely connected with too high data volume transfer to S3 bucket. It will be investigated in story created by firebird team.
  • Average CPU utilization for mod-data-export didn't exceed 18% with spikes. During 100k it was 10-12%. 
  • Average Memory consumption for mod-data-export was close to 100%. 
  • Average DB utilization - 18%. DB connections - 200. During tests spikes with 40% observed every 15 minutes.

Recommendations & Jiras

Test Results

This table contains durations for jobs with 2 job profiles. 


Comparison

This table contains durations comparison between Poppy and Quesnelia releases

Instance CPU Utilization


Service CPU Utilization


Memory Utilization


DB CPU Utilization

DB Connections


DB Load


SQL queries


Top-SQL statement: 

Errors / Additional information


Methodology/Approach

3 files were prepared with query: SELECT id FROM [tenant_id]_mod_inventory_storage.instance where jsonb->>'source'='MARC' LIMIT 1000|100000|500000;

All tests were carried out sequentially with each job profile. 

To get status and time range for export jobs the query used: 

SELECT jsonb->>'status',jsonb->>'startedDate' AS startedDate,jsonb->>'completedDate' AS completedDate
FROM [REPLACE_tenant_id_HERE]_mod_data_export.job_executions
WHERE jsonb->>'jobProfileName'='[REPLACE_WITH_DE_JOB_HERE]'
ORDER BY jsonb->>'startedDate' desc LIMIT 10;

Infrastructure

PTF -environment qcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 2 database  instances, writer/reader

    NameMemory GIBvCPUsmax_connections

    db.r6g.xlarge

    		32 GiB4 vCPUs2731
  • MSK tenant
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

QCP1 modules



  • No labels