Table of Contents |
---|
Overview
...
- DE jobs perform dramatically better in Quesnelia release if to compare with Poppy. No issues with token. All jobs with file's volume 1000, 100k, 500k records completed successfully.
- The improvement varies from file size or job profile - from 4 to 9 times better duration. Additional test was conducted with job profile prepared by script to check consistency of results.
- The Average CPU utilization for mod-data-export depends on file size and job profile. Exporting 100k records- 63% in Default and 92% in custom job profile. Exporting 500k - Default instances export job profile- 434%, srs - holdings and items- 296%
- Average Memory consumption for mod-data-export was close to 100%. Almost the same as in Poppy release.
- Average DB utilization - 17% with 100k and 33% with 500k. DB connections - 1360 instead of 200 in Poppy.
Recommendations & Jiras
- Consider increasing of cpu for mod-data-export module to smoothen spikes with big size files.
Test Results
This table contains durations for jobs with 3 job profiles.
...
3 files were prepared with query: SELECT id FROM [tenant_id]_mod_inventory_storage.instance where jsonb->>'source'='MARC' LIMIT 1000|100000|500000;
...
Infrastructure
PTF -environment qcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...