Table of Contents |
---|
Overview
...
- DE jobs perform dramatically better in Quesnelia release if to compare with Poppy. No issues with token. All jobs with file's volume 1000, 100k, 500k records completed successfully.
- The improvement varies from file size or job profile - from 4 to 9 times better duration. Additional test was conducted with job profile prepared by script to check consistency of results.
- The Average CPU utilization for mod-data-export depends on file size and job profile. Exporting 100k records- 63% in Default and 92% in custom job profile. Exporting 500k - Default instances export job profile- 434%, srs - holdings and items- 296%
- Average Memory consumption for mod-data-export was close to 100%. Almost the same as in Poppy release.
- Average DB utilization - 17% with 100k and 33% with 500k. DB connections - 1360 instead of 200 in Poppy.
...
3 files were prepared with query: SELECT id FROM [tenant_id]_mod_inventory_storage.instance where jsonb->>'source'='MARC' LIMIT 1000|100000|500000;
All tests were carried out sequentially with each job profile on main tenant fs09000000.
To get status and time range for export jobs the query used:
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'srs - holdings and items' ORDER BY completed_date DESC select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'Default instances export job profile' ORDER BY completed_date DESC select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'Export for Data Import updates(test)1' ORDER BY completed_date DESC |
Infrastructure
PTF -environment qcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 Data set for fs09000000
- Instances - 25606331
- Items - 26779913
- Holdings - 25576735
- MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...