Table of Contents |
---|
...
- DE jobs perform dramatically better in Quesnelia release if to compare with Poppy. No issues with token. All jobs with file's volume 1000, 100k, 500k records completed successfully.
- The improvement varies from file size or job profile - from 4 to 9 times better duration. Additional test was conducted with job profile prepared by script to check consistency of results.
- The Average CPU utilization for mod-data-export depends on file size and job profile. Exporting 100k records- 63% in Default and 92% in custom job profile. Exporting 500k - Default instances export job profile- 434%, srs - holdings and items- 296%
- Average Memory consumption for mod-data-export was close to 100%. Almost the same as in Poppy release.
- Average DB utilization - 17% with 100k and 33% with 500k. DB connections - 1360 instead of 200 in Poppy.
...
Profile | CSV File | DE Duration/Status Poppy 2 set | DE Duration/Status Quesnelia | DE Duration, DELTA Poppy/Quesnelia | ||
---|---|---|---|---|---|---|
Result | Status | Result | Status | hh:mm:ss | ||
DE MARC Bib (Default instances export job profile) | 1kDE.csv | 00:00:08 | COMPLETED | 00:00:02 | COMPLETED | 00:00:06 - 4 times improvement |
100kDE.csv | 00:15:36 | COMPLETED | 00:02:17 | COMPLETED | 00:13:19 - 7 times improvement | |
500kDE.csv | 00:57:25 | FAIL | 00:05:10 | COMPLETED | ||
DE MARC Bib (srs - holdings and items) | 1kDE.csv | 00:00:29 | COMPLETED | 00:00:04 | COMPLETED | 00:00:25 - 7 times improvement |
100kDE.csv | 00:47:23 | COMPLETED | 00:05:13 | COMPLETED | 00:42:10 - 9 times improvement | |
500kDE.csv | 04:11:09 | FAIL | 00:08:58 | COMPLETED |
Instance CPU Utilization
Service CPU Utilization
Expand | ||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||
Default instances export job profile with 500k file
srs - holdings and items with 500k file
|
...
TOP 20 modules
|
...
For Default instances export job profile with 100k file used 92% , during exporting 500k file - 434%.
For job profile "Export for Data Import updates" (created by script) - exporting with 100k - 33%, 500k - 202%.
For srs - holdings and items job profile - 100k - 63%, 500k - 296%.
Memory Utilization
Expand | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Mod-data-export-b
For Default instances export job profile with 100k file used 92% , during exporting 500k file - 434%.
For job profile "Export for Data Import updates" (created by script) - exporting with 100k - 33%, 500k - 202%.
For srs - holdings and items job profile - 100k - 63%, 500k - 296%.
Memory Utilization
Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
TOP 20 modules srs - holdings and items with 500k file
|
This graph contains DE related modules.
...
3 files were prepared with query: SELECT id FROM [tenant_id]_mod_inventory_storage.instance where jsonb->>'source'='MARC' LIMIT 1000|100000|500000;
All tests were carried out sequentially with each job profile on main tenant fs09000000.
To get status and time range for export jobs the query used:
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'srs - holdings and items' ORDER BY completed_date DESC select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'Default instances export job profile' ORDER BY completed_date DESC select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'Export for Data Import updates(test)1' ORDER BY completed_date DESC |
Infrastructure
PTF -environment qcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 Data set for fs09000000
- Instances - 25606331
- Items - 26779913
- Holdings - 25576735
- MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...