| Table of Contents |
|---|
Overview
...
This document contains the results of testing Data Export (MARC BIB) on Quesnelia release with Data Export tests for 1k, 100k, 500k. Three csv files were prepared to run Data Export with Default instances export job profile and srs - holdings and items job profiles.
...
- DE jobs perform dramatically better in Quesnelia release if to compare with Poppy. No issues with token. All jobs with file's volume 1000, 100k, 500k records completed successfully.
- The improvement varies from file size or job profile - from 4 to 9 times better duration. Additional test was conducted with job profile prepared by script to check consistency of results.
- The Average CPU utilization for mod-data-export depends on file size and job profile. Exporting 100k records- 63% in Default and 92% in custom job profile. Exporting 500k - Default instances export job profile- 434%, srs - holdings and items- 296%
- Average Memory consumption for mod-data-export was close to 100%. Almost the same as in Poppy release.
- Average DB utilization - 17% with 100k and 33% with 500k. DB connections - 1360 instead of 200 in Poppy.
Recommendations & Jiras
- Consider increasing of cpu for mod-data-export module to smoothen spikes with big size files.
Test Results
This table contains durations for jobs with 3 job profiles.
...
| Profile | CSV File | DE Duration/Status Poppy 2 set | DE Duration/Status Quesnelia | DE Duration, DELTA Poppy/Quesnelia | ||
|---|---|---|---|---|---|---|
| Result | Status | Result | Status | hh:mm:ss | ||
| DE MARC Bib (Default instances export job profile) | 1kDE.csv | 00:00:08 | COMPLETED | 00:00:02 | COMPLETED | 00:00:06 - 4 times improvement |
| 100kDE.csv | 00:15:36 | COMPLETED | 00:02:17 | COMPLETED | 00:13:19 - 7 times improvement | |
| 500kDE.csv | 00:57:25 | FAIL | 00:05:10 | COMPLETED | ||
| DE MARC Bib (srs - holdings and items) | 1kDE.csv | 00:00:29 | COMPLETED | 00:00:04 | COMPLETED | 00:00:25 - 7 times improvement |
| 100kDE.csv | 00:47:23 | COMPLETED | 00:05:13 | COMPLETED | 00:42:10 - 9 times improvement | |
| 500kDE.csv | 04:11:09 | FAIL | 00:08:58 | COMPLETED | ||
...
Service CPU Utilization
| Expand | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||
Default instances export job profile with 500k file
srs - holdings and items with 500k file
|
...
TOP 20 modules
|
For Default instances export job profile with 100k file used 92% , during exporting 500k file - 434%.
For job profile "Export for Data Import updates" (created by script) - exporting with 100k - 33%, 500k - 202%.
For srs - holdings and items job profile - 100k - 63%, 500k - 296%.
Memory Utilization
| Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Mod-data-export-b
For Default instances export job profile with 100k file used 92% , during exporting 500k file - 434%.
For job profile "Export for Data Import updates" (created by script) - exporting with 100k - 33%, 500k - 202%.
For srs - holdings and items job profile - 100k - 63%, 500k - 296%.
Memory Utilization
| Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
TOP 20 modules srs - holdings and items with 500k file
|
This graph contains DE related modules.
...
3 files were prepared with query: SELECT id FROM [tenant_id]_mod_inventory_storage.instance where jsonb->>'source'='MARC' LIMIT 1000|100000|500000;
All tests were carried out sequentially with each job profile on main tenant fs09000000.
To get status and time range for export jobs the query used:
...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'srs - holdings and items' ORDER BY completed_date DESC select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'Default instances export job profile' ORDER BY completed_date DESC select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'Export for Data Import updates(test)1' ORDER BY completed_date DESC |
Infrastructure
PTF -environment qcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 Data set for fs09000000
- Instances - 25606331
- Items - 26779913
- Holdings - 25576735
- MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...