Overview IN PROGRES
...
Ticket: Jira Legacy |
---|
server | System Jira |
---|
columnIds | issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution |
---|
columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
---|
serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
---|
key | PERF-822 |
---|
|
Summary
- DE jobs with files 1k, 100k, 500k durations has no significant changes compared with Orchidperform dramatically better in Quesnelia release if to compare with Poppy. No issues with token. DE All jobs with 500k were completed with FAIL statuses in a both profiles.
- FAIL status for DE srs - holdings and items job profile with 500k file is likely connected with too high data volume transfer to S3 bucket. It will be investigated in story created by firebird team.
- file's volume 1000, 100k, 500k records completed successfully.
- The improvement varies from file size or job profile - from 4 to 9 times better duration. Additional test was conducted with job profile prepared by script to check consistency of results.
- The Average CPU utilization for mod-data-export didn't exceed 18% with spikes. During 100k it was 10-12%. depends on file size and job profile. Exporting 100k records- 63% in Default and 92% in custom job profile. Exporting 500k - Default instances export job profile- 434%, srs - holdings and items- 296%
- Average Memory consumption for mod-data-export was close to 100%. . Almost the same as in Poppy release.
- Average DB utilization - 18%17% with 100k and 33% with 500k. DB connections - 1360 instead of 200 . During tests spikes with 40% observed every 15 minutes.in Poppy.
Recommendations & Jiras
Test Results
This table contains durations for jobs with 2 3 job profiles.
Profile | CSV File | DE Duration/Status Quesnelia |
---|
|
| Result | Status |
DE MARC Bib (Default instances export job profile) | 1kDE.csv | 00:00:02 | COMPLETED |
100kDE.csv | 00:02:17 | COMPLETED |
500kDE.csv | 00:05:10 | COMPLETED |
DE MARC Bib (srs - holdings and items) | 1kDE.csv | 00:00:04 | COMPLETED |
100kDE.csv | 00:05:13 | COMPLETED |
500kDE.csv | 00:08:58 | COMPLETED |
Export for Data Import updates(created by script) | 1kDE.csv | 00:00:04 | COMPLETED |
100kDE.csv | 00:05:08 | COMPLETED |
500kDE.csv | 00:10:41 | COMPLETED |
Comparison
This table contains durations comparison between Poppy and Quesnelia releases
Profile | CSV File | DE Duration/Status Poppy 2 set | DE Duration/Status Quesnelia | DE Duration, DELTA Poppy/Quesnelia |
---|
|
|
|
|
|
|
|
|
| Result | Status | Result | Status | hh:mm:ss |
DE MARC Bib (Default instances export job profile) | 1kDE.csv | 00:00:08 | COMPLETED | 00:00:02 | COMPLETED | 00:00:06 - 4 times improvement |
| 100kDE.csv | 00:15:36 | COMPLETED | 00:02:17 | COMPLETED | 00:13:19 - 7 times improvement |
| 500kDE.csv | 00:57:25 | FAIL | 00:05:10 | COMPLETED |
|
DE MARC Bib (srs - holdings and items) | 1kDE.csv | 00:00:29 | COMPLETED | 00:00:04 | COMPLETED | 00:00:25 - 7 times improvement |
| 100kDE.csv | 00:47:23 | COMPLETED | 00:05:13 | COMPLETED | 00:42:10 - 9 times improvement |
| 500kDE.csv | 04:11:09 | FAIL | 00:08:58 | COMPLETED |
|
Instance CPU Utilization
Service CPU Utilization
Expand |
---|
|
Default instances export job profile with 500k file Module | CPU |
---|
mod-data-export-b | 434.91 | mod-inventory-b | 10.79 | mod-source-record-manager-b | 1.74 | mod-source-record-storage-b | 1.5 | okapi-b | 0.96 | mod-users-bl-b | 0.63 | mod-authtoken-b | 0.61 | mod-inventory-storage-b | 0.39 | nginx-okapi | 0.24 | pub-okapi | 0.16 |
srs - holdings and items with 500k file Module | CPU |
---|
mod-data-export-b | 296.31 | mod-inventory-b | 12.79 | mod-source-record-manager-b | 1.85 | mod-source-record-storage-b | 1.49 | okapi-b | 1.1 | mod-authtoken-b | 0.95 | mod-users-bl-b | 0.69 | mod-inventory-storage-b | 0.54 | nginx-okapi | 0.32 | pub-okapi | 0.21 |
|
Mod-data-export-b
For Default instances export job profile with 100k file used 92% , during exporting 500k file - 434%.
For job profile "Export for Data Import updates" (created by script) - exporting with 100k - 33%, 500k - 202%.
Image Added
For srs - holdings and items job profile - 100k - 63%, 500k - 296%.
Image Added
Memory Utilization
Expand |
---|
|
Module | Memory |
---|
mod-data-export-b | 97 | mod-inventory-b | 55 | okapi-b | 42 | mod-source-record-manager-b | 42 | mod-users-bl-b | 32 | mod-source-record-storage-b | 27 | mod-authtoken-b | 25 | mod-inventory-storage-b | 14 | nginx-okapi | 5 | pub-okapi | 4 |
|
This graph contains DE related modules.
Mod-data-export-b use 97%, mod-inventory - 55%. The rest was under 50%
Image Added
All modules memory consumption graph. Here we see a lot of modules that go with more than 50% consumption.
Image Added
DB CPU Utilization
DB utilized with Default instances export job profile 100k file - 17%, 500k file - 25%. For Export for Data Import updates(created by script) job profile - 100k file - 17%, 500k file - 38%.
Image Added
For srs - holdings and items job profile - 100k file - 17%, 500k file - 33%.
Image Added
DB Connections
DB connections - 1260 in Average. High peaks go with 1360.
Image Added
DB Load
Default instances export job profile
Image Added
Export for Data Import updates(test)1 job profile
Image Added
srs - holdings and items job profile
Image Added
SQL queries
Default instances export job profile
Image Added
Export for Data Import updates(test)1 job profile
Image Added
srs - holdings and items job profile
Image Added
Top-SQL statement:
Default instances export job profile
autovacuum: VACUUM fs09000000_mod_data_export.job_executions_export_ids
INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING
srs - holdings and items job profile
autovacuum: VACUUM fs09000000_mod_data_export.job_executions_export_ids
select ie1_0.id,ie1_0.holdings_record_id,ie1_0.jsonb from v_item ie1_0 where ie1_0.holdings_record_id in ($1)
select hre1_0.id,hre1_0.instance_id,hre1_0.jsonb from v_holdings_record hre1_0 where hre1_0.instance_id=$1
INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING
Errors / Additional information
Note: inconsistency of data or duplicated records in the file with instances may lead to job failures. Zip file generates on UI side if number of records more than 100.000.
Methodology/Approach
3 files were prepared with query: SELECT id FROM [tenant_id]_mod_inventory_storage.instance where jsonb->>'source'='MARC' LIMIT 1000|100000|500000;
...
To get status and time range for export jobs the query used:
...
Expand |
---|
|
select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'srs - holdings and items' ORDER BY completed_date DESC select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions
|
...
where job_profile_name = 'Default instances export job profile' ORDER BY completed_date DESC select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'Export for Data Import updates(test)1' ORDER BY completed_date DESC |
Infrastructure
PTF -environment qcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name | Memory GIB | vCPUs | max_connections |
db.r6g.xlarge | 32 GiB | 4 vCPUs | 2731 |
- MSK tenant
QCP1 modules
Expand |
---|
|
Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize |
---|
qcp1-pvt |
|
|
|
|
|
|
|
|
| Tue Jun 04 07:31:53 UTC 2024 |
|
|
|
|
|
|
|
|
| mod-remote-storage | 4 | mod-remote-storage:3.2.0 | 2 | 4920 | 4472 | 1024 | 3960 | 512 | 512 | mod-ncip | 4 | mod-ncip:1.14.4 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-finance-storage | 4 | mod-finance-storage:8.6.0 | 2 | 1024 | 896 | 1024 | 700 | 88 | 128 | mod-agreements | 4 | mod-agreements:7.0.0 | 2 | 1592 | 1488 | 128 | 0 | 0 | 0 | mod-ebsconet | 4 | mod-ebsconet:2.2.0 | 2 | 1248 | 1024 | 128 | 700 | 128 | 256 | mod-organizations | 4 | mod-organizations:1.9.0 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | mod-consortia | 2 | mod-consortia:1.1.0 | 2 | 3072 | 2048 | 128 | 2048 | 512 | 1024 | edge-sip2 | 2 | edge-sip2:3.2.0-SNAPSHOT.209 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-serials-management | 4 | mod-serials-management:1.0.0 | 2 | 2480 | 2312 | 128 | 1792 | 384 | 512 | mod-settings | 4 | mod-settings:1.0.3 | 2 | 1024 | 896 | 200 | 768 | 88 | 128 | mod-data-import | 7 | mod-data-import:3.1.0 | 1 | 2048 | 1844 | 256 | 1292 | 384 | 512 | edge-dematic | 4 | edge-dematic:2.2.0 | 1 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-search | 4 | mod-search:3.2.0 | 2 | 2592 | 2480 | 2048 | 1440 | 512 | 1024 | mod-inn-reach | 2 | mod-inn-reach:3.2.0-SNAPSHOT.86 | 2 | 3600 | 3240 | 1024 | 2880 | 512 | 1024 | mod-tags | 4 | mod-tags:2.2.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | edge-courses | 4 | edge-courses:1.4.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-authtoken | 5 | mod-authtoken:2.15.1 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 | mod-inventory-update | 4 | mod-inventory-update:3.3.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-notify | 4 | mod-notify:3.2.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-configuration | 4 | mod-configuration:5.10.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-orders-storage | 4 | mod-orders-storage:13.7.0 | 2 | 1024 | 896 | 512 | 700 | 88 | 128 | edge-caiasoft | 4 | edge-caiasoft:2.2.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-login-saml | 4 | mod-login-saml:2.8.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-erm-usage-harvester | 4 | mod-erm-usage-harvester:4.5.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-licenses | 4 | mod-licenses:6.0.0 | 2 | 2480 | 2312 | 128 | 1792 | 384 | 512 | mod-gobi | 4 | mod-gobi:2.8.0 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | mod-password-validator | 4 | mod-password-validator:3.2.0 | 2 | 1440 | 1298 | 128 | 768 | 384 | 512 | mod-bulk-operations | 4 | mod-bulk-operations:2.0.0 | 2 | 3072 | 2600 | 1024 | 1536 | 384 | 512 | mod-fqm-manager | 4 | mod-fqm-manager:2.0.1 | 2 | 3000 | 2600 | 128 | 2048 | 384 | 512 | edge-dcb | 4 | edge-dcb:1.1.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-graphql | 5 | mod-graphql:1.12.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-finance | 4 | mod-finance:4.9.0 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | mod-erm-usage | 4 | mod-erm-usage:4.7.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-batch-print | 5 | mod-batch-print:1.1.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-copycat | 4 | mod-copycat:1.6.0 | 2 | 1024 | 512 | 128 | 768 | 88 | 128 | mod-lists | 4 | mod-lists:2.0.0 | 2 | 3000 | 2600 | 128 | 2048 | 384 | 512 | mod-entities-links | 5 | mod-entities-links:3.0.0 | 2 | 2592 | 2480 | 400 | 1440 | 0 | 1024 | mod-permissions | 8 | mod-permissions:6.5.0 | 2 | 1684 | 1544 | 512 | 1024 | 384 | 512 | pub-edge | 3 | pub-edge:2023.06.14 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 | mod-orders | 4 | mod-orders:12.8.0 | 2 | 2048 | 1440 | 1024 | 1024 | 384 | 512 | edge-patron | 4 | edge-patron:5.1.0 | 2 | 1024 | 896 | 256 | 768 | 88 | 128 | edge-ncip | 4 | edge-ncip:1.9.2 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | edge-inn-reach | 2 | edge-inn-reach:3.1.1-SNAPSHOT.45 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-users-bl | 4 | mod-users-bl:7.7.0 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 | mod-oa | 2 | mod-oa:2.1.0-SNAPSHOT.62 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-inventory-storage | 4 | mod-inventory-storage:27.1.0 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | mod-invoice | 5 | mod-invoice:5.8.0 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 | mod-user-import | 4 | mod-user-import:3.8.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-sender | 5 | mod-sender:1.12.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | edge-oai-pmh | 4 | edge-oai-pmh:2.9.0 | 2 | 1512 | 1360 | 1024 | 1440 | 384 | 512 | mod-data-export-worker | 4 | mod-data-export-worker:3.2.1 | 2 | 3072 | 2048 | 1024 | 2048 | 384 | 512 | mod-rtac | 4 | mod-rtac:3.6.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-circulation-storage | 4 | mod-circulation-storage:17.2.0 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 | mod-calendar | 4 | mod-calendar:3.1.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-source-record-storage | 4 | mod-source-record-storage:5.8.0 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | mod-event-config | 4 | mod-event-config:2.7.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-courses | 4 | mod-courses:1.4.10 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-circulation-item | 4 | mod-circulation-item:1.0.0 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 | mod-inventory | 4 | mod-inventory:20.2.0 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 | mod-email | 4 | mod-email:1.17.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-pubsub | 4 | mod-pubsub:2.13.0 | 2 | 1536 | 1440 | 1024 | 922 | 384 | 512 | mod-circulation | 4 | mod-circulation:24.2.0 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 | mod-di-converter-storage | 4 | mod-di-converter-storage:2.2.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | edge-rtac | 4 | edge-rtac:2.7.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | edge-orders | 4 | edge-orders:3.0.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-users | 5 | mod-users:19.3.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-template-engine | 4 | mod-template-engine:1.20.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-patron-blocks | 4 | mod-patron-blocks:1.10.0 | 2 | 1024 | 896 | 1024 | 768 | 88 | 128 | mod-audit | 4 | mod-audit:2.9.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | edge-fqm | 4 | edge-fqm:2.0.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-source-record-manager | 5 | mod-source-record-manager:3.9.0-SNAPSHOT.330 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | nginx-edge | 3 | nginx-edge:2023.06.14 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 | mod-quick-marc | 4 | mod-quick-marc:5.1.0 | 1 | 2288 | 2176 | 128 | 1664 | 384 | 512 | nginx-okapi | 3 | nginx-okapi:2023.06.14 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 | okapi-b | 4 | okapi:5.3.0 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 | mod-feesfines | 4 | mod-feesfines:19.1.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-invoice-storage | 4 | mod-invoice-storage:5.8.0 | 2 | 1872 | 1536 | 1024 | 1024 | 384 | 512 | mod-dcb | 5 | mod-dcb:1.1.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-service-interaction | 4 | mod-service-interaction:4.0.1 | 2 | 2048 | 1844 | 256 | 1290 | 384 | 512 | mod-data-export | 11 | mod-data-export:5.0.4 | 1 | 2048 | 1524 | 1024 | 0 | 0 | 0 | mod-patron | 4 | mod-patron:6.1.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-oai-pmh | 4 | mod-oai-pmh:3.13.0 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | edge-connexion | 4 | edge-connexion:1.2.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-kb-ebsco-java | 4 | mod-kb-ebsco-java:4.0.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | mod-notes | 4 | mod-notes:5.2.0 | 2 | 1024 | 896 | 128 | 952 | 384 | 512 | mod-data-export-spring | 4 | mod-data-export-spring:3.2.0 | 1 | 2048 | 1844 | 256 | 1536 | 384 | 512 | mod-organizations-storage | 4 | mod-organizations-storage:4.7.0 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | mod-login | 4 | mod-login:7.11.0 | 2 | 1440 | 1298 | 1024 | 768 | 384 | 512 | pub-okapi | 3 | pub-okapi:2023.06.14 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 | mod-eusage-reports | 4 | mod-eusage-reports:2.1.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
|
Expand |
---|
title | Data Export related modules |
---|
|
Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize |
---|
qcp1-pvt |
|
|
|
|
|
|
|
|
| Tue Jun 04 07:31:53 UTC 2024 |
|
|
|
|
|
|
|
|
| mod-authtoken | 5 | mod-authtoken:2.15.1 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 | mod-users-bl | 4 | mod-users-bl:7.7.0 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 | mod-inventory-storage | 4 | mod-inventory-storage:27.1.0 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | mod-source-record-storage | 4 | mod-source-record-storage:5.8.0 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | mod-inventory | 4 | mod-inventory:20.2.0 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 | mod-source-record-manager | 5 | mod-source-record-manager:3.9.0-SNAPSHOT.330 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | nginx-okapi | 3 | nginx-okapi:2023.06.14 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 | okapi-b | 4 | okapi:5.3.0 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 | mod-data-export | 11 | mod-data-export:5.0.4 | 1 | 2048 | 1524 | 1024 | 0 | 0 | 0 | pub-okapi | 3 | pub-okapi:2023.06.14 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 |
|