OverviewThe purpose of the document is getting results of testing Data Import Create MARC holdings records and to detect performance trends in Poppy in scope of ticket
PERF-700
-
Getting issue details...
STATUS
Compared with results in previous test report: Data Import Create MARC holdings records [Orchid]
Summary
- Data import jobs duration decreased significantly compared to the Orchid release results for all tests
- Top CPU utilization: nginx-okapi - 52%, mod-quick-marc-b - 45%, mod-inventory-b - 42%, mod-source-record-storage-b - 25%
- Top Memory consumption: mod-users-b - 60%, mod-data-import-b - 60%, okapi-b - 48%. There're no issues with memory.
- DI jobs duration is directly proportional to the size of the files being processed.
Recommendations & Jiras
- Autovacuum as background process increase DI job duration for 10%-25% so it's necessary to check DB activity before DI jobs start
- SQL queries are the possible place where performance may be improved:
- SQL DB Load with WAITS - Lock:tuple, Lock:transactionid take a major time in queries:
INSERT INTO [tenant]_mod_inventory_storage.holdings_record (id, jsonb) VALUES ($1, $2) RETURNING jsonb
UPDATE [tenant]
_mod_inventory_storage.holdings_record SET jsonb = $1::jsonb WHERE id = 'UUID'
Test Results
Profile used for testing - Default - Create Holdings and SRS MARC Holdings
Test | File | Duration: Morning_Glory (previous results) | Duration: Nolana (previous results) | Duration: Orchid (previous results) | Duration: Poppy |
---|
1 | 1k | 28s | 32s | 45s | 32s |
2 | 5k | 1m 48s | 4m 20s | 7m 47s | 2m 14s |
3 | 10k | 4m 4s | 3m 24s | 19m 46s | 4m 35s |
4 | 80k | 29m 6s | 21m 22s | 20m (error*) | 36m 25s |
Comparison
Table contains comparison between Poppy and Orchid
Test | File | Duration: Orchid | Duration: Poppy | Diff_absolute | Diff_percentage |
---|
1 | 1k | 45s | 32s | 13s | - 40% |
2 | 5k | 7m 47s | 2m 14s | 5m 33s | - 248% |
3 | 10k | 19m 46s | 4m 35s | 15m 11s | - 331% |
4 | 80k | 20m (error*) | 36m 25s | No data to compare in Orchid |
|
Service CPU Utilization
CPU utilization, 1k, 5k, 10k, 80k
| 1k |
|
| 5k |
|
| 10k |
|
| 80k |
---|
mod-inventory-b | 26.42 |
| nginx-okapi | 51.81 |
| nginx-okapi | 51.66 |
| nginx-okapi | 51.8 |
mod-quick-marc-b | 23.78 |
| mod-quick-marc-b | 44.77 |
| mod-quick-marc-b | 44.05 |
| mod-quick-marc-b | 44.97 |
nginx-okapi | 21.44 |
| mod-inventory-b | 44.21 |
| mod-inventory-b | 43.83 |
| mod-inventory-b | 41.39 |
mod-di-converter-storage-b | 11.78 |
| okapi-b | 25.48 |
| okapi-b | 25.16 |
| okapi-b | 25.29 |
okapi-b | 11.54 |
| mod-source-record-storage-b | 24.27 |
| mod-source-record-storage-b | 24.19 |
| mod-source-record-storage-b | 25.15 |
mod-source-record-storage-b | 10.67 |
| mod-di-converter-storage-b | 20.29 |
| mod-di-converter-storage-b | 20.36 |
| mod-di-converter-storage-b | 19.79 |
mod-source-record-manager-b | 8.93 |
| mod-inventory-storage-b | 17.44 |
| mod-inventory-storage-b | 17.11 |
| mod-inventory-storage-b | 17.79 |
mod-inventory-storage-b | 7.53 |
| mod-source-record-manager-b | 16.93 |
| mod-source-record-manager-b | 16.8 |
| mod-source-record-manager-b | 16.92 |
mod-pubsub-b | 6.73 |
| mod-pubsub-b | 6.67 |
| mod-pubsub-b | 6.88 |
| mod-pubsub-b | 6.79 |
mod-users-b | 6.32 |
| mod-users-b | 6.5 |
| mod-users-b | 6.39 |
| mod-users-b | 6.37 |
mod-data-import-b | 4.04 |
| mod-permissions-b | 2.85 |
| mod-organizations-storage-b | 2.61 |
| mod-permissions-b | 2.87 |
mod-organizations-storage-b | 2.29 |
| mod-organizations-storage-b | 2.42 |
| mod-notes-b | 2.27 |
| mod-organizations-storage-b | 2.43 |
mod-notes-b | 2.05 |
| mod-notes-b | 2.1 |
| mod-gobi-b | 2.01 |
| mod-notes-b | 2.14 |
mod-gobi-b | 2.01 |
| mod-gobi-b | 2.01 |
| mod-permissions-b | 1.43 |
| mod-gobi-b | 2 |
mod-permissions-b | 1.46 |
| mod-data-import-b | 1.41 |
| mod-search-b | 1.31 |
| mod-search-b | 1.48 |
mod-search-b | 1.25 |
| mod-search-b | 1.26 |
| mod-data-import-b | 1.29 |
| mod-data-import-b | 1.32 |
mod-circulation-storage-b | 0.57 |
| mod-circulation-storage-b | 0.59 |
| mod-circulation-storage-b | 0.6 |
| mod-circulation-storage-b | 0.58 |
mod-circulation-b | 0.32 |
| mod-circulation-b | 0.47 |
| mod-circulation-b | 0.32 |
| mod-circulation-b | 0.32 |
pub-okapi | 0.27 |
| pub-okapi | 0.19 |
| pub-okapi | 0.18 |
| pub-okapi | 0.18 |
nginx-okapi - 52%, mod-quick-marc-b - 45%, mod-inventory-b - 42%, mod-source-record-storage-b - 25%
![](https://folio-org.atlassian.net/wiki/download/thumbnails/91848706/image2024-3-15_13-41-16.png?version=1&modificationDate=1710502880494&cacheVersion=1&api=v2&width=1200&height=456)
Memory Utilization
Memory consumption
mod-inventory-b | 61.01 |
---|
mod-users-b | 59.97 |
mod-data-import-b | 59.38 |
okapi-b | 48.24 |
mod-source-record-manager-b | 37.31 |
mod-search-b | 31.38 |
mod-di-converter-storage-b | 31.19 |
mod-quick-marc-b | 27.52 |
mod-pubsub-b | 24.53 |
mod-remote-storage-b | 21.59 |
mod-inventory-storage-b | 16.71 |
nginx-okapi | 4.68 |
pub-okapi | 4.58 |
![](https://folio-org.atlassian.net/wiki/download/thumbnails/91848706/image2024-3-15_13-46-52.png?version=1&modificationDate=1710503216969&cacheVersion=1&api=v2&width=1200&height=450)
DB CPU Utilization
DB CPU was about 80% for all tests except of 1k - 23%.
![](https://folio-org.atlassian.net/wiki/download/thumbnails/91848706/image2024-3-15_14-52-23.png?version=1&modificationDate=1710507146737&cacheVersion=1&api=v2&width=1200&height=465)
DB Connections
DB connections number- 505
![](https://folio-org.atlassian.net/wiki/download/thumbnails/91848706/image2024-3-15_14-53-32.png?version=1&modificationDate=1710507215506&cacheVersion=1&api=v2&width=1200&height=449)
DB Load
![](https://folio-org.atlassian.net/wiki/download/thumbnails/91848706/image2024-3-15_14-54-33.png?version=1&modificationDate=1710507276919&cacheVersion=1&api=v2&width=1200&height=449)
![](https://folio-org.atlassian.net/wiki/download/thumbnails/91848706/image2024-3-15_14-54-54.png?version=1&modificationDate=1710507296983&cacheVersion=1&api=v2&width=1200&height=457)
SQL queries
![](https://folio-org.atlassian.net/wiki/download/thumbnails/91848706/image2024-3-15_14-55-22.png?version=1&modificationDate=1710507325078&cacheVersion=1&api=v2&width=1200&height=438)
![](https://folio-org.atlassian.net/wiki/download/thumbnails/91848706/image2024-3-15_14-55-44.png?version=1&modificationDate=1710507346943&cacheVersion=1&api=v2&width=1200&height=484)
![](https://folio-org.atlassian.net/wiki/download/thumbnails/91848706/image2024-3-15_14-56-36.png?version=1&modificationDate=1710507399861&cacheVersion=1&api=v2&width=1200&height=436)
Infrastructure
PTF - environment pcp1
Name | Memory GIB | vCPUs |
---|
db.r6g.4xlarge | 128 GiB | 16 vCPUs |
- MSK ptf-mobius-testing2
- 2 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
Modules
Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize | R/W split enabled |
---|
pcp1-pvt |
|
|
|
|
|
|
|
|
|
|
Thu Mar 14 14:24:59 UTC 2024 |
|
|
|
|
|
|
|
|
|
|
mod-remote-storage | 11 | mod-remote-storage:3.0.0 | 2 | 4920 | 4472 | 1024 | 3960 | 512 | 512 | FALSE |
mod-ncip | 10 | mod-ncip:1.14.4 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-finance-storage | 10 | mod-finance-storage:8.5.0 | 2 | 1024 | 896 | 1024 | 700 | 88 | 128 | FALSE |
mod-agreements | 10 | mod-agreements:6.0.2 | 2 | 1592 | 1488 | 128 | 0 | 0 | 0 | FALSE |
mod-ebsconet | 10 | mod-ebsconet:2.1.1 | 2 | 1248 | 1024 | 128 | 700 | 128 | 256 | FALSE |
edge-sip2 | 8 | edge-sip2:3.1.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-organizations | 10 | mod-organizations:1.8.0 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | FALSE |
mod-settings | 11 | mod-settings:1.0.2 | 2 | 1024 | 896 | 200 | 768 | 88 | 128 | FALSE |
edge-dematic | 10 | edge-dematic:2.1.0 | 1 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-data-import | 18 | mod-data-import:3.0.7 | 1 | 2048 | 1844 | 256 | 1292 | 384 | 512 | FALSE |
mod-search | 20 | mod-search:3.0.5 | 2 | 2592 | 2480 | 2048 | 1440 | 512 | 1024 | FALSE |
mod-tags | 10 | mod-tags:2.1.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-authtoken | 16 | mod-authtoken:2.14.1 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 | FALSE |
edge-courses | 2 | edge-courses:1.3.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-notify | 10 | mod-notify:3.1.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-inventory-update | 10 | mod-inventory-update:3.2.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-configuration | 10 | mod-configuration:5.9.2 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-orders-storage | 10 | mod-orders-storage:13.6.0 | 2 | 1024 | 896 | 512 | 700 | 88 | 128 | FALSE |
edge-caiasoft | 10 | edge-caiasoft:2.1.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-login-saml | 18 | mod-login-saml:2.7.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-erm-usage-harvester | 11 | mod-erm-usage-harvester:4.4.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-password-validator | 10 | mod-password-validator:3.1.0 | 2 | 1440 | 1298 | 128 | 768 | 384 | 512 | FALSE |
mod-licenses | 10 | mod-licenses:5.0.2 | 2 | 2480 | 2312 | 128 | 1792 | 384 | 512 | FALSE |
mod-gobi | 10 | mod-gobi:2.7.1 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | FALSE |
mod-fqm-manager | 9 | mod-fqm-manager:1.1.0-SNAPSHOT.1078 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-bulk-operations | 9 | mod-bulk-operations:1.1.7 | 2 | 3072 | 2600 | 1024 | 1536 | 384 | 512 | FALSE |
mod-graphql | 16 | mod-graphql:1.12.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-finance | 10 | mod-finance:4.8.0 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | FALSE |
mod-erm-usage | 13 | mod-erm-usage:4.6.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-copycat | 10 | mod-copycat:1.5.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-lists | 8 | mod-lists:1.1.0-SNAPSHOT.1261 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-entities-links | 15 | mod-entities-links:2.0.4 | 2 | 2592 | 2480 | 400 | 1440 | 0 | 1024 | FALSE |
mod-permissions | 47 | mod-permissions:6.5.0-SNAPSHOT.369 | 2 | 1684 | 1544 | 512 | 1024 | 384 | 512 | FALSE |
pub-edge | 9 | pub-edge:2023.06.14 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 | FALSE |
mod-orders | 10 | mod-orders:12.7.1 | 2 | 2048 | 1440 | 1024 | 1024 | 384 | 512 | FALSE |
edge-patron | 10 | edge-patron:5.0.0 | 2 | 1024 | 896 | 256 | 768 | 88 | 128 | FALSE |
edge-ncip | 11 | edge-ncip:1.9.2 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-users-bl | 10 | mod-users-bl:7.6.0 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 | FALSE |
mod-inventory-storage | 15 | mod-inventory-storage:27.0.4 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | FALSE |
mod-invoice | 10 | mod-invoice:5.7.2 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 | FALSE |
mod-user-import | 10 | mod-user-import:3.8.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-sender | 10 | mod-sender:1.11.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
edge-oai-pmh | 8 | edge-oai-pmh:2.7.1 | 2 | 1512 | 1360 | 1024 | 1440 | 384 | 512 | FALSE |
mod-data-export-worker | 10 | mod-data-export-worker:3.1.2 | 2 | 3072 | 2800 | 1024 | 2048 | 384 | 512 | FALSE |
mod-rtac | 10 | mod-rtac:3.5.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-circulation-storage | 14 | mod-circulation-storage:17.1.7 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 | FALSE |
mod-calendar | 10 | mod-calendar:2.5.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-source-record-storage | 18 | mod-source-record-storage:5.7.5 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | FALSE |
mod-event-config | 10 | mod-event-config:2.6.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-courses | 10 | mod-courses:1.4.8 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-inventory | 15 | mod-inventory:20.1.8 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 | FALSE |
mod-email | 10 | mod-email:1.16.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-di-converter-storage | 18 | mod-di-converter-storage:2.1.5 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-circulation | 14 | mod-circulation:24.0.11 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 | FALSE |
mod-pubsub | 13 | mod-pubsub:2.11.3 | 2 | 1536 | 1440 | 1024 | 922 | 384 | 512 | FALSE |
edge-orders | 10 | edge-orders:2.9.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
edge-rtac | 7 | edge-rtac:2.6.2 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-template-engine | 10 | mod-template-engine:1.19.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-users | 34 | mod-users:19.3.0-SNAPSHOT.677 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-patron-blocks | 10 | mod-patron-blocks:1.9.0 | 2 | 1024 | 896 | 1024 | 768 | 88 | 128 | FALSE |
edge-fqm | 21 | edge-fqm:1.0.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-audit | 10 | mod-audit:2.8.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-source-record-manager | 17 | mod-source-record-manager:3.7.8 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | FALSE |
nginx-edge | 9 | nginx-edge:2023.06.14 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 | FALSE |
mod-quick-marc | 11 | mod-quick-marc:5.0.1 | 1 | 2288 | 2176 | 128 | 1664 | 384 | 512 | FALSE |
nginx-okapi | 9 | nginx-okapi:2023.06.14 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 | FALSE |
okapi-b | 11 | okapi:5.1.2 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 | FALSE |
mod-feesfines | 11 | mod-feesfines:19.0.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-invoice-storage | 10 | mod-invoice-storage:5.7.0 | 2 | 1872 | 1536 | 1024 | 1024 | 384 | 512 | FALSE |
mod-service-interaction | 10 | mod-service-interaction:3.0.2 | 2 | 2048 | 1844 | 256 | 1290 | 384 | 512 | FALSE |
mod-data-export | 12 | mod-data-export:4.8.7 | 1 | 1024 | 896 | 1024 | 768 | 88 | 128 | FALSE |
mod-patron | 10 | mod-patron:6.0.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-oai-pmh | 5 | mod-oai-pmh:3.12.8 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | FALSE |
edge-connexion | 10 | edge-connexion:1.1.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-kb-ebsco-java | 10 | mod-kb-ebsco-java:4.0.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
mod-notes | 10 | mod-notes:5.1.0 | 2 | 1024 | 896 | 128 | 952 | 384 | 512 | FALSE |
mod-organizations-storage | 10 | mod-organizations-storage:4.6.0 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | FALSE |
mod-data-export-spring | 12 | mod-data-export-spring:3.0.2 | 1 | 2048 | 1844 | 256 | 1536 | 384 | 512 | FALSE |
mod-login | 10 | mod-login:7.10.1 | 2 | 1440 | 1298 | 1024 | 768 | 384 | 512 | FALSE |
pub-okapi | 9 | pub-okapi:2023.06.14 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 | FALSE |
mod-eusage-reports | 13 | mod-eusage-reports:2.0.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |