PTF - Data Export Test Report (Sunflower) [NON-ECS]
- Azimjon Alijonov
1 Overview
2 Summary
6 Appendix
6.1 Infrastructure
Overview
This document contains the results of three times testing Data Export (MARC BIB) on the Sunflower[NON-ECS] release. After one month of the first test, two additional tests were conducted. In the second test, mod circulation storage was enabled. In the third test, mod circulation storage on module was disabled. All test results have been summarized correctly.
PERF-1115: [Sunflower] [non-ECS] [Data Export] MARC BIBIn QA
Summary
Data Export tests finished successfully(except some duplications) on using the profiles Default instances export job profile and SRS - holdings and items job profile.
Comparing with previous results of Ramsons and Sunflower releases
Data Export processed all files including file with 500k records with negligible duplication errors in Sunflower env.
Data export durations are nearly twice as slow compared to Ramsons releases when mod circulation storage is enabled.
When mod circulation storage is disabled, it faster than run #1 and #2 but slower compared to Ramsons/
Test Results
This table contains durations for Data Export with two job profiles.
Profile | CSV File | Tenant (fs09000000) | |
|---|---|---|---|
Result | Status | ||
DE MARC Bib (Default instances export job profile) | 1k.csv | 0:00:09 | COMPLETED |
100k.csv | 0:06:03 | COMPLETED | |
500k.csv | 0:06:46 | COMPLETED | |
DE MARC Bib (SRS - holdings and items job profile) | 1k.csv | 0:00:17 | COMPLETED |
100k.csv | 0:12:20 | COMPLETED | |
500k.csv | 0:15:13 | COMPLETED | |
Comparison
This table contains durations comparison between Sunflower and Ramsons releases.
Profile | CSV File | DE Duration | DE Duration | DE Duration, DELTA Sunflower/Ramsons |
Result | Result | hh:mm:ss / percent | ||
DE MARC Bib (Default instances export job profile) | 1k.csv | 0:00:09 | 0:00:03 | +00:00:06 |
100k.csv | 0:06:03 | 0:02:19 | +00:03:44 | |
500k.csv | 0:07:49 | 0:04:33 | +00:03:16 | |
DE MARC Bib (SRS - holdings and items job profile) | 1k.csv | 0:00:17 | 0:00:06 | +00:00:11 |
100k.csv | 0:12:20 | 0:07:02 | +00:05:18 | |
500k.csv | 0:17:45 | 0:08:57 | +00:08:48 |
Resource utilization for Tests
Resource utilization table
CPU | RAM | ||
|---|---|---|---|
mod-data-export-b | 88.2% | mod-inventory-b | 78.3% |
mod-remote-storage-b | 13.7% | mod-data-export-b | 63% |
mod-inn-reach-b | 13.7% | mod-users-keycloak-b | 56.4% |
mod-audit-b | 11.3% | mod-audit-b | 55.1% |
mod-pubsub-b | 8.2% | mod-pubsub-b | 45.1% |
mod-users-keycloak-b | 7.8% | mod-quick-marc-b | 37.6% |
mod-inventory-b | 7.7% | mod-inn-reach-b | 31.8% |
mod-quick-marc-b | 7.4% | mod-invoice-storage-b | 31.8% |
mod-invoice-storage-b | 7.4% | mod-remote-storage-b
| 22.2% |
Instance CPU Utilization
Service CPU Utilization
Here we can see that mod-data-export used 88% CPU in spike.
Service Memory Utilization
Here we can see all services have stable trends but mod-data-export which used more memory when data volume increased.
DB CPU Utilization
DB CPU spike was 77% when 1k data was exported and first 100k data started its job. However, for data 500k, DB cpu utilization fluctuated between 50% and 60%.
DB Connections
DB connections was 1201.
Kafka metrics
OpenSearch Data Nodes metrics
DB load
Top SQL-queries
Top applications
Appendix
Infrastructure
PTF - environment Sunflower NON-ECS (secp1)
secp1 12 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r7.xlarge database instance: Writer instance
MSK fse-test
4 kafka.m7g.xlarge brokers in 2 zones (2 brokers per zone)
Apache Kafka version 3.7.x, metadata mode - KRaft
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
revision - 26
OpenSearch 2.13 ptf-test cluster
r7g.2xlarge.search 4 data nodes
r6g.large.search 3 dedicated master nodes