PTF - Export deleted MARC authority records (Ramsons) [ECS]
Test status: PASSED
Overview
Regression testing of export of deleted MARC authority records via API. Measurement the performance of an export operation of 2K (paged) records for all 100K/300K deleted records
ECS environment with PTF data set
Classic PTF configuration with no additional improvements.
The purpose of this testing is to compare test results of Ramsons release with previous, Quasnelia release, check for improvements, possible issues/degradation.
Expected export duration - under a minute.
Jiras/ links:
Quesnelia release ticket: https://folio-org.atlassian.net/browse/PERF-897.
Quesnelia export delete MARC authority records via API Report
Current ticket: https://folio-org.atlassian.net/browse/PERF-1028
Related to improvement task: https://folio-org.atlassian.net/browse/MDEXP-769
Previous report https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/273055746
Summary
All tests are passed successfully.
SLA (duration of export should be less than a minute) met.
Test duration is faster in 100K test for more than 50% in Ramsons compared to Quesnelia
for 300K tests, performance is more or less the same, however average api response time is better
No memory leaks or spikes found
CPU usage showed stable trend. Most used module is mod-entities-links.
DB CPU usage is low for 100K tests (around 6%) and 15%-17% for 300K tests. For tests with additional load (tests with 10 loops of JMeter scrip) it’s 15-17% for 100K and 25-30% for 300K.
Test Runs/Results
Each test run got additional rerun to ensure of performance consistency. Moreover tests 3,4,7,8 was performed to check system behaviour under additional load. This tests are “fast“ so additional tests was performed to get more information about system behaviour. (tests with 10 loops of JMeter script), Named in table (10 times)
Test # | Test Conditions | Duration |
|---|---|---|
1 | 100K | 8s 652 ms |
2 | 100K (rerun) | 8s 440 ms |
3 | 100K (10 times ) | 7s 860 ms (avg) |
4 | 100K (10 times ) | 7s 591 ms (avg) |
5 | 300K | 29s 567 ms |
6 | 300K | 29s 989 ms |
7 | 300K (10 times ) | 29s 579 ms (avg) |
8 | 300K (10 times ) | 28s 393 ms (avg) |
Comparisons
Tests duration was compared to same tests in Quesnelia release https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/273055746
In overall - performance is better by more than 50% for 100K tests and the same for 300K test.
Test | Ramsons | Quesnelia | ||
|---|---|---|---|---|
| Duration (s/ ms) | GET_authority-storage/authorities response time (ms) | Duration (s/ ms) | GET_authority-storage/authorities response time (ms) |
100K | 8s 652 ms | 178 ms | 13s 317 ms | 262 ms |
300K | 29s 989 ms | 216 ms | 29s 109 ms | 288 ms |
Memory Utilization
Memory showed stable trend without visible signs of memory leaks or anomalies. Most used module is mod-entities-links and it consumes 32% memory in average during whole testing set. Please look on memory trend to all related modules in chart below.
Note: Taking into account that tests durations are less then a minute - in order to show trend, all tests included into chart. Below - table with top 3 modules
Module | Ramsons Avg | Quesnelia Avg |
|---|---|---|
mod-entities-links | 32% | 37% |
okapi | 32% | - |
mod-users | 25% | - |
CPU Utilization
In 100K and 300K tests CPU usage is barely visible because of fast test duration that’s why additional tests was performed (100K x10 and 300K x 10, basically same tests but 10 times one after another without pauses) to see if there are anomalies sudden spices, etc.. CPU usage stabilise on ±3%. CPU usage spiked only on mod-entities-links (as obvious), okapi, nginx-okapi and supportive services like mod-users-bl, mod-login, mod-authtoken.
No unexpected spikes or anomalies observed.
Note: Instance-level CPU usage is not included into this report as on instance scale CPU usage during this tests is not visible.
RDS CPU Utilization
RDS CPU usage is almost invisible during 100K tests (CPU utilization was ±6% and have no difference from IDLE state of DB), however with 10X for 300K deleted Authority records, tests it’s visible that CPU reached ±30%.
On database connections chart it’s no visible signs of tests, probably because the only module created connections to DB during that time was mod-entities-links.
Note: No slow queries defined in logs and in performance insights.
Errors
No errors found during tests. All tests passed successfully all response codes and responses checked to be valid.
Appendix
Infrastructure
PTF -environment rcon
11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6.xlarge database, writer. Standard fse configuration
MSK fse-tenant
Metadata mode: KRaft
4 kafka.m7g.xlarge brokers in 2 zones
Apache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=2
Open Search ptf-test cluster
r6g.2xlarge.search 4 data nodes
r6g.large.search 3 dedicated master nodes