Users-search-filter test report [non-ECS] Trillium
Overview
The purpose of this test is to define system behavior under load with user-search-filter workflow. Compare with previous report. Find any possible issues.
Current ticket: https://folio-org.atlassian.net/browse/PERF-1352
Related resource: https://folio-org.atlassian.net/wiki/x/AYABg
Previous report: https://folio-org.atlassian.net/wiki/x/fB8V
Summary
All tests passed successfully without errors, issues or anomalies
Both tests showed significant response times improvements comparing to previous result (Honeysuckle) by more than 10 times.
Main improvement happened on BD side in query:
SELECT jsonb,id FROM cs00000int_0001_mod_users.users WHERE get_tsvector(f_unaccent(users.jsonb->>?)) @@ tsquery_phrase(f_unaccent(?)) ORDER BY left(lower(f_unaccent(users.jsonb->?->>?)),?), lower(f_unaccent(users.jsonb->?->>?)), left(lower(f_unaccent(users.jsonb->?->>?)),?), lower(f_unaccent(users.jsonb->?->>?)) LIMIT ? OFFSET ?
Query improvement reduced CPU consumption by 80% during both tests.
Test Runs/Results/Comparison
Test # | Virtual Users | Duration (sec) |
|---|---|---|
1 | 5 | 1800 |
2 | 8 | 1800 |
Requests | Trillium | Honeysuckle | Diff | Trillium | Honeysuckle | Diff | ||
|---|---|---|---|---|---|---|---|---|
avg | 50th pct | 50th pct | avg | 50th pct | 50th pct |
| ||
GET_/groups?limit=40 sortby group | 0.014 | 0.013 | 0.031 | 0.018 / 138.5% | 0.017 | 0.017 | 0.053 | 0.036 / 211.8% |
GET_/tags | 0.011 | 0.010 | 0.022 | 0.012 / 120.0% | 0.010 | 0.010 | 0.028 | 0.018 / 180.0% |
GET_/users?limit=100 offset=100 query=(active="true") | 0.037 | 0.36 | 4.751 | 4.715 / 13097.2% | 0.043 | 0.042 | 7.292 | 7.250 / 17261.9% |
GET_/users?limit=100 query=(active="true") | 0.035 | 0.035 | 4.749 | 4.714 / 13468.6% | 0.042 | 0.041 | 7.175 | 7.134 / 17400.0% |
GET_/Departments | 0.021 | 0.021 | 0.023 | 0.002 / 9.5% | 0.024 | 0.024 | 0.034 | 0.010 / 41.7% |
User transaction | 0.118 | 0.115 | 9.625 | 9.510 / 8269.6% | 0.135 | 0.134 | 15.884 | 15.750 / 11753.7% |
Resource utilization
Service CPU Utilization
most loaded module is mod-users with ±1130% CPU usage.
Instance CPU Utilization
Max CPU utilization of EC2 instance during test was 33%
Service Memory Utilization
Memory remain stable during both tests. No signs of memory leaks.
DB CPU Utilization
Significant DB optimization is visible. In previous tests DB CPU was 100% while now it’s 40% max.
Note. There was background job running from mod-search during tests - that’s explaining higher CPU usage than on ECS tests.
DB Connections
FreeableMemory
DB load
Top SQL-queries
Note: there was background job from mod-search running during tests. However this job didn’t affect overall response times.
Appendix
Infrastructure
PTF - environment Trillium(tecp1)
tecon 8 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r7g.xlarge database instance: Writer instance
MSK fse-test
4 kafka.m7g.2xlarge brokers in 2 zones (2 brokers per zone)
Apache Kafka version 3.7.x, metadata mode - KRaft
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
revision - 26
OpenSearch 2.13 ptf-test cluster
r7g.2xlarge.search 4 data nodes
r7g.large.search 3 dedicated master nodes
Kafka consolidated topics enabled
Module versions: