| Table of Contents |
|---|
...
- Load tests results comparison showed that there is no significant degradation in CICO response times with decreased number of partitions.
- Resource consumption of server, database and Kafka instances also didn't change with decreased number of partitions.
- Data Import time is also +/- 10% duration on average of baseline results.
- Bulk Edits response times are also within seconds of the baseline results.
- Reindexing is faster and consumed less CPU with 2 partitions each in all topics compared to 10 or 50 partitions in several search topics.
Test Runs
Test # | Test Conditions | Duration | Load generator size (recommended) | Load generator Memory(GiB) (recommended) | Notes |
|---|---|---|---|---|---|
1. | Checkin/Ckeckout with 8, 20, 25 users | 30 min | t3.medium | 3 |
|
| 2. | Data import with 5K, 25K, 50K, 100K Create imports | x |
...
| 10/50 Partitions | 2 Partitions | |
|---|---|---|
| Reindexing 1 | 14hr 30m (1/29 4:30 UTC - 1/29 19:00 UTC) | |
| Reindexing 2 | 11hr 20m (2/3 21:30 UTC - 2/4 8:50 UTC) | |
| Reindexing 3 | 11hr (2/4 18:30 UTC - 2/5 5:30 UTC) |
Reindexing 1
OpenSearch Graphs (
...
Reindexing 1
...
with 10/50 Partitions)
Indexing Data Rate graph 1 shows a spike of up to 126K/min for about 9 hours, then it tailed off for another 6+ hours
...
Indexing Data Rate graph 2 shows the tail end of the reindexing where the indexing rate drops to below 5K/min and drags out until 19:00.
Service CPU Utilization (Reindexing 10/50 Partitions)
The main services, mod-inventory-storage and mod-search, doesn't show much if any activity after 12:30 (and through 19:00) when the indexing rate dropped to a few operations/min from 60K+/min
Database CPU Utilization (Reindexing 10/50 Partitions)
The database CPU utilization graph also shows the same story. Note that the time here is +5 UTC
MSK Cluster CPU Utilizations (Reindexing 10/50 Partitions)
Not much happening, only a bump in the CPU for the entire duration after the initial short-lived spikes.
Reindexing 2
OpenSearch Graphs (Reindexing 2 w/2 Partitions)
Comparing to Reindexing 1 which there had been 10/50 partitions each, reindexing 2 with topics having 2 partitions had a shorter burst of a high rate of indexing, between 32K and 65K operations per minute, but even then this number is about half of Reindexing 1 (between 64K and 120K). The initial high rate burst lasted for about 4 hours compared to 9 hours and the long tail of lower indexing rate is about 8 hours long compared to 6.5 hours in Reindexing 1.
Service CPU Utilization (Reindexing 2 w/2 Partitions)
Service CPU utilization is also lowered for mod-search and mod-inventory-storage (well below 50%) compared to reindexing 1 (around 30%-50%)
Database CPU Utilization (Reindexing 2 w/2 Partitions)
Database CPU Utilization is also lowered in reindexing 2 during the initial 4 hours spikes, to about 5% on average compared to 10-13% average in reindexing 1
MSK Cluster CPU Utilizations (Reindexing 2 w/2 Partitions)
MSK cluster CPU Utilization graph also doesn't show a noticeable bump in utilization during te initial 4 hours burst.
Reindexing 3
OpenSearch Graphs (Reindexing 3 w/2 Partitions)
Service CPU Utilization (Reindexing 3 w/2 Partitions)
Reindexing 3 with all topics having 2 partitions have the same indexing rate pattern as Reindexing 2 with the initial high rate burst lasted for about 4 hours and a 7 hours tail end.
...
Reindexing 3's service CPU utilization graph also shows the same behavior as in Reindexing 2 with the main modules' CPU utilization is well less than 505.
Database CPU Utilization (Reindexing 3 w/2 Partitions)
Reindexing 3's DB CPU utilization graph also shows the same behavior as in Reindexing 2 with the DB CPU utilization around 7%..
MSK Cluster CPU Utilization (Reindexing 3 w/2 Partitions)
Reindexing 3's service MSK Cluster utilization graph also shows the same behavior as in Reindexing 2 with a little bump throughout the reindexing.
...