mod-search: Test Reindexing of Instances (Orchid)
- 1 Overview
- 2 Test Runs /Results
- 2.1 Memory Utilization
- 2.2 CPU Utilization
- 2.2.1 CPU usage Orchid
- 2.2.2 CPU usage Nolana
- 2.3 RDS CPU Utilization
- 2.3.1 DB cpu usage Orchid
- 2.3.2 DB CPU usage Nolana
- 2.4 Open Search KPIs
- 3 Appendix
Overview
Reindexing with recreating the instance index on Nolana is faster than on Orchid env. 8hr 25 min against 11 hr 10 minutes
Reindexing with recreating the instance index latency is higher on Orchid env than on Nolana 18ms against 7 ms
Recommendations & Jiras
Original ticket to test: PERF-430
Additional info ElasticSearch Reindex Performance Recommendations
Test Runs /Results
Test # | Test Conditions | Duration | Notes
|
1. | Reindexing on Nolana release | 8 hr 25 min |
|
2. | Reindexing on Orchid release | 11 hr 10 min |
|
Memory Utilization
Memory utilisation Orchid
Memory utilisation Nolana
| Initial spike | avg. usage |
|---|---|---|
mod-search Orchid | 47% | 36% |
mod-inventory-storage Orchid | 50% | 50% |
mod-search Nolana | 25.8% | 24% |
mod-inventory-storage Nolana | 40% | 40% |
CPU Utilization
CPU usage Orchid
CPU usage Nolana
RDS CPU Utilization
DB cpu usage Orchid
DB CPU usage Nolana
Open Search KPIs
Indexing rate Orchid
Indexing rate Nolana
Indexing latency Orchid
Indexing latency Nalana
Appendix
Infrastructure
PTF -environment ncp2/ocp2
9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK ptf-kakfa-3
4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
Number of instances
ncp2 -10M
ocp2 -10M
Kafka partitioning
| Nolana | Orchid |
|---|---|---|
inventory.item | 0 | 50 |
inventory.instance | 50 | 50 |
inventory.holdings-record | 0 | 50 |
inventory.bound-with | 0 | 50 |
inventory.authorit | 1 | 50 |
search.instance-contributor | 1 | 50 |
Modules memory and CPU parameters:
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
mod-search Orchid |
| 3 | 8 | 400 | 2592 | 2480 | 1024 | 1440 |
mod-inventory-storage Orchid | 26.1.0-SNAPSHOT.644 | 3 | 2 | 1024 | 1952 | 2208 | 512 | 1440 |
mod-search Nolana |
| 1 | 8 | 400 | 2592 | 2480 | 1024 | 1440 |
mod-inventory-storage Nolana | 25.0.1 | 1 | 2 | 1024 | 1952 | 2208 | 512 | 1440 |
Methodology/Approach
Use PTF's "Bugfest" Orchid cluster, which has 8M records, to test with (2 times)
Reindex on ncp2 (Nolana version) first, get the results for indexing time, index size
Reindex on the new Orchid environment, get the results for indexing time, index size