mod-search: Test Reindexing of Instances (Orchid)
- Roman_Fedynyshyn
- Martin Tran
Owned by Roman_Fedynyshyn
Overview
- Reindexing with recreating the instance index on Nolana is faster than on Orchid env. 8hr 25 min against 11 hr 10 minutes
- Reindexing with recreating the instance index latency is higher on Orchid env than on Nolana 18ms against 7 ms
Recommendations & Jiras
- Original ticket to test: PERF-430
- Additional info ElasticSearch Reindex Performance Recommendations
Test Runs /Results
Test # | Test Conditions | Duration | Notes |
1. | Reindexing on Nolana release | 8 hr 25 min |
|
2. | Reindexing on Orchid release | 11 hr 10 min |
|
Memory Utilization
Memory utilisation Orchid
Memory utilisation Nolana
Initial spike | avg. usage | |
|---|---|---|
| mod-search Orchid | 47% | 36% |
| mod-inventory-storage Orchid | 50% | 50% |
| mod-search Nolana | 25.8% | 24% |
| mod-inventory-storage Nolana | 40% | 40% |
CPU Utilization
CPU usage Orchid
CPU usage Nolana
RDS CPU Utilization
DB cpu usage Orchid
DB CPU usage Nolana
Open Search KPIs
Indexing rate Orchid
Indexing rate Nolana
Indexing latency Orchid
Indexing latency Nalana
Appendix
Infrastructure
PTF -environment ncp2/ocp2
- 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Number of instances
ncp2 -10M
ocp2 -10M
Kafka partitioning
| Nolana | Orchid | |
|---|---|---|
inventory.item | 0 | 50 |
inventory.instance | 50 | 50 |
inventory.holdings-record | 0 | 50 |
inventory.bound-with | 0 | 50 |
inventory.authorit | 1 | 50 |
search.instance-contributor | 1 | 50 |
Modules memory and CPU parameters:
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-search Orchid | 2.1.0-SNAPSHOT.108 | 3 | 8 | 400 | 2592 | 2480 | 1024 | 1440 |
| mod-inventory-storage Orchid | 26.1.0-SNAPSHOT.644 | 3 | 2 | 1024 | 1952 | 2208 | 512 | 1440 |
| mod-search Nolana | 1.8.0 | 1 | 8 | 400 | 2592 | 2480 | 1024 | 1440 |
| mod-inventory-storage Nolana | 25.0.1 | 1 | 2 | 1024 | 1952 | 2208 | 512 | 1440 |
Methodology/Approach
- Use PTF's "Bugfest" Orchid cluster, which has 8M records, to test with (2 times)
- Reindex on ncp2 (Nolana version) first, get the results for indexing time, index size
- Reindex on the new Orchid environment, get the results for indexing time, index size