IN PROGRESS
Overview
Infrastructure
- 10 m6i.2xlarge EC2 instances (changed. In Lotus it was m5.xlarge)
- 2 instances of db.r6.xlarge database instances, one reader and one writer
- MSK
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=120
- 2 partitions per DI topics
- mod-inventory memory
- 1024 CPU units, 2592MB mem
- inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
- inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
- kafka.consumer.max.poll.records=10
- mod-inventory-storage
- 1024 CPU units, 1684MB mem
- mod-source-record-storage
- 1024 CPU units, 1296MB mem
- mod-source-record-manager
- 1024 CPU units, 1844MB mem
- mod-data-import
- 256 CPU units, 1844MB mem
- mod-data-import-cs
- 128 CPU units, 896MB mem
Infrastructure comparison with MG vs Lotus
Please note that infrastructure has being migrated from 6 m5.xlarge EC2 instances to 10 m6i.2xlarge. Key differences below:
Instance | CPU | RAM |
|---|---|---|
| m5.xlarge | 4 | 16 |
| m6i.2xlarge | 8 | 32 |
Differences in modules memory and CPU parameters
| Module | Lotus | Morning Glory | ||
| CPU | RAM | CPU | RAM | |
|---|---|---|---|---|
| mod-inventory | 256 | 1814MB | 1024 | 2592MB |
| mod-inventory-storage | 128 | 544MB | 1024 | 1684MB |
| mod-SRS | 128 | 908MB | 1024 | 1296MB |
| mod-SRM | 128 | 1292MB | 1024 | 1844MB |
| mod-data-import | 128 | 1024MB | 256 | 1844MB |
| mod-data-import-cs | - | - | 128 | 896MB |
Software versions
- mod-data-import v2.5.0
- mod-data-import-converter-storage v1.14.0
- mod-source-record-manager v3.4.0
- mod-source-record-storage v5.4.0
- mod-inventory v18.2.0
- mod-inventory-storage v24.0.0
Results
Summary
Profile | Duration | |
|---|---|---|
| 1K | Default - Create SRS MARC Authority | 24 sec |
5K | Default - Create SRS MARC Authority | 1 min 21 sec |
| 10K | Default - Create SRS MARC Authority | 2 min 32 sec |
| 25K | Default - Create SRS MARC Authority | 11 min 14 s |
| 50K | Default - Create SRS MARC Authority | 22 min |
Comparison to Lotus
MG Duration | Lotus Duration | Delta | |
|---|---|---|---|
| 1K | 24 sec | ||
5K | 1 min 21 sec | ||
| 10K | 2 min 32 sec | ||
| 25K | 11 min 14 s | ||
| 50K | 22 min |
Resource Usages
- CPU usage on mod-data-import-converter-storage increases over time PERF-282 - Getting issue details... STATUS
- during a 1 week of testing memory usage increased for mod-srm, mod-srs, mod-data-import-cs, and mod-inventory-storage - JIRA ticket was created PERF-281 - Getting issue details... STATUS
- Tasks for mod-srm, and mod-srs were unexpectedly restarted.
