...
...
| Info |
|---|
It's been found after testing that the actual durations of the imports performed were about 2 (two) times longer than what was reported. This is due to the PTF environment missing a DB trigger that, when restored, doubled the imports' durations. |
| Table of Contents |
|---|
Overview
...
- mod-data-import:2.5.0
- mod-data-import-converter-storage:1.14.1
- mod-source-record-storage:5.4.0
- mod-source-record-manager:3.4.1
- mod-inventory:18.2.2
- mod-inventory-storage:24.1.0
- mod-search:1.7.4
- mod-quick-marc:2.4.1
Infrastructure
- 10 m6i.2xlarge EC2 instances (changed. In Lotus it was m5.xlarge)
- 2 instances of db.r6.xlarge database instances, one reader and one writer
- MSK
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=120
- 2 partitions per DI topics
- mod-inventory memory
- 1024 CPU units, 2592MB mem
- inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
- inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
- kafka.consumer.max.poll.records=10
- mod-inventory-storage
- 1024 CPU units, 1684MB mem
- mod-source-record-storage
- 1024 CPU units, 1296MB mem
- mod-source-record-manager
- 1024 CPU units, 1844MB mem
- mod-data-import
- 256 CPU units, 1844MB mem
- mod-data-import-cs
- 128 CPU units, 896MB mem
Results
test | file | duration |
|---|---|---|
1 | 1k | 28s |
| 2 | 5k | 1 m 48s |
| 3 | 10k | 4 m 4s |
| 4 | 80k | 29 m 6 s |
Resources usage
Here CPU usage is not higher than 60% for all related modules.
...