...
...
| Info |
|---|
It's been found after testing that the actual durations of the imports performed were about 2 (two) times longer than what was reported. This is due to the PTF environment missing a DB trigger that, when restored, doubled the imports' durations. |
| Table of Contents |
|---|
Overview
...
- mod-data-import:2.5.0
- mod-data-import-converter-storage:1.14.1
- mod-source-record-storage:5.4.0
- mod-source-record-manager:3.4.1
- mod-inventory:18.2.2
- mod-inventory-storage:24.1.0
- mod-search:1.7.4
- mod-quick-marc:2.4.1
Infrastructure
- 10 m6i.2xlarge EC2 instances (changed. In Lotus it was m5.xlarge)
- 2 instances of db.r6.xlarge database instances, one reader and one writer
- MSK
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=120
- 2 partitions per DI topics
- mod-inventory memory
- 1024 CPU units, 2592MB mem
- inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
- inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
- kafka.consumer.max.poll.records=10
- mod-inventory-storage
- 1024 CPU units, 1684MB mem
- mod-source-record-storage
- 1024 CPU units, 1296MB mem
- mod-source-record-manager
- 1024 CPU units, 1844MB mem
- mod-data-import
- 256 CPU units, 1844MB mem
- mod-data-import-cs
- 128 CPU units, 896MB mem
Results
test | file | duration |
|---|---|---|
1 | 1k | 28s |
| 2 | 5k | 1 m 48s |
| 3 | 10k | 4 m 4s |
| 4 | 80k | 29 m 6 s |
Resources usage
Here CPU usage is not higher than 60% for all related modules.
Concerning behavior on :
- mod-source-record-manager growing memory from 55% up to 93%
- mod-source-record-storage growing memory from 41% up to 64%
However last test for 80K didn't showed any memory growing for any of module, so maybe growing of mem usage can be explained as working condition of this modules.
RDS CPU usage reached 80% maximum during test.
