Data Import MARC Authorities (Morning Glory)
It's been found after testing that the actual durations of the imports performed were about 2 (two) times longer than what was reported. This is due to the PTF environment missing a DB trigger that, when restored, doubled the imports' durations.
Overview
This is a test report of MARC Authorities Data Import running on the Morning Glory software.
Infrastructure
10 m6i.2xlarge EC2 instances (changed. In Lotus it was m5.xlarge)
2 instances of db.r6.xlarge database instances, one reader and one writer
MSK
4 m5.2xlarge brokers in 2 zones
auto.create-topics.enable = true
log.retention.minutes=120
2 partitions per DI topics
mod-inventory memory
1024 CPU units, 2592MB mem
inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
kafka.consumer.max.poll.records=10
mod-inventory-storage
1024 CPU units, 1684MB mem
mod-source-record-storage
1024 CPU units, 1296MB mem
mod-source-record-manager
1024 CPU units, 1844MB mem
mod-data-import
256 CPU units, 1844MB mem
mod-data-import-cs
128 CPU units, 896MB mem
Infrastructure comparison with MG vs Lotus
Please note that infrastructure has being migrated from 6 m5.xlarge EC2 instances to 10 m6i.2xlarge. Key differences below:
Instance | CPU | RAM |
|---|---|---|
m5.xlarge | 4 | 16 |
m6i.2xlarge | 8 | 32 |
Differences in modules memory and CPU parameters
Module | Lotus | Morning Glory | ||
CPU | RAM | CPU | RAM | |
|---|---|---|---|---|
mod-inventory | 256 | 1814MB | 1024 | 2592MB |
mod-inventory-storage | 128 | 544MB | 1024 | 1684MB |
mod-SRS | 128 | 908MB | 1024 | 1296MB |
mod-SRM | 128 | 1292MB | 1024 | 1844MB |
mod-data-import | 128 | 1024MB | 256 | 1844MB |
mod-data-import-cs | - | - | 128 | 896MB |
Software versions
mod-data-import v2.5.0
mod-data-import-converter-storage v1.14.0
mod-source-record-manager v3.4.0
mod-source-record-storage v5.4.0
mod-inventory v18.2.0
mod-inventory-storage v24.0.0
Results
Summary
Profile | Duration | |
|---|---|---|
1K | Default - Create SRS MARC Authority | 24 sec |
5K | Default - Create SRS MARC Authority | 1 min 21 sec |
10K | Default - Create SRS MARC Authority | 2 min 32 sec |
25K | Default - Create SRS MARC Authority | 11 min 14 s |
50K | Default - Create SRS MARC Authority | 22 min |
Comparison to Lotus
MG Duration | Lotus Duration | Delta | |
|---|---|---|---|
1K | 24 sec | ||
5K | 1 min 21 sec | ||
10K | 2 min 32 sec | ||
25K | 11 min 14 s | ||
50K | 22 min |
Resource Usages
CPU usage on mod-data-import-converter-storage increases over time
PERF-282: Investigate High CPU usage on mod-data-import-converter-storageClosed
during a 1 week of testing memory usage increased for mod-srm, mod-srs, mod-data-import-cs, and mod-inventory-storage - JIRA ticket was created
Tasks for mod-srm, and mod-srs were unexpectedly restarted.