Data Import MARC Authorities (Morning Glory)
It's been found after testing that the actual durations of the imports performed were about 2 (two) times longer than what was reported. This is due to the PTF environment missing a DB trigger that, when restored, doubled the imports' durations.
Overview
This is a test report of MARC Authorities Data Import running on the Morning Glory software.
Infrastructure
- 10 m6i.2xlarge EC2 instances (changed. In Lotus it was m5.xlarge)
- 2 instances of db.r6.xlarge database instances, one reader and one writer
- MSK
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=120
- 2 partitions per DI topics
- mod-inventory memory
- 1024 CPU units, 2592MB mem
- inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
- inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
- kafka.consumer.max.poll.records=10
- mod-inventory-storage
- 1024 CPU units, 1684MB mem
- mod-source-record-storage
- 1024 CPU units, 1296MB mem
- mod-source-record-manager
- 1024 CPU units, 1844MB mem
- mod-data-import
- 256 CPU units, 1844MB mem
- mod-data-import-cs
- 128 CPU units, 896MB mem
Infrastructure comparison with MG vs Lotus
Please note that infrastructure has being migrated from 6 m5.xlarge EC2 instances to 10 m6i.2xlarge. Key differences below:
Instance | CPU | RAM |
---|---|---|
m5.xlarge | 4 | 16 |
m6i.2xlarge | 8 | 32 |
Differences in modules memory and CPU parameters
Module | Lotus | Morning Glory | ||
CPU | RAM | CPU | RAM | |
---|---|---|---|---|
mod-inventory | 256 | 1814MB | 1024 | 2592MB |
mod-inventory-storage | 128 | 544MB | 1024 | 1684MB |
mod-SRS | 128 | 908MB | 1024 | 1296MB |
mod-SRM | 128 | 1292MB | 1024 | 1844MB |
mod-data-import | 128 | 1024MB | 256 | 1844MB |
mod-data-import-cs | - | - | 128 | 896MB |
Software versions
- mod-data-import v2.5.0
- mod-data-import-converter-storage v1.14.0
- mod-source-record-manager v3.4.0
- mod-source-record-storage v5.4.0
- mod-inventory v18.2.0
- mod-inventory-storage v24.0.0
Results
Summary
Profile | Duration | |
---|---|---|
1K | Default - Create SRS MARC Authority | 24 sec |
5K | Default - Create SRS MARC Authority | 1 min 21 sec |
10K | Default - Create SRS MARC Authority | 2 min 32 sec |
25K | Default - Create SRS MARC Authority | 11 min 14 s |
50K | Default - Create SRS MARC Authority | 22 min |
Comparison to Lotus
MG Duration | Lotus Duration | Delta | |
---|---|---|---|
1K | 24 sec | ||
5K | 1 min 21 sec | ||
10K | 2 min 32 sec | ||
25K | 11 min 14 s | ||
50K | 22 min |
Resource Usages
- CPU usage on mod-data-import-converter-storage increases over time - PERF-282Getting issue details... STATUS
- during a 1 week of testing memory usage increased for mod-srm, mod-srs, mod-data-import-cs, and mod-inventory-storage - JIRA ticket was created - PERF-281Getting issue details... STATUS
- Tasks for mod-srm, and mod-srs were unexpectedly restarted.