Data Import MARC Authorities (Morning Glory)

Owned by Roman_Fedynyshyn
Last updated: Aug 17, 2023 by Martin Tran

It's been found after testing that the actual durations of the imports performed were about 2 (two) times longer than what was reported. This is due to the PTF environment missing a DB trigger that, when restored, doubled the imports' durations.


Overview 

This is a test report of MARC Authorities Data Import running on the Morning Glory software. 

Infrastructure

  • 10 m6i.2xlarge EC2 instances  (changed. In Lotus it was m5.xlarge)
  • 2 instances of db.r6.xlarge database instances, one reader and one writer
  • MSK
    • 4 m5.2xlarge brokers in 2 zones
    • auto.create-topics.enable = true
    • log.retention.minutes=120
    • 2 partitions per DI topics
  • mod-inventory memory
    • 1024 CPU units, 2592MB mem
    • inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
    • inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
    • kafka.consumer.max.poll.records=10
  • mod-inventory-storage
    • 1024 CPU units, 1684MB mem
  • mod-source-record-storage
    • 1024 CPU units, 1296MB mem
  • mod-source-record-manager
    • 1024 CPU units, 1844MB mem
  • mod-data-import
    • 256 CPU units, 1844MB mem
  • mod-data-import-cs 
    • 128 CPU units, 896MB mem


Infrastructure comparison with MG vs Lotus

Please note that infrastructure has being migrated from 6 m5.xlarge EC2 instances to 10 m6i.2xlarge. Key differences below:

Instance

CPU

RAM

m5.xlarge416 
m6i.2xlarge832


Differences in modules memory and CPU parameters

ModuleLotusMorning Glory
CPU RAMCPURAM
mod-inventory2561814MB10242592MB
mod-inventory-storage128544MB10241684MB
mod-SRS128908MB10241296MB
mod-SRM1281292MB10241844MB
mod-data-import1281024MB2561844MB
mod-data-import-cs--128896MB



Software versions

  • mod-data-import v2.5.0
  • mod-data-import-converter-storage v1.14.0
  • mod-source-record-manager v3.4.0
  • mod-source-record-storage v5.4.0
  • mod-inventory v18.2.0
  • mod-inventory-storage v24.0.0



Results

Summary



Profile

Duration

1KDefault - Create SRS MARC Authority

24 sec

5K

Default - Create SRS MARC Authority1 min 21 sec
10KDefault - Create SRS MARC Authority2 min 32 sec
25KDefault - Create SRS MARC Authority11 min 14 s
50KDefault - Create SRS MARC Authority22 min

Comparison to Lotus


MG Duration

Lotus DurationDelta
1K

24 sec



5K

1 min 21 sec

10K2 min 32 sec

25K11 min 14 s

50K22 min

Resource Usages

  • CPU usage on mod-data-import-converter-storage increases over time PERF-282 - Getting issue details... STATUS


  • during a 1 week of testing memory usage increased for mod-srm, mod-srs, mod-data-import-cs, and mod-inventory-storage - JIRA ticket was created  PERF-281 - Getting issue details... STATUS
  • Tasks for mod-srm, and mod-srs were unexpectedly restarted.