Skip to end of banner
Go to start of banner

Data Import MARC Authorities (Morning Glory)

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Current »

IN PROGRESS




Overview 

Infrastructure

  • 10 m6i.2xlarge EC2 instances  (changed. In Lotus it was m5.xlarge)
  • 2 instances of db.r6.xlarge database instances, one reader and one writer
  • MSK
    • 4 m5.2xlarge brokers in 2 zones
    • auto.create-topics.enable = true
    • log.retention.minutes=120
    • 2 partitions per DI topics
  • mod-inventory memory
    • 1024 CPU units, 2592MB mem
    • inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
    • inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
    • kafka.consumer.max.poll.records=10
  • mod-inventory-storage
    • 1024 CPU units, 1684MB mem
  • mod-source-record-storage
    • 1024 CPU units, 1296MB mem
  • mod-source-record-manager
    • 1024 CPU units, 1844MB mem
  • mod-data-import
    • 256 CPU units, 1844MB mem
  • mod-data-import-cs 
    • 128 CPU units, 896MB mem


Infrastructure comparison with MG vs Lotus

Please note that infrastructure has being migrated from 6 m5.xlarge EC2 instances to 10 m6i.2xlarge. Key differences below:

Instance

CPU

RAM

m5.xlarge416 
m6i.2xlarge832


Differences in modules memory and CPU parameters

ModuleLotusMorning Glory
CPU RAMCPURAM
mod-inventory2561814MB10242592MB
mod-inventory-storage128544MB10241684MB
mod-SRS128908MB10241296MB
mod-SRM1281292MB10241844MB
mod-data-import1281024MB2561844MB
mod-data-import-cs--128896MB



Software versions

  • mod-data-import v2.5.0
  • mod-data-import-converter-storage v1.14.0
  • mod-source-record-manager v3.4.0
  • mod-source-record-storage v5.4.0
  • mod-inventory v18.2.0
  • mod-inventory-storage v24.0.0



Results

Summary



Profile

Duration

1KDefault - Create SRS MARC Authority

24 sec

5K

Default - Create SRS MARC Authority1 min 21 sec
10KDefault - Create SRS MARC Authority2 min 32 sec
25KDefault - Create SRS MARC Authority11 min 14 s
50KDefault - Create SRS MARC Authority22 min

Comparison to Lotus


MG Duration

Lotus DurationDelta
1K

24 sec



5K

1 min 21 sec

10K2 min 32 sec

25K11 min 14 s

50K22 min

Resource Usages

  • CPU usage on mod-data-import-converter-storage increases over time PERF-282 - Getting issue details... STATUS


  • during a 1 week of testing memory usage increased for mod-srm, mod-srs, mod-data-import-cs, and mod-inventory-storage - JIRA ticket was created  PERF-281 - Getting issue details... STATUS
  • Tasks for mod-srm, and mod-srs were unexpectedly restarted.

  • No labels