UIDATIMP-1034 - Getting issue details... STATUS

Overview 

This document contains the results of testing Data Import Create MARC holdings records in pre-lotus to detect performance trends.

Infrastructure

• 6 m5.xlarge EC2 instances 
• 2 instances of db.r6.xlarge database instances, one reader and one writer
• MSK
  • 4 m5.2xlarge brokers in 2 zones
  • auto.create-topics.enable = true
  • log.retention.minutes=120
• mod-inventory memory
  • 256 CPU units, 1814MB mem
  • inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
  • inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
  • kafka.consumer.max.poll.records=10
• mod-inventory-storage
  • 128 CPU units, 544MB mem
• mod-source-record-storage
  • 128 CPU units, 908MB mem
• mod-source-record-manager
  • 128 CPU units, 1292MB mem
• mod-data-import
  • 128 CPU units, 1024MB mem

Software versions

All of this snapshot versions were taken in the week of January 10

• mod-data-import:2.3.0-SNAPSHOT.78
• mod-data-import-converter-storage:1.13.0-SNAPSHOT.69
• mod-source-record-storage:5.3.0-SNAPSHOT.132
• mod-source-record-manager:3.3.0-SNAPSHOT.115
• mod-inventory:18.1.0-SNAPSHOT.344
• mod-inventory-storage:23.0.0-SNAPSHOT.573
• mod-search:1.6.0-SNAPSHOT.452
• mod-quick-marc:2.3.0-SNAPSHOT.47

Results

User Story Acceptance Criteria :

• What is the recommended maximum file size a user can upload to create MARC holdings records and also FOLIO holdings records?
  • so far we have only 5K file for creation MARC holdings records.
• How many MARC holdings records and also FOLIO holdings records can be created in 30 minutes?
  • Theoretically it's 50K
• How many MARC holdings records and also FOLIO holdings records can be created in an hour?
• How long will it take to create 1,000 MARC holdings records and also FOLIO holdings records via data import?
• How long will it take to create 5,000 MARC holdings records and also FOLIO holdings records via data import?