Overview 

This document contains the results of testing Data Import Create MARC holdings records in pre-lotus to detect performance trends.

Software versions

All of this snapshot versions were taken in the week of January 10

• mod-data-import:2.4.2
• mod-data-import-converter-storage:1.13.3
• mod-source-record-storage:5.3.2
• mod-source-record-manager:3.3.8
• mod-inventory:18.1.6
• mod-inventory-storage:23.0.2
• mod-search:1.6.4
• mod-quick-marc:2.3.3

Infrastructure

• 6 m5.xlarge EC2 instances 
• 2 instances of db.r6.xlarge database instances, one reader and one writer
• MSK
  • 4 m5.2xlarge brokers in 2 zones
  • auto.create-topics.enable = true
  • log.retention.minutes=120
• mod-inventory memory
  • 256 CPU units, 1814MB mem
  • inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
  • inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
  • kafka.consumer.max.poll.records=10
• mod-inventory-storage
  • 128 CPU units, 544MB mem
• mod-source-record-storage
  • 128 CPU units, 908MB mem
• mod-source-record-manager
  • 128 CPU units, 1292MB mem
• mod-data-import
  • 128 CPU units, 1024MB mem

User Story Acceptance Criteria :

• What is the recommended maximum file size a user can upload to create MARC holdings records and also FOLIO holdings records?
  • The biggest file we were able to import is 80K (we just don't have bigger files, however system works stable, so supposably it can handle 100K)
• How many MARC holdings records and also FOLIO holdings records can be created in 30 minutes?
  • it's 60K (system was able to import 80K ±40 minutes)
• How many MARC holdings records and also FOLIO holdings records can be created in an hour?
  • Theoretically it's 100K
• How long will it take to create 1,000 MARC holdings records and also FOLIO holdings records via data import?
  • 33 sec.
• How long will it take to create 5,000 MARC holdings records and also FOLIO holdings records via data import?
  • 3 min 40 sec

Results

Test 1K-5K-10K

Test 80K + CICO