Overview

Compared to Honeysuckle, Data Import in Iris is more robust, predictable, and reliable.  Up to 50K MARC records can be imported reliably in a reasonable 2 hours, and up to 100K records may be imported in less than 8 hours. These imports create new instances, holdings, and items. Tests were performed on imports that UPDATE records but there are still ongoing issues in this workflow. 

Profiles used for PTF testing

PTF Test Results

Environment

The following combination of hardware specs and software configuration on which data import tests were successfully performed.

...

• inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
• inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10

AWS MSK Configurations:

• v2.7.0
• 2 Brokers: kafka.m5.large, 500 GiB each
• auto.create.topics.enable=true
• log.retention.minutes=300
• Data Import Topics
  • Number of partitions: 1
  • Replication Factor = 2
• mod-inventory's topics (inventory.instance, inventory.holdings-record, inventory.item)
  • Number of partitions: 50
  • Replication Factor = 2

Troubleshooting