PERF-210 - Getting issue details... STATUS

UIDATIMP-1033 - Getting issue details... STATUS

Overview 

This document contains the results of testing Data Import MARC Authorities in pre-lotus to detect performance trends. 

This document contains only stand alone MARC Authorities imports without background activities (Check-in/Check-out). Check-in/Check-out testing wasn't performed in parallel to imports due to issues with Check-in/Check-out workflow (possibly because of snapshot versions of modules)

Infrastructure

• 6 m5.xlarge EC2 instances 
• 2 instances of db.r6.xlarge database instances, one reader and one writer
• MSK
  • 4 m5.2xlarge brokers in 2 zones
  • auto.create-topics.enable = true
  • log.retention.minutes=120
• mod-inventory memory
  • 256 CPU units, 1814MB mem
  • inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
  • inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
  • kafka.consumer.max.poll.records=10
• mod-inventory-storage
  • 128 CPU units, 544MB mem
• mod-source-record-storage
  • 128 CPU units, 908MB mem
• mod-source-record-manager
  • 128 CPU units, 1292MB mem
• mod-data-import
  • 128 CPU units, 1024MB mem

Software versions

All of this snapshot versions were taken in the week of January 10

• mod-data-import-2.3.0-SNAPSHOT.220
• mod-data-import-converter-storage-1.13.0-SNAPSHOT.183
• mod-source-record-storage-5.3.0-SNAPSHOT.378
• mod-source-record-manager-3.3.0-SNAPSHOT.526
• mod-inventory-18.1.0-SNAPSHOT.475
• mod-inventory-storage-22.1.0-SNAPSHOT.650
• mod-search-1.6.0-SNAPSHOT.172
• mod-quick-marc-2.3.0-SNAPSHOT.145
• folio/marc-authorities 1.0.100034 (folio_marc-authorities-1.0.100034)

Results

User Story Acceptance Criteria :

Answering questions:

• What is the recommended maximum file size a user can upload to create MARC authority records?    
  • Maximum file size tried - 371K, however there is problems with this particular file (chunk issue). Maximum file that passed successfully - 50K 
• How many MARC authority records can be created in 30 minutes?
  • ±100K . all of tests above were performed in a row. 
• How many MARC authority records can be created in an hour?
  • -----
• How long will it take to create 1,000 MARC authority records via data import?
  • 40 sec
• How long will it take to create 5,000 MARC authority records via data import?
  • 1 min 14 sec
• How long will it take to create 10,000 MARC authority records via data import?
  • 2 min 32 sec
• How long will it take to create 20,000 MARC authority records via data import
  • <6 min

Resource Usages

Notable observations: 

• More or less resource usages are the same as for KIWI release Data Import Test Report (Kiwi)#5K-10KImports.
• With part of files we've got error: [vert.x-worker-thread-10] ERROR rdChunksKafkaHandler [72160511eqId] RecordCollection processing has failed with errors with event: 'null', chunkId: '6c96f806-e9f5-4720-b680-b44361a3a0e7', chunkNumber '2119'-'19' with recordId: 'null'.   UIDATIMP-1087 - Getting issue details... STATUS

max resource usage:

max resource usage:

For all test set instances CPU was less than 50%

Results after modules updating