It's been found after testing that the actual durations of the imports performed were about 2 (two) times longer than what was reported. This is due to the PTF environment missing a DB trigger that, when restored, doubled the imports' durations.

UIDATIMP-1034 - Getting issue details... STATUS

Overview

This document contains the results of testing Data Import Create MARC holdings records in pre-lotus to detect performance trends.

Infrastructure

6 m5.xlarge EC2 instances
2 instances of db.r6.xlarge database instances, one reader and one writer
MSK
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=120
mod-inventory memory
- 256 CPU units, 1814MB mem
- inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
- inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
- kafka.consumer.max.poll.records=10
mod-inventory-storage
- 128 CPU units, 544MB mem
mod-source-record-storage
- 128 CPU units, 908MB mem
mod-source-record-manager
- 128 CPU units, 1292MB mem
mod-data-import
- 128 CPU units, 1024MB mem

Software versions

All of this snapshot versions were taken in the week of January 10

mod-data-import:2.3.0-SNAPSHOT.78
mod-data-import-converter-storage:1.13.0-SNAPSHOT.69
mod-source-record-storage:5.3.0-SNAPSHOT.132
mod-source-record-manager:3.3.0-SNAPSHOT.115
mod-inventory:18.1.0-SNAPSHOT.344
mod-inventory-storage:23.0.0-SNAPSHOT.573
mod-search:1.6.0-SNAPSHOT.452
mod-quick-marc:2.3.0-SNAPSHOT.47

Results

User Story Acceptance Criteria :

What is the recommended maximum file size a user can upload to create MARC holdings records and also FOLIO holdings records?
- so far we have only 5K file for creation MARC holdings records.
How many MARC holdings records and also FOLIO holdings records can be created in 30 minutes?
- Theoretically it's 50K
How many MARC holdings records and also FOLIO holdings records can be created in an hour?
- Theoretically it's 100K
How long will it take to create 1,000 MARC holdings records and also FOLIO holdings records via data import?
- 25 sec.
How long will it take to create 5,000 MARC holdings records and also FOLIO holdings records via data import?
- 2 min 20 sec

Notable observations

There's issue when running two jobs in parallel (marc holding import and ptf create 2 job)

ERRORS:

10:07:18 [] [] [] [] ERROR HoldingsEventHandler Error loading inventory holdings for MARC BIB. StatusCode: 502. Message: <html>

<head><title>502 Bad Gateway</title></head>

<body>

<center><h1>502 Bad Gateway</h1></center>

</body>

</html>

10:07:18 [] [] [] [] ERROR HoldingsEventHandler Can`t save new holding

org.folio.processing.exceptions.EventProcessingException: <html>

<head><title>502 Bad Gateway</title></head>

<body>

<center><h1>502 Bad Gateway</h1></center>

</body>

</html>

at org.folio.inventory.dataimport.handlers.actions.CreateMarcHoldingsEventHandler.lambda$findInstanceIdByHrid$12(CreateMarcHoldingsEventHandler.java:193) ~[ms.jar:?]

at org.folio.inventory.storage.external.ExternalStorageModuleCollection.interpretMultipleRecordResponse(ExternalStorageModuleCollection.java:267) ~[ms.jar:?]

at org.folio.inventory.storage.external.ExternalStorageModuleCollection.lambda$find$3(ExternalStorageModuleCollection.java:239) ~[ms.jar:?]

at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]

at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]

at io.vertx.ext.web.client.impl.HttpContext.handleDispatchResponse(HttpContext.java:400) ~[ms.jar:?]

at io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:387) ~[ms.jar:?]

at io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:365) ~[ms.jar:?]

at io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:332) ~[ms.jar:?]

at io.vertx.ext.web.client.impl.HttpContext.dispatchResponse(HttpContext.java:294) ~[ms.jar:?]

at io.vertx.ext.web.client.impl.HttpContext.lambda$null$8(HttpContext.java:550) ~[ms.jar:?]

at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[ms.jar:?]

at io.vertx.core.impl.WorkerContext.lambda$run$1(WorkerContext.java:83) ~[ms.jar:?]

at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[ms.jar:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [ms.jar:?]

at java.lang.Thread.run(Thread.java:829) [?:?]

There's issue when running marc holdings with CICO (CICO becomes unstable until the end of a CICO test )