| Info |
|---|
It's been found after testing that the actual durations of the imports performed were about 2 (two) times longer than what was reported. This is due to the PTF environment missing a DB trigger that, when restored, doubled the imports' durations. |
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
| Table of Contents |
|---|
Overview
This document contains the results of testing Data Import Create MARC holdings records in pre-lotus to detect performance trends.
Infrastructure
- 6 m5.xlarge EC2 instances
- 2 instances of db.r6.xlarge database instances, one reader and one writer
- MSK
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=120
- mod-inventory memory
- 256 CPU units, 1814MB mem
- inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
- inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
- kafka.consumer.max.poll.records=10
- mod-inventory-storage
- 128 CPU units, 544MB mem
- mod-source-record-storage
- 128 CPU units, 908MB mem
- mod-source-record-manager
- 128 CPU units, 1292MB mem
- mod-data-import
- 128 CPU units, 1024MB mem
Software versions
All of this snapshot versions were taken in the week of January 10
- mod-data-import:2.3.0-SNAPSHOT.78
- mod-data-import-converter-storage:1.13.0-SNAPSHOT.69
- mod-source-record-storage:5.3.0-SNAPSHOT.132
- mod-source-record-manager:3.3.0-SNAPSHOT.115
- mod-inventory:18.1.0-SNAPSHOT.344
- mod-inventory-storage:23.0.0-SNAPSHOT.573
- mod-search:1.6.0-SNAPSHOT.452
- mod-quick-marc:2.3.0-SNAPSHOT.47
Results
User Story Acceptance Criteria :
- What is the recommended maximum file size a user can upload to create MARC holdings records and also FOLIO holdings records?
- so far we have only 5K file for creation MARC holdings records.
- How many MARC holdings records and also FOLIO holdings records can be created in 30 minutes?
- Theoretically it's 50K
- How many MARC holdings records and also FOLIO holdings records can be created in an hour?
- Theoretically it's 100K
- How long will it take to create 1,000 MARC holdings records and also FOLIO holdings records via data import?
- 25 sec.
- How long will it take to create 5,000 MARC holdings records and also FOLIO holdings records via data import?
- 2 min 20 sec
Notable observations
There's issue when running two jobs in parallel (marc holding import and ptf create 2 job)
ERRORS:
10:07:18 [] [] [] [] ERROR HoldingsEventHandler Error loading inventory holdings for MARC BIB. StatusCode: 502. Message: <html> |
<head><title>502 Bad Gateway</title></head> |
<body> |
<center><h1>502 Bad Gateway</h1></center> |
</body> |
</html> |
10:07:18 [] [] [] [] ERROR HoldingsEventHandler Can`t save new holding |
org.folio.processing.exceptions.EventProcessingException: <html> |
<head><title>502 Bad Gateway</title></head> |
<body> |
<center><h1>502 Bad Gateway</h1></center> |
</body> |
</html> |
at org.folio.inventory.dataimport.handlers.actions.CreateMarcHoldingsEventHandler.lambda$findInstanceIdByHrid$12(CreateMarcHoldingsEventHandler.java:193) ~[ms.jar:?] |
at org.folio.inventory.storage.external.ExternalStorageModuleCollection.interpretMultipleRecordResponse(ExternalStorageModuleCollection.java:267) ~[ms.jar:?] |
at org.folio.inventory.storage.external.ExternalStorageModuleCollection.lambda$find$3(ExternalStorageModuleCollection.java:239) ~[ms.jar:?] |
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?] |
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] |
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?] |
at io.vertx.ext.web.client.impl.HttpContext.handleDispatchResponse(HttpContext.java:400) ~[ms.jar:?] |
at io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:387) ~[ms.jar:?] |
at io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:365) ~[ms.jar:?] |
at io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:332) ~[ms.jar:?] |
at io.vertx.ext.web.client.impl.HttpContext.dispatchResponse(HttpContext.java:294) ~[ms.jar:?] |
at io.vertx.ext.web.client.impl.HttpContext.lambda$null$8(HttpContext.java:550) ~[ms.jar:?] |
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[ms.jar:?] |
at io.vertx.core.impl.WorkerContext.lambda$run$1(WorkerContext.java:83) ~[ms.jar:?] |
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[ms.jar:?] |
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] |
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] |
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [ms.jar:?] |
at java.lang.Thread.run(Thread.java:829) [?:?] |
There's issue when running marc holdings with CICO (CICO becomes unstable until the end of a CICO test )
