Multiple tenant Data Import test report (Nolana)

Owned by Olga Kondratenko
Last updated: Apr 03, 2023 by Martin Tran

IN PROGRESS


Overview

This document contains the results of testing Data Import for MARC Bibliographic records in the Nolana release to detect performance trends for multiple concurrent jobs. PERF-387 - Getting issue details... STATUS


Summary

  • For all tenants, we can observe different data import duration: for example, 50k MARC BIB Create tenant fs09000000 - 21min, tenant fs07000001 - 34 min, and tenant fs07000002 - 42 min. PERF-414 - Getting issue details... STATUS
  • Data import is not able to work on multiple tenants: When data import starts on the second tenant, both jobs are stuck. Data Import becomes unavailable for several hours. PERF-415 - Getting issue details... STATUS Could be related to PERF-388 - Getting issue details... STATUS

Results

Baseline tests


Profile

Duration

tenant fs09000000

Duration

tenant fs07000002

Duration

tenant fs07000001

1K MARC CreatePTF - Create 240 s1 min 19 s55 s

1K MARC Update

PTF - Updates Success - 135 s1 min 24 s1 min
2K MARC CreatePTF - Create 256 s2 min 15 s1 min 37 s
2K MARC UpdatePTF - Updates Success - 158 s2 min 6 s1 min 44 s

5K MARC Create

PTF - Create 22m 8 s5 min 19 s4 min 14 s
5K MARC UpdatePTF - Updates Success - 12 min 10 s5 min3 min 47 s

10K MARC Create 

PTF - Create 24 min 20 s10 min 3 s7 min 20 s
10K MARC Update PTF - Updates Success - 14 min 8 s8 min 43 s8 min 8 s
25K MARC CreatePTF - Create 210 min 41 s23 min 3 s17m 43 s
25K MARC UpdatePTF - Updates Success - 110 min 40 s20 min 47 s17 min 56 s
50K MARC CreatePTF - Create 221 min 11 s42 min 28 s34min 40 s

50K MARC Update

PTF - Updates Success - 120 min 57 s40 min 20 s35 min 24 s
100K MARC CreatePTF - Create 242 min 35 s1 hour 29 min1 hour 20 min

100K MARC Update

PTF - Updates Success - 141 min 56 s1 hour 27 min1 hour 20 min
500K MARC CreatePTF - Create 2---
  • 500K MARC Create import failed due to 500K records file corruption. 

Multiple tenant tests


Profile

Duration

tenant fs09000000

Duration

tenant fs07000002

Duration

tenant fs07000001

50K MARC Create (2 concurrent jobs)

PTF - Create 2-

PERF-415 - Getting issue details... STATUS

PERF-415 - Getting issue details... STATUS

50K MARC Update (2 concurrent jobs)

PTF - Updates Success - 1-DNRDNR
50K MARC Create (3 concurrent jobs)PTF - Create 2DNRDNRDNR

50K MARC Update (3 concurrent jobs)

PTF - Updates Success - 1DNRDNRDNR

Comparison Multiple tenant tests to baseline


Profile

Duration

tenant fs09000000

Baseline

tenant fs09000000

Duration

tenant fs07000002

Baseline

tenant fs07000002

Duration

tenant fs07000001

Baseline

tenant fs07000001

50K MARC Create (2 concurrent jobs)

PTF - Create 2--

PERF-415 - Getting issue details... STATUS


PERF-415 - Getting issue details... STATUS


50K MARC Update (2 concurrent jobs)

PTF - Updates Success - 1--



50K MARC Create (3 concurrent jobs)PTF - Create 2

PERF-415 - Getting issue details... STATUS

21 min 11 s



50K MARC Update (3 concurrent jobs)

PTF - Updates Success - 1DNR20 min 57 s



Nolana version
mod-data-import2.6.2
mod-data-import-converter-storage1.15.2
mod-source-record-manager 3.5.6
mod-source-record-storage5.5.2
mod-inventory19.0.2
mod-inventory-storage25.0.3

Resources usage

Baseline tests

Resources usage MARC Create


Service Memory usage


Service CPU usage

Instance CPU usage

RDS CPU usage

Kafka resources

Resources usage MARC Update

Service Memory usage

Service CPU usage

Instance CPU usage


RDS CPU usage




Issues in the logs observed during testing

FieldValue@ingestionTime1673614146585@log054267740449:ncp3-folio-eis@logStreamncp3/mod-data-export-spring/a353ecd6739e469c8e6cf198b667ab0c@message12:49:01 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR LogAccessor Exception thrown when sending a message with key='6c32c128-fde4-4754-90c4-3499c0c48fca' and payload='JobCommand(type=DELETE, id=6c32c128-fde4-4754-90c4-3499c0c48fca, name=null, description=null, export...' to topic ncp3.fs09000000.data-export.job.command:@timestamp1673614141676

FieldValue@ingestionTime1673614466212@log054267740449:ncp3-folio-eis@logStreamncp3/mod-inventory/b84baa871ca844fda9259c19b83d00ed@message12:54:24 [] [] [] [] ERROR KafkaConsumerWrapper Error while processing a record - id: 16 subscriptionPattern: SubscriptionDefinition(eventType=DI_INVENTORY_INSTANCE_CREATED, subscriptionPattern=ncp3\.Default\.\w{1,}\.DI_INVENTORY_INSTANCE_CREATED) offset: 32104@timestamp1673614464009

Is related to PERF-388 - Getting issue details... STATUS

Or the job can be finished with errors:

io.netty.channel.StacklessClosedChannelException

Instance --- Created
Holdings ---Discarded

FieldValue
@ingestionTime

1673876991708

@log

054267740449:ncp3-folio-eis

@logStream

ncp3/mod-inventory/b84baa871ca844fda9259c19b83d00ed

@message

13:49:49 [] [] [] [] WARN Fetcher [Consumer clientId=consumer-DI_SRS_MARC_BIB_RECORD_MODIFIED.mod-inventory-19.0.2-66, groupId=DI_SRS_MARC_BIB_RECORD_MODIFIED.mod-inventory-19.0.2] Received unknown topic or partition error in fetch for partition ncp3.Default.fs07000001.DI_SRS_MARC_BIB_RECORD_MODIFIED-0

@timestamp

1673876989113

Infrastructure

PTF -environment ncp3

  • 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 2 instances of db.r6.xlarge database instances, one reader, and one writer
  • MSK ptf-kakfa-3
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


Modules memory and CPU parameters

Modules

Version

Task Definition

Running Tasks 

CPU

Memory

MemoryReservation

MaxMetaspaceSize

Xmx

mod-inventory19.0.272102428802592512m1814m

mod-inventory-storage

25.0.3321024

2208

(1872 in MG)

1952

(1684 in MG)

512m1440m
okapi4.14.7131024

1684

(1512 in MG)

1440

(1360 in MG)

512m922m
mod-permissions6.2.082

512

1684

1544

512

1024

mod-search1.8.2224002592248010241440
mod-data-import-cs1.15.1122581024896128m768m
mod-quick-marc2.5.031128

2288

(2098 in MG)

2176

(1920 in MG)

512m1664m
mod-source-record-storage5.5.2421024

1536

(1440 in MG)

1440

(1296 in MG)

512m908m
mod-data-import2.6.24125620481844512m1292m
mod-source-record-manager3.5.6421024

4096

(2048 in MG)

3688

(1844 in MG)

512m

(800M in MG)

2048m

(1024m in MG)

MG- Morning Glory release