Following resources are used:

6 m4large EC2 spot instances for kubernetes cluster;
1 db.r5.xlarge instance for RDS service (writer)
one m5.large per 2 zones for kafks on MSK

...

Previous Kiwi testing performance results:

Data Import Test Report (Kiwi)

...

MAX_REQUEST_SIZE = 4000000 (for all modules)

2 items for all items

2 partition for all DI kafka topics:

Examples:

delete old topic
# ./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --delete --topic perf-eks-folijet.Default.fs09000000.DI_ERROR

recreate topic with "--partitions 2 --replication-factor 1"
# ./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --create --topic perf-eks-folijet.Default.fs09000000.DI_ERROR --partitions 2 --replication-factor 1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic perf-eks-folijet.Default.fs09000000.DI_ERROR.

get topic info
# ./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --describe --topic perf-eks-folijet.Default.fs09000000.DI_ERROR
Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR PartitionCount: 2   ReplicationFactor: 1    Configs: min.insync.replicas=1,message.format.version=2.6-IV0,unclean.leader.election.enable=true
    Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR Partition: 0    Leader: 1   Replicas: 1 Isr: 1
    Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR Partition: 1    Leader: 2   Replicas: 2 Isr: 2 (edited)

mod-source-record-manager: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mod-source-record-manager.hprof -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -Xmx1292m -XX:+UseG1GC" (Memory Hard/Soft limits: 2048Mi/1844Mi, CPU Hard/Soft limits: 1024m/512m)
mod-source-record-storage: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mod-source-record-storage.hprof -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=384m -Xmx908m -XX:+UseG1GC" (Memory Hard/Soft limits: 1440Mi/1296Mi, CPU Hard/Soft limits: 1024m/512m)
mod-inventory: "-Dorg.folio.metadata.inventory.storage.type=okapi -XX:MetaspaceSize=182m -XX:MaxMetaspaceSize=182m -Xmx1814m -XX:+UseG1GC" (Memory Hard/Soft limits: 2880Mi/2592Mi, CPU Hard/Soft limits: 1024m/512m)
mod-inventory-storage: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mod-inventory-storage.hprof -XX:MetaspaceSize=232m -XX:MaxMetaspaceSize=232m -Xmx544m -XX:+UseG1GC" (Memory Hard/Soft limits: 846Mi/778Mi, CPU Hard/Soft limits: 512m/512m)

...

45 222

22 errors → 11 selected as ERROR without any error messages + 2 + 9

io.vertx.core.impl.NoStackTraceThrowable
id value already exists in table holdings_record/instances/items - (9)
proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657
http://mod-inventory-storage: Connection was closed: POST /item-storage/items

50000012h 42min22

61 error:

"contributors[0].name" = "null - (7)

Field 'title' is a required field and can not be null - (1)

proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: Connection was closed - (53)

env	profile	records number	time	kafka partition number	module instance number	cpu	decription
Kiwi https://folio-kiwi.dev.folio.org	default	1025	8 min	1	1
Lotus Perf Rancher	default	1025	23 min	1	1	128	inventory - CPU 200
Lotus Perf Rancher	default	1025	25 min	1	2	128	inventory - CPU 256 2 pods for every module, except data-import
Lotus Perf Rancher	default	1025	11 min	1	1	256	all modules - CPU 256, except data-import
Lotus Perf Rancher	ptf-create2	1025	35 min	1	1	128	inventory - CPU 200
Lotus Perf Rancher	ptf-create2	1025	15 min	1	1	256	all modules - CPU 256, except data-import 4 errors: "id value already exists in table holdings_record"
Lotus Perf Rancher	ptf-create2	1025	6 min	1	1	512	all modules - CPU 512, except data-import
Lotus Perf Rancher	ptf-create2	1025	4 min	1	1	srm, inventory 512/1024 512	srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	ptf-create2	1025	4 min	2	1	srm, inventory 512/1024 512	srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	ptf-create2	1025	3 min	2	2	srm, inventory 512/1024 512	srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	PTF - Updates Success - 1	1025	4 min	2	2	srm, inventory 512/1024 512	srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	ptf-create2	5000	56 min	1	1	256	all modules - CPU 256, except data-import
Lotus Perf Rancher	ptf-create2	5000	16 min	1	1	srm, inventory 512/1024 512	srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	ptf-create2	5000	11 min	2	2	srm, inventory 512/1024 512	srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	ptf-create2	5000	8 min	2	2	srm, srs, inventory 512/1024 512	srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	PTF - Updates Success - 1	5000	13 min	2	2	srm, srs, inventory 512/1024 512	srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	ptf-create2	10000	36 min	1	1	srm, inventory 512/1024 512	srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	ptf-create2	10000	19 min	2	2	srm, srs, inventory 512/1024 512	srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	PTF - Updates Success - 1	10000	25 min	2	2	srm, srs, inventory 512/1024 512	srm, inventory 512/1024, other modules 512, except data-import Completed with errors: org.folio.processing.exceptions.MatchingException "Found multiple records matching"
Lotus Perf Rancher	ptf-create2	30000	5h 34min	1	1	256	all modules - CPU 256, except data-import
Lotus Perf Rancher	ptf-create2	30000	1h 4 min	1	1	srm, srs, inventory 512/1024 512	srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	ptf-create2	50000	1h 19min / 1h 25min	2	PTF - Updates Success - 1	30000	1h 31min	1	1	srm, srs, inventory 512/1024 512	srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	ptf-create2	10000030000	2h 24min45 min	2	2	srm, srs, inventory 512/1024 512	srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	ptf-create2	50000	1h 19min / 1h 25min	2	2	srm, srs, inventory 512/1024 512	srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancher	ptf-create2	50000	1h 34min	1	1	srm, srs, inventory 512/1024 512	srm, srs, inventory 512/1024, other modules 512, except data-import

...

Lotus Perf Rancher

PTF - Updates Success - 1

50000

2h 17min

1

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

ptf-create2

100000

3h 21min

1

srm, srs, inventory 512/1024

...

SRM

SRM Image Removed

...

inventory

inventory Image Removed

...

SRS

SRS Image Removed

...

inventory-storage

...

512	srm, srs, inventory 512/1024

...

SRM

Image Removed

...

inventory

Image Removed

...

SRS

Image Removed

...

Inventory-storage

Image Removed

24h:

SRM:

Image Removed

SRS:

Image Removed

Inventory:

Image Removed

Inventory storage (2 restarts):

Image RemovedImage Removed

Comparation SRM with CPU 256m and 999m

...

Image Removed

...

Image Removed

Tasks for improvement that were created by the results of the tests:

...

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURMAN-695

...

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURCE-461

...

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURMAN-694

...

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODINV-643

...

, other modules 512, except data-import
Lotus Perf Rancher	PTF - Updates Success - 1	100000	4h 40min	1	1	srm, srs, inventory 512/1024 512	srm, srs, inventory 512/1024, other modules 512, except data-import 22 errors → `io.vertx.core.impl.NoStackTraceThrowable: Current retry number 1 exceeded or equal given number 1 for the Item update for jobExecutionId '3eff3c04-055c-4663-8730-05985688a911'`
Lotus Perf Rancher	ptf-create2	100000	2h 24min	2	2	srm, srs, inventory 512/1024 512	srm, srs, inventory 512/1024, other modules 512, except data-import 22 errors → 11 selected as ERROR without any error messages + 2 + 9 io.vertx.core.impl.NoStackTraceThrowable id value already exists in table holdings_record/instances/items - (9) proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: Connection was closed: POST /item-storage/items
Lotus Perf Rancher	ptf-create2	500000	12h 42min	2	2	srm, srs, inventory 512/1024 512	srm, srs, inventory 512/1024, other modules 512, except data-import 61 error: "contributors[0].name" = "null - (7) Field 'title' is a required field and can not be null - (1) proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: Connection was closed - (53)
Lotus Perf Rancher	ptf-create2	500000	15h 37min	2	2	srm, srs, inventory 512/1024 512	srm, srs, inventory 512/1024, other modules 512, except data-import 31 error: "contributors[0].name" = "null - (7) Field 'title' is a required field and can not be null - (1) proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: Connection was closed - (23)

1 pod (srm, inventory 512/1024)

SRM Image Added	inventory Image Added
SRS Image Added	inventory-storage Image Added

2 pods (srm, srs, inventory 512/1024)

SRM Image Added	inventory Image Added
SRS Image Added	Inventory-storage Image Added

24h:

SRM:

Image Added

SRS:

Image Added

Inventory:

Image Added

Inventory storage (2 restarts):

Image AddedImage Added

Comparation SRM with CPU 256m and 999m

SRM CPU 256m	SRM CPU 999m
Image Added	Image Added

Tasks for improvement that were created by the results of the tests:

Jira Legacy

server	System JIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURCEMODSOURMAN-464695

Jira Legacy

server	System JiraJIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURCE-462461

Jira Legacy

server	System JiraJIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODINVMODSOURMAN-646694

Jira Legacy

server	System JiraJIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURMANMODINV-699643

Jira Legacy

server	System JiraJIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODINVMODSOURCE-642464

Jira Legacy

server	System JiraJIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODINVSTORMODSOURCE-878462

Jira Legacy

server	System JiraJIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURCEMODINV-467646

Jira Legacy

server	System JiraJIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURMAN-710699

Jira Legacy

server	System JiraJIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURMANMODINV-711642

Jira Legacy

server	System JiraJIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURCEMODINVSTOR-468878

Jira Legacy

server	System JiraJIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODINV-648
01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURCE-467

Jira Legacy

server	System JIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURMAN-710

Jira Legacy

server	System JIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURMAN-711

Jira Legacy

server	System JIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODSOURCE-468

Jira Legacy

server	System JIRA
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	MODINV-648

Attempt 1: Grouping ERROR import messages of 500K records by type:

io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"must not be null","type":"1","code":"javax.validation.constraints.NotNull.message","parameters":[{"key":"contributors[0].name","value":"null"}]}]} - (6)
io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"must not be null","type":"1","code":"javax.validation.constraints.NotNull.message","parameters":[{"key":"contributors[2].name","value":"null"}]}]} - (1)
io.vertx.core.impl.NoStackTraceThrowable: Mapped Instance is invalid: [Field 'title' is a required field and can not be null], by jobExecutionId: '851a4c9a-7640-4cfc-813a-ff90973bec9d' and recordId: '31932487-67f6-466e-b037-736dc21975f2' and chunkId: '6e166525-8244-48a3-a367-b5e83ee18aad' - (1)
io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: Connection was closed: POST /holdings-storage/holdings - (7)
1. /instance-storage/instances - (9)
2. /item-storage/items - (9)
io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: finishConnect(..) failed: Connection refused: mod-inventory-storage.folijet.svc.cluster.local/172.20.83.37:80: POST /holdings-storage/holdings - (7)
1. /instance-storage/instances - (3)
2. /item-storage/items - (13)
io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: readAddress(..) failed: Connection reset by peer: POST /holdings-storage/holdings - (5)

Attempt 2: Grouping ERROR import messages of 500K records by type:

io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"must not be null","type":"1","code":"javax.validation.constraints.NotNull.message","parameters":[{"key":"contributors[0].name","value":"null"}]}]} - ( 6)
io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"must not be null","type":"1","code":"javax.validation.constraints.NotNull.message","parameters":[{"key":"contributors[2].name","value":"null"}]}]} - ( 1)
io.vertx.core.impl.NoStackTraceThrowable: Mapped Instance is invalid: [Field 'title' is a required field and can not be null], by jobExecutionId: '851a4c9a3d4a1812-76408478-4cfc4554-813ab0c1-ff90973bec9d027afd3ff117' and recordId: '319324871d251c07-67f6b7a3-466e4f22-b0378426-736dc21975f234d415b88f9c' and chunkId: '6e1665255769decb-8244567f-48a347b3-a3678013-b5e83ee18aad172c48f8ee83' - ( 1)
io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 666 http://mod-inventory-storage: Connection was closed: POST /holdings-storage/holdings - (7)6
1. POST /instance-storage/instances - (9)4
2. POST /item-storage/items - (9)3
io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: finishConnect(..) failed: Connection refused: mod-inventory-storage.folijet.svc.cluster.local/172.20.83.37:80: POST /holdings-storage/holdings - (7)
/instance-storage/instances - (3)

/item-storage/items - (13)

666

http://mod-inventory-storage

: readAddress(..) failed: Connection reset by peer: POST /holdings-storage/holdings - 2

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 666 http://mod-inventory-storage: readAddressfinishConnect(..) failed: Connection reset by peerrefused: mod-inventory-storage.folijet.svc.cluster.local/172.20.83.37:80: POST /holdings-storage/holdings - (5)- 2
1. POST /instance-storage/instances - 3
2. POST /item-storage/items - 1

Kafka metrics during 500K DI create

Image Added

Database metrics during 500K DI create

Image Added

Update 10K:

org.folio.processing.exceptions.MatchingException:

...

Failed

...

to

...

encode

...

as

...

JSON:

...

No

...

serializer

...

found

...

for

...

class

...

org.folio.inventory.domain.items.LastCheckIn

...

and

...

no

...

properties

...

discovered

...

to

...

create

...

BeanSerializer

...

(to

...

avoid

...

exception,

...

disable

...

SerializationFeature.FAIL_ON_EMPTY_BEANS)

...

(through

...

reference

...

chain:

...

java.util.ArrayList[3]->org.folio.inventory.domain.items.Item["lastCheckIn"])

Versions Compared

Old Version 84

New Version Current

Key

Following resources are used:

Previous Kiwi testing performance results:

24h:

Tasks for improvement that were created by the results of the tests:

24h:

Tasks for improvement that were created by the results of the tests:

Attempt 1: Grouping ERROR import messages of 500K records by type:

Attempt 2: Grouping ERROR import messages of 500K records by type:

Kafka metrics during 500K DI create

Database metrics during 500K DI create

Image Added

Update 10K:

Page Comparison

Versions Compared

Old Version 84

New Version Current

Key

Previous Kiwi testing performance results:

24h:

Tasks for improvement that were created by the results of the tests:

24h:

Tasks for improvement that were created by the results of the tests:

Attempt 1: Grouping ERROR import messages of 500K records by type:

Attempt 2: Grouping ERROR import messages of 500K records by type:

Kafka metrics during 500K DI create

Database metrics during 500K DI create

Image Added

Update 10K: