Versions Compared

Old Version 82

changes.mady.by.user Aliaksandr Fedasiuk

Saved on

compared with

New Version Current

changes.mady.by.user Aliaksandr Fedasiuk

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Following resources are used:

  • 6 m4large EC2 spot instances for kubernetes cluster;
  • 1 db.r5.xlarge instance for RDS service (writer)
  • one m5.large per 2 zones for kafks on MSK

...

Previous Kiwi testing performance results:

Data Import Test Report (Kiwi)

...

MAX_REQUEST_SIZE = 4000000 (for all modules)

2 items for all items

2 partition for all DI kafka topics:

Examples:

delete old topic
# ./kafka-topics.sh --bootstrap-server=b-1.kafka-folijet.p1n5o2.c11.kafka.us-west-2.amazonaws.com<kafka-ip>:9092 --delete --topic perf-eks-folijet.Default.fs09000000.DI_ERROR

recreate topic with "--partitions 2 --replication-factor 1"
# ./kafka-topics.sh --bootstrap-server=b-1.kafka<kafka-ip>:9092 --create --topic perf-eks-folijet.p1n5o2.c11.kafka.us-west-2.amazonaws.com:9092 --create --topic perf-eks-folijet.Default.Default.fs09000000.DI_ERROR --partitions 2 --replication-factor 1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic perf-eks-folijet.Default.fs09000000.DI_ERROR.

get topic info
# ./kafka-topics.sh --bootstrap-server=b-1.kafka-folijet.p1n5o2.c11.kafka.us-west-2.amazonaws.com<kafka-ip>:9092 --describe --topic perf-eks-folijet.Default.fs09000000.DI_ERROR
Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR PartitionCount: 2   ReplicationFactor: 1    Configs: min.insync.replicas=1,message.format.version=2.6-IV0,unclean.leader.election.enable=true
    Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR Partition: 0    Leader: 1   Replicas: 1 Isr: 1
    Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR Partition: 1    Leader: 2   Replicas: 2 Isr: 2 (edited) 


  • mod-source-record-manager: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mod-source-record-manager.hprof -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -Xmx1292m -XX:+UseG1GC"  (Memory Hard/Soft limits: 2048Mi/1844Mi, CPU Hard/Soft limits: 1024m/512m)
  • mod-source-record-storage: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mod-source-record-storage.hprof -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=384m -Xmx908m -XX:+UseG1GC"  (Memory Hard/Soft limits: 1440Mi/1296Mi, CPU Hard/Soft limits: 1024m/512m)
  • mod-inventory: "-Dorg.folio.metadata.inventory.storage.type=okapi -XX:MetaspaceSize=182m -XX:MaxMetaspaceSize=182m -Xmx1814m -XX:+UseG1GC" (Memory Hard/Soft limits: 2880Mi/2592Mi, CPU Hard/Soft limits: 1024m/512m)
  • mod-inventory-storage: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mod-inventory-storage.hprof -XX:MetaspaceSize=232m -XX:MaxMetaspaceSize=232m -Xmx544m -XX:+UseG1GC"  (Memory Hard/Soft limits: 846Mi/778Mi, CPU Hard/Soft limits: 512m/512m)

Tests:

45 222

22 errors → 11 selected as ERROR without any error messages + 2 + 9

  1. io.vertx.core.impl.NoStackTraceThrowable
  2. id value already exists in table holdings_record/instances/items - (9)
  3. proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 
            http://mod-inventory-storage: Connection was closed: POST /item-storage/items
50000012h 42min22

61 error:

"contributors[0].name" = "null - (7)

Field 'title' is a required field and can not be null - (1)

proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: Connection was closed - (53)

envprofile

records

number

time

kafka

partition

number

module

instance

number

cpudecription

Kiwi

https://folio-kiwi.dev.folio.org

default

10258 min11

Lotus Perf Rancherdefault
102523 min11128inventory - CPU 200
Lotus Perf Rancherdefault
102525 min12128

inventory - CPU 256

2 pods for every module, except data-import

Lotus Perf Rancherdefault
102511 min11256all modules - CPU 256, except data-import
Lotus Perf Rancherptf-create2
102535 min11128inventory - CPU 200
Lotus Perf Rancherptf-create2
102515 min11256

all modules - CPU 256, except data-import

4 errors: "id value already exists in table holdings_record"

Lotus Perf Rancherptf-create210256 min11512all modules - CPU 512, except data-import
Lotus Perf Rancherptf-create2
10254 min11

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancherptf-create210254 min21

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancherptf-create210253 min22

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf RancherPTF - Updates Success - 110254 min22

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancherptf-create2
500056 min11256all modules - CPU 256, except data-import
Lotus Perf Rancherptf-create2500016 min11

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancherptf-create2500011 min22

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancherptf-create250008 min22

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf RancherPTF - Updates Success - 1500013 min22

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancherptf-create21000036 min11

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancherptf-create21000019 min22

srm, srs, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import
Lotus Perf RancherPTF - Updates Success - 11000025 min22

srm, srs, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancherptf-create2

Completed with errors:

org.folio.processing.exceptions.MatchingException

"Found multiple records matching"

Lotus Perf Rancherptf-create2
300005h 34min11256all modules - CPU 256, except data-import
Lotus Perf Rancherptf-create2
300001h 4 min11

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancherptf-create2500001h 19min / 1h 25min2PTF - Updates Success - 1300001h 31min11

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancherptf-create2100000300002h 24min45 min22

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancherptf-create2500001h 19min / 1h 25min22

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancherptf-create2500001h 34min11

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

1 pod (srm, inventory 512/1024)

...

SRM

SRMImage Removed

...

inventory

inventoryImage Removed

...

SRS

SRSImage Removed

...

inventory-storage

...

Lotus Perf RancherPTF - Updates Success - 1500002h 17min11

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf Rancherptf-create21000003h 21min11

srm, srs, inventory 512/1024

...

SRM

Image Removed

inventory

Image Removed

SRS

Image Removed

Inventory-storage

Image Removed

24h:

SRM:

Image Removed

SRS:

Image Removed

Inventory:

Image Removed

Inventory storage (2 restarts):

Image RemovedImage Removed

Comparation SRM with CPU 256m and 999m

...

Image Removed

...

Image Removed

Tasks for improvement that were created by the results of the tests:

...

Jira Legacy
serverSystem Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODSOURMAN-695

...

Jira Legacy
serverSystem Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODSOURCE-461

...

Jira Legacy
serverSystem Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODSOURMAN-694

...

Jira Legacy
serverSystem Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODINV-643

...

512

srm, srs, inventory 512/1024, other modules 512, except data-import
Lotus Perf RancherPTF - Updates Success - 11000004h 40min11

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

22 errors → 

io.vertx.core.impl.NoStackTraceThrowable: Current retry number 1 exceeded or equal given number 1 
for the Item update for jobExecutionId '3eff3c04-055c-4663-8730-05985688a911'
Lotus Perf Rancherptf-create21000002h 24min22

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

22 errors → 11 selected as ERROR without any error messages + 2 + 9

  1. io.vertx.core.impl.NoStackTraceThrowable
  2. id value already exists in table holdings_record/instances/items - (9)
  3. proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 
            http://mod-inventory-storage: Connection was closed: POST /item-storage/items

Lotus Perf Rancherptf-create250000012h 42min22

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

61 error:

"contributors[0].name" = "null - (7)

Field 'title' is a required field and can not be null - (1)

proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: Connection was closed - (53)

Lotus Perf Rancherptf-create250000015h 37min22

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

31 error:

"contributors[0].name" = "null - (7)

Field 'title' is a required field and can not be null - (1)

proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: Connection was closed - (23)


1 pod (srm, inventory 512/1024)

SRM

SRMImage Added

inventory

inventoryImage Added

SRS

SRSImage Added

inventory-storage

inventory-storageImage Added


2 pods (srm, srs, inventory 512/1024)

SRM

Image Added

inventory

Image Added

SRS

Image Added

Inventory-storage

Image Added



24h:

SRM:

Image Added


SRS:

Image Added


Inventory:

Image Added


Inventory storage (2 restarts):

Image AddedImage Added


Comparation SRM with CPU 256m and 999m

SRM CPU 256mSRM CPU 999m

Image Added

Image Added



Tasks for improvement that were created by the results of the tests:

  1. Jira Legacy
    01505d01-b853-
    serverSystem JIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolutionserverId
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODSOURCEMODSOURMAN-464695
  2. Jira Legacy
    serverSystem JiraJIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODSOURCE-462461
  3. Jira Legacy
    serverSystem JiraJIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODINVMODSOURMAN-646694
  4. Jira Legacy
    serverSystem JiraJIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODSOURMANMODINV-699643
  5. Jira Legacy
    serverSystem JiraJIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODINVMODSOURCE-642464
  6. Jira Legacy
    serverSystem JiraJIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODINVSTORMODSOURCE-878462
  7. Jira Legacy
    serverSystem JiraJIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODSOURCEMODINV-467646
  8. Jira Legacy
    serverSystem JiraJIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODSOURMAN-710699
  9. Jira Legacy
    serverSystem JiraJIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODSOURMANMODINV-711642
  10. Jira Legacy
    serverSystem JiraJIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODSOURCEMODINVSTOR-468878
  11. Jira Legacy
    serverSystem JiraJIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODINV-648
    -3c2e-90f1-ee9b165564fc
    keyMODSOURCE-467
  12. Jira Legacy
    serverSystem JIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODSOURMAN-710
  13. Jira Legacy
    serverSystem JIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODSOURMAN-711
  14. Jira Legacy
    serverSystem JIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODSOURCE-468
  15. Jira Legacy
    serverSystem JIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODINV-648


Attempt 1: Grouping ERROR import messages of 500K records by type:

  1. io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"must not be null","type":"1","code":"javax.validation.constraints.NotNull.message","parameters":[{"key":"contributors[0].name","value":"null"}]}]} - (6)
  2. io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"must not be null","type":"1","code":"javax.validation.constraints.NotNull.message","parameters":[{"key":"contributors[2].name","value":"null"}]}]} - (1)
  3. io.vertx.core.impl.NoStackTraceThrowable: Mapped Instance is invalid: [Field 'title' is a required field and can not be null], by jobExecutionId: '851a4c9a-7640-4cfc-813a-ff90973bec9d' and recordId: '31932487-67f6-466e-b037-736dc21975f2' and chunkId: '6e166525-8244-48a3-a367-b5e83ee18aad'  - (1)
  4. io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: Connection was closed: POST /holdings-storage/holdings - (7)
    1. /instance-storage/instances - (9)
    2. /item-storage/items - (9)
  5. io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: finishConnect(..) failed: Connection refused: mod-inventory-storage.folijet.svc.cluster.local/172.20.83.37:80: POST /holdings-storage/holdings - (7)
    1. /instance-storage/instances - (3)
    2. /item-storage/items - (13)
  6. io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: readAddress(..) failed: Connection reset by peer: POST /holdings-storage/holdings - (5)

Attempt 2: Grouping ERROR import messages of 500K records by type:

  1. io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"must not be null","type":"1","code":"javax.validation.constraints.NotNull.message","parameters":[{"key":"contributors[0].name","value":"null"}]}]} - ( 6)
  2. io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"must not be null","type":"1","code":"javax.validation.constraints.NotNull.message","parameters":[{"key":"contributors[2].name","value":"null"}]}]} - ( 1)
  3. io.vertx.core.impl.NoStackTraceThrowable: Mapped Instance is invalid: [Field 'title' is a required field and can not be null], by jobExecutionId: '851a4c9a3d4a1812-76408478-4cfc4554-813ab0c1-ff90973bec9d027afd3ff117' and recordId: '319324871d251c07-67f6b7a3-466e4f22-b0378426-736dc21975f234d415b88f9c' and chunkId: '6e1665255769decb-8244567f-48a347b3-a3678013-b5e83ee18aad172c48f8ee83'   - ( 1)
  4. io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 666 http://mod-inventory-storage: Connection was closed: POST /holdings-storage/holdings - (7)6
    1. POST /instance-storage/instances - 4
    2. POST /item-storage/items - 3
  5. io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0- (9)/itemSNAPSHOT.666 http://mod-inventory-storage: readAddress(..) failed: Connection reset by peer: POST /holdings-storage/items holdings - (9)2
  6. io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: finishConnect(..) failed: Connection refused: mod-inventory-storage.folijet.svc.cluster.local/172.20.83.37:80: POST /holdings-storage/holdings - (7)
    1. /instance-storage/instances - (3)
    2. /item-storage/items - (13)
    io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: readAddress(..) failed: Connection reset by peer: POST /holdings-storage/holdings - (5666 http://mod-inventory-storage: finishConnect(..) failed: Connection refused: mod-inventory-storage.folijet.svc.cluster.local/172.20.83.37:80: POST /holdings-storage/holdings - 2
    1. POST /instance-storage/instances - 3
    2. POST /item-storage/items - 1


Kafka metrics during 500K DI create

Image Added


Database metrics during 500K DI create

Image Added

Update 10K:

org.folio.processing.exceptions.MatchingException:

Failed to encode as JSON: No serializer found for class org.folio.inventory.domain.items.LastCheckIn and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS)
(through reference chain: java.util.ArrayList[3]->org.folio.inventory.domain.items.Item["lastCheckIn"])