/
Folijet - Lotus Snapshot Performance testing

Folijet - Lotus Snapshot Performance testing

Apr 04, 2022

Following resources are used:

  • 6 m4large EC2 spot instances for kubernetes cluster;

  • 1 db.r5.xlarge instance for RDS service (writer)

  • one m5.large per 2 zones for kafks on MSK

Previous Kiwi testing performance results:

Data Import Test Report (Kiwi)

Modules:

Data Import Module (mod-data-import-2.3.0-SNAPSHOT.224)

Source Record Manager Module (mod-source-record-manager-3.3.0-SNAPSHOT.556)

Source Record Storage Module (mod-source-record-storage-5.3.0-SNAPSHOT.394)

Inventory Module (mod-inventory-18.1.0-SNAPSHOT.493)

Inventory Storage Module (mod-inventory-storage-23.0.0-SNAPSHOT.657)

Invoice business logic module (mod-invoice-5.3.0-SNAPSHOT.284)

 

Initial configuration:

kcp1-mod-data-import: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-data-import.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MaxRAMPercentage=66.0 -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=384m -Xmx1024m"  (Hard/Soft limits: 2048/1024)

kcp1-mod-source-record-manager: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-source-record-manager.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MaxRAMPercentage=66.0 -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -Xmx1292m"  (Hard/Soft limits: 2048/1844)

kcp1-mod-source-record-storage: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-source-record-storage.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MaxRAMPercentage=66.0 -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=384m -Xmx908m"  (Hard/Soft limits: 1440/1296)

kcp1-mod-inventory: "-Dport=8082 -XX:MaxRAMPercentage=66.0 -Dorg.folio.metadata.inventory.storage.type=okapi -XX:MetaspaceSize=182m -XX:MaxMetaspaceSize=182m -Xmx1814m" (Hard/Soft limits: 2880/2592)

kcp1-mod-inventory-storage: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-inventory-storage.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=232m -XX:MaxMetaspaceSize=232m -Xmx544m"  (Hard/Soft limits: 846/778)

kcp1-mod-invoice: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-invoice.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=88m -Xmx360m" (Hard/Soft limits: 512/360)

 

Performance-optimized configuration:

MAX_REQUEST_SIZE = 4000000 (for all modules)

2 items for all items

2 partition for all DI kafka topics:

Examples:

delete old topic
# ./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --delete --topic perf-eks-folijet.Default.fs09000000.DI_ERROR

recreate topic with "--partitions 2 --replication-factor 1"
# ./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --create --topic perf-eks-folijet.Default.fs09000000.DI_ERROR --partitions 2 --replication-factor 1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic perf-eks-folijet.Default.fs09000000.DI_ERROR.

get topic info
# ./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --describe --topic perf-eks-folijet.Default.fs09000000.DI_ERROR
Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR PartitionCount: 2   ReplicationFactor: 1    Configs: min.insync.replicas=1,message.format.version=2.6-IV0,unclean.leader.election.enable=true
    Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR Partition: 0    Leader: 1   Replicas: 1 Isr: 1
    Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR Partition: 1    Leader: 2   Replicas: 2 Isr: 2 (edited) 

 

  • mod-source-record-manager: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mod-source-record-manager.hprof -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -Xmx1292m -XX:+UseG1GC"  (Memory Hard/Soft limits: 2048Mi/1844Mi, CPU Hard/Soft limits: 1024m/512m)

  • mod-source-record-storage: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mod-source-record-storage.hprof -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=384m -Xmx908m -XX:+UseG1GC"  (Memory Hard/Soft limits: 1440Mi/1296Mi, CPU Hard/Soft limits: 1024m/512m)

  • mod-inventory: "-Dorg.folio.metadata.inventory.storage.type=okapi -XX:MetaspaceSize=182m -XX:MaxMetaspaceSize=182m -Xmx1814m -XX:+UseG1GC" (Memory Hard/Soft limits: 2880Mi/2592Mi, CPU Hard/Soft limits: 1024m/512m)

  • mod-inventory-storage: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mod-inventory-storage.hprof -XX:MetaspaceSize=232m -XX:MaxMetaspaceSize=232m -Xmx544m -XX:+UseG1GC"  (Memory Hard/Soft limits: 846Mi/778Mi, CPU Hard/Soft limits: 512m/512m)

Tests:

env

profile

records

number

time

kafka

partition

number

module

instance

number

cpu

decription

env

profile

records

number

time

kafka

partition

number

module

instance

number

cpu

decription

Kiwi

https://folio-kiwi.dev.folio.org

default

1025

8 min

1

1

 

 

Lotus Perf Rancher

default

1025

23 min

1

1

128

inventory - CPU 200

Lotus Perf Rancher

default

1025

25 min

1

2

128

inventory - CPU 256

2 pods for every module, except data-import

Lotus Perf Rancher

default

1025

11 min

1

1

256

all modules - CPU 256, except data-import

Lotus Perf Rancher

ptf-create2

1025

35 min

1

1

128

inventory - CPU 200

Lotus Perf Rancher

ptf-create2

1025

15 min

1

1

256

all modules - CPU 256, except data-import

4 errors: "id value already exists in table holdings_record"

Lotus Perf Rancher

ptf-create2

1025

6 min

1

1

512

all modules - CPU 512, except data-import

Lotus Perf Rancher

ptf-create2

1025

4 min

1

1

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

ptf-create2

1025

4 min

2

1

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

ptf-create2

1025

3 min

2

2

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

PTF - Updates Success - 1

1025

4 min

2

2

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

ptf-create2

5000

56 min

1

1

256

all modules - CPU 256, except data-import

Lotus Perf Rancher

ptf-create2

5000

16 min

1

1

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

ptf-create2

5000

11 min

2

2

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

ptf-create2

5000

8 min

2

2

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

PTF - Updates Success - 1

5000

13 min

2

2

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

ptf-create2

10000

36 min

1

1

srm, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

ptf-create2

10000

19 min

2

2

srm, srs, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

PTF - Updates Success - 1

10000

25 min

2

2

srm, srs, inventory 512/1024

512

srm, inventory 512/1024, other modules 512, except data-import

Completed with errors:

org.folio.processing.exceptions.MatchingException

"Found multiple records matching"

Lotus Perf Rancher

ptf-create2

30000

5h 34min

1

1

256

all modules - CPU 256, except data-import

Lotus Perf Rancher

ptf-create2

30000

1h 4 min

1

1

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

PTF - Updates Success - 1

30000

1h 31min

1

1

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

ptf-create2

30000

45 min

2

2

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

ptf-create2

50000

1h 19min / 1h 25min

2

2

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

ptf-create2

50000

1h 34min

1

1

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

PTF - Updates Success - 1

50000

2h 17min

1

1

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

ptf-create2

100000

3h 21min

1

1

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

Lotus Perf Rancher

PTF - Updates Success - 1

100000

4h 40min

1

1

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

22 errors → 

io.vertx.core.impl.NoStackTraceThrowable: Current retry number 1 exceeded or equal given number 1
for the Item update for jobExecutionId '3eff3c04-055c-4663-8730-05985688a911'

Lotus Perf Rancher

ptf-create2

100000

2h 24min

2

2

srm, srs, inventory 512/1024

512

srm, srs, inventory 512/1024, other modules 512, except data-import

22 errors → 11 selected as ERROR without any error messages + 2 + 9

  1. io.vertx.core.impl.NoStackTraceThrowable

  2. id value already exists in table holdings_record/instances/items - (9)