Following resources are used:
- 6 m4large EC2 spot instances for kubernetes cluster;
- 1 db.r5.xlarge instance for RDS service (writer)
- one m5.large per 2 zones for kafks on MSK
KIWI testing performance results:
Data Import Test Report (Kiwi)
Modules:
Data Import Module (mod-data-import-2.3.0-SNAPSHOT.224)
Source Record Manager Module (mod-source-record-manager-3.3.0-SNAPSHOT.556)
Source Record Storage Module (mod-source-record-storage-5.3.0-SNAPSHOT.394)
Inventory Module (mod-inventory-18.1.0-SNAPSHOT.493)
Inventory Storage Module (mod-inventory-storage-23.0.0-SNAPSHOT.657)
Invoice business logic module (mod-invoice-5.3.0-SNAPSHOT.284)
Initial configuration:
kcp1-mod-data-import: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-data-import.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MaxRAMPercentage=66.0 -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=384m -Xmx1024m" (Hard/Soft limits: 2048/1024)
kcp1-mod-source-record-manager: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-source-record-manager.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MaxRAMPercentage=66.0 -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -Xmx1292m" (Hard/Soft limits: 2048/1844)
kcp1-mod-source-record-storage: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-source-record-storage.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MaxRAMPercentage=66.0 -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=384m -Xmx908m" (Hard/Soft limits: 1440/1296)
kcp1-mod-inventory: "-Dport=8082 -XX:MaxRAMPercentage=66.0 -Dorg.folio.metadata.inventory.storage.type=okapi -XX:MetaspaceSize=182m -XX:MaxMetaspaceSize=182m -Xmx1814m" (Hard/Soft limits: 2880/2592)
...
Following resources are used:
- 6 m4large EC2 spot instances for kubernetes cluster;
- 1 db.r5.xlarge instance for RDS service (writer)
- one m5.large per 2 zones for kafks on MSK
KIWI testing performance results:
Data Import Test Report (Kiwi)
Modules:
Data Import Module (mod-data-import-2.3.0-SNAPSHOT.224)
Source Record Manager Module (mod-source-record-manager-3.3.0-SNAPSHOT.556)
Source Record Storage Module (mod-source-record-storage-5.3.0-SNAPSHOT.394)
Inventory Module (mod-inventory-18.1.0-SNAPSHOT.493)
Inventory Storage Module (mod-inventory-storage-23.0.0-SNAPSHOT.657)
Invoice business logic module (mod-invoice-5.3.0-SNAPSHOT.284)
Initial configuration:
kcp1-mod-data-import: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-data-import.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MaxRAMPercentage=66.0 -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=384m -Xmx1024m" (Hard/Soft limits: 2048/1024)
kcp1-mod-source-record-manager: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-source-record-manager.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MaxRAMPercentage=66.0 -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -Xmx1292m" (Hard/Soft limits: 2048/1844)
kcp1-mod-source-record-storage: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-source-record-storage.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MaxRAMPercentage=66.0 -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=384m -Xmx908m" (Hard/Soft limits: 1440/1296)
kcp1-mod-inventory: "-Dport=8082 -XX:MaxRAMPercentage=66.0 -Dorg.folio.metadata.inventory.storage.type=okapi -XX:MetaspaceSize=182m -XX:MaxMetaspaceSize=182m -Xmx1814m" (Hard/Soft limits: 2880/2592)
kcp1-mod-inventory-storage: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-inventory-storage.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=232m -XX:MaxMetaspaceSize=232m -Xmx544m" (Hard/Soft limits: 846/778)
kcp1-mod-invoice: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-invoice.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=88m -Xmx360m" (Hard/Soft limits: 512/360)
Performance-optimized configuration:
MAX_REQUEST_SIZE = 4000000 (for all modules)
2 items for all items
2 partition for all DI kafka topics:
Examples:
delete old topic
# ./kafka-topics.sh --bootstrap-server=b-1.kafka-folijet.p1n5o2.c11.kafka.us-west-2.amazonaws.com:9092 --delete --topic perf-eks-folijet.Default.fs09000000.DI_ERROR
recreate topic with "--partitions 2 --replication-factor 1"
# ./kafka-topics.sh --bootstrap-server=b-1.kafka-folijet.p1n5o2.c11.kafka.us-west-2.amazonaws.com:9092 --create --topic perf-eks-folijet.Default.fs09000000.DI_ERROR --partitions 2 --replication-factor 1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic perf-eks-folijet.Default.fs09000000.DI_ERROR.
get topic info
# ./kafka-topics.sh --bootstrap-server=b-1.kafka-folijet.p1n5o2.c11.kafka.us-west-2.amazonaws.com:9092 --describe --topic perf-eks-folijet.Default.fs09000000.DI_ERROR
Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR PartitionCount: 2 ReplicationFactor: 1 Configs: min.insync.replicas=1,message.format.version=2.6-IV0,unclean.leader.election.enable=true
Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR Partition: 1 Leader: 2 Replicas: 2 Isr: 2
mod-source-record-storage: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/mstmp/mod-source-inventoryrecord-storage.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh MetaspaceSize=384m -XX:MetaspaceSizeMaxMetaspaceSize=232m 384m -Xmx908m -XX:MaxMetaspaceSize=232m -Xmx544m" (+UseG1GC" (Memory Hard/Soft limits: 1440Mi/1296Mi, CPU Hard/Soft limits: 8461024m/778512m)
kcp1-mod-invoiceinventory: "-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-invoice.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh Dorg.folio.metadata.inventory.storage.type=okapi -XX:MetaspaceSize=88m 182m -XX:MaxMetaspaceSize=88m -Xmx360m182m -Xmx1814m -XX:+UseG1GC" (Memory Hard/Soft limits: 512/360)
Performance-optimized configuration:
MAX_REQUEST_SIZE = 4000000 (for all modules2880Mi/2592Mi, CPU Hard/Soft limits: 1024m/512m)
mod-source-recordinventory-storage: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mod-source-recordinventory-storage.hprof -XX:MetaspaceSize=384m 232m -XX:MaxMetaspaceSize=384m 232m -Xmx908m Xmx544m -XX:+UseG1GC" (Memory Hard/Soft limits: 1440m846Mi/1296m778Mi, CPU Hard/Soft limits: 1024m512m/512m)
Tests:
| env | profile | records number | time | kafka partition number | module instance number | cpu | decription |
|---|---|---|---|---|---|---|---|
Kiwi | default | 1025 | 8 min | 1 | 1 | ||
| Lotus Perf Rancher | default | 1025 | 23 min | 1 | 1 | 128 | inventory - CPU 200 |
| Lotus Perf Rancher | default | 1025 | 25 min | 1 | 2 | 128 | inventory - CPU 256 2 pods for every module, except data-import |
| Lotus Perf Rancher | default | 1025 | 11 min | 1 | 1 | 256 | all modules - CPU 256, except data-import |
| Lotus Perf Rancher | ptf-create2 | 1025 | 35 min | 1 | 1 | 128 | inventory - CPU 200 |
| Lotus Perf Rancher | ptf-create2 | 1025 | 15 min | 1 | 1 | 256 | all modules - CPU 256, except data-import 4 errors: "id value already exists in table holdings_record" |
| Lotus Perf Rancher | ptf-create2 | 1025 | 6 min | 1 | 1 | 512 | all modules - CPU 512, except data-import |
| Lotus Perf Rancher | ptf-create2 | 1025 | 4 min | 1 | 1 | srm, inventory 512/1024 512 | srm, inventory 512/1024, other modules 512, except data-import |
| Lotus Perf Rancher | ptf-create2 | 1025 | 4 min | 2 | 1 | srm, inventory 512/1024 512 | srm, inventory 512/1024, other modules 512, except data-import |
| Lotus Perf Rancher | ptf-create2 | 1025 | 3 min | 2 | 2 | srm, inventory 512/1024 512 | srm, inventory 512/1024, other modules 512, except data-import |
| Lotus Perf Rancher | PTF - Updates Success - 1 | 1025 | 4 min | 2 | 2 | srm, inventory 512/1024 512 | srm, inventory 512/1024, other modules 512, except data-import |
| Lotus Perf Rancher | ptf-create2 | 5000 | 56 min | 1 | 1 | 256 | all modules - CPU 256, except data-import |
| Lotus Perf Rancher | ptf-create2 | 5000 | 16 min | 1 | 1 | srm, inventory 512/1024 512 | srm, inventory 512/1024, other modules 512, except data-import |
| Lotus Perf Rancher | ptf-create2 | 5000 | 11 min | 2 | 2 | srm, inventory 512/1024 512 | srm, inventory 512/1024, other modules 512, except data-import |
| Lotus Perf Rancher | ptf-create2 | 5000 | 8 min | 2 | 2 | srm, srs, inventory 512/1024 512 | srm, srs, inventory 512/1024, other modules 512, except data-import |
| Lotus Perf Rancher | PTF - Updates Success - 1 | 5000 | 13 min | 2 | 2 | srm, srs, inventory 512/1024 512 | srm, srs, inventory 512/1024, other modules 512, except data-import |
| Lotus Perf Rancher | ptf-create2 | 10000 | 36 min | 1 | 1 | srm, inventory 512/1024 512 | srm, inventory 512/1024, other modules 512, except data-import |
| Lotus Perf Rancher | ptf-create2 | 10000 | 19 min | 2 | 2 | srm, srs, inventory 512/1024 512 | srm, inventory 512/1024, other modules 512, except data-import |
| Lotus Perf Rancher | PTF - Updates Success - 1 | 10000 | 2 | 2 | srm, srs, inventory 512/1024 512 | srm, inventory 512/1024, other modules 512, except data-import | |
| Lotus Perf Rancher | ptf-create2 | 30000 | 5h 34min | 1 | 1 | 256 | all modules - CPU 256, except data-import |
| Lotus Perf Rancher | ptf-create2 | 30000 | 45 min | 2 | 2 | srm, srs, inventory 512/1024 512 | srm, srs, inventory 512/1024, other modules 512, except data-import |
| Lotus Perf Rancher | ptf-create2 | 50000 | 1h 19min / 1h 25min | 2 | 2 | srm, srs, inventory 512/1024 512 | srm, srs, inventory 512/1024, other modules 512, except data-import |
| Lotus Perf Rancher | ptf-create2 | 100000 | 2h 24min | 2 | 2 | srm, srs, inventory 512/1024 512 | srm, srs, inventory 512/1024, other modules 512, except data-import 22 errors → 11 selected as ERROR without any error messages + 2 + 9
|
| Lotus Perf Rancher | ptf-create2 | 500000 | 12h 42min | 2 | 2 | srm, srs, inventory 512/1024 512 | srm, srs, inventory 512/1024, other modules 512, except data-import 61 error: "contributors[0].name" = "null - (7) Field 'title' is a required field and can not be null - (1) proxyClient failure: mod-inventory-storage-23.0.0-SNAPSHOT.657 http://mod-inventory-storage: Connection was closed - (53) |
...