Versions Compared

Old Version 4

changes.mady.by.user Leonid Kolesnykov

Saved on

compared with

New Version 5

changes.mady.by.user Leonid Kolesnykov

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
Overview In Progress

...

  • Comparing kafka.m5.2xlarge, zookeeper metadata mode against kafka.m7g.2xlarge, KRaft metadata mode
    • Tests with KRaft mode enabled utilize less resources (5% at least, for some broker - 13%) CPU by brokers during CI/CO and during DI + CI/CO and the same time it is more balanced compared to zookeeper mode
    • Main workflows KPIs do not degrade in tests with Check-in/Check-out with Data import
  • Resource utilization

...

MSK instance: kafka.m5.2xlarge, metadata mode - ZooKeeper
Job profileFile sizeDI Duration without CI/CODI Duration with CI/COCI with DI Average secCO with DI Average sec
PTF - Create 25k00:03:4500:02:440.7361.16

25k00:14:4000:13:360.7871.176
PTF - Updates Success - 65k00:04:4300:04:180.7641.153

25k00:20:2100:21:250.7671.179
MSK instance: kafka.m7g.2xlarge, metadata mode - KRaft
Job profileFile sizeDI Duration without CI/CODI Duration with CI/COCI with DI Average secCO with DI Average sec
PTF - Create 25k00:02:4900:02:390.7651.118

25k00:13:3100:12:040.7771.186
PTF - Updates Success - 65k00:04:3600:04:310.7061.095

25k00:24:0700:21:500.741.16

...

MSK instance: kafka.m7g.2xlarge


Service CPU Utilization

Delta for CPU utilization shows in mod-di-converter-storage-b 20% decrease for update job and 10% decrease for mod-feesfines-b module. The most part of modules CPU utilization deltas fluctuate under 10%. CPU utilization table

Expand
titleMSK instance: kafka.m5.2xlarge vs MSK instance: kafka.m7g.2xlarge

DI MARC BIB Create and Update + CICO

MSK instance: kafka.m5.2xlarge

MSK instance: kafka.m7g.2xlarge

Service Memory Utilization

MSK instance: kafka.m5.2xlarge vs MSK instance: kafka.m7g.2xlarge
Expand
title


MSK instance: kafka.m5.2xlarge
MSK instance: kafka.m7g.2xlarge
ModuleCPU (CICO + 25k Create)CPU (CICO + 25k Update)
ModuleCPU (CICO + 25k Create)CPU (CICO + 25k Update)
mod-inventory-b107.84147.17
mod-inventory-b139.1133.42
mod-quick-marc-b79.9881.45
mod-di-converter-storage-b103.4996.49
mod-di-converter-storage-b75.12116.01
mod-quick-marc-b75.4572.77
nginx-okapi51.779.05
nginx-okapi75.3373.68
okapi-b27.6543.17
okapi-b41.7451.2
mod-source-record-storage-b24.8239.79
mod-source-record-storage-b38.1134.22
mod-inventory-storage-b18.5220.75
mod-inventory-storage-b23.1326.1
mod-source-record-manager-b16.9418.05
mod-source-record-manager-b17.1616.33
mod-dcb-b8.037.8
mod-users-b9.1821.82
mod-search-b7.871.44
mod-dcb-b8.329.84
mod-pubsub-b7.327.3
mod-search-b7.198.53
mod-users-b6.326.23
mod-pubsub-b4.325.66
mod-entities-links-b3.862.27
mod-configuration-b3.2410.32
mod-configuration-b3.613.4
mod-oa-b2.953.35
mod-patron-b2.862.66
mod-patron-b2.862.4
mod-authtoken-b2.862.13
mod-feesfines-b2.519.08
mod-oa-b2.82.86
mod-authtoken-b2.1712.71
mod-feesfines-b2.32.15
mod-entities-links-b2.151.81
mod-circulation-storage-b2.012.15
mod-circulation-storage-b2.012.9
mod-data-import-b1.61.72
mod-data-import-b1.61.58
edge-patron-b1.081.08
edge-patron-b1.131.02
mod-users-bl-b0.530.52
mod-users-bl-b0.611.11
mod-patron-blocks-b0.470.43
mod-circulation-b0.552.09
mod-circulation-b0.350.37
mod-patron-blocks-b0.410.95
pub-okapi0.140.15
pub-okapi0.183.98
pub-edge0.070.07
pub-edge0.050.12


DI MARC BIB Create and Update + CICO

MSK instance: kafka.m5.2xlarge

Image Added

MSK instance: kafka.m7g.2xlarge

Image Added


Service Memory Utilization

Expand
titleMSK instance: kafka.m5.2xlarge vs MSK instance: kafka.m7g.2xlarge
  • The comparison of memory resource utilization revealed no difference between tests
ModuleMemory (kafka.m5.2xlarge)Memory (kafka.m7g.2xlarge)Delta
mod-oa-b80.7--
mod-dcb-b74.6174.770.16
mod-inventory-b59.6159.630.02
mod-data-import-b57.7557.770.02
mod-users-b53.2253.1-0.12
okapi-b49.6549.680.03
mod-di-converter-storage-b49.6149.670.06
mod-search-b47.97480.03
mod-source-record-storage-b46.3846.380
mod-users-bl-b45.8745.82-0.05
mod-feesfines-b44.244.03-0.17
mod-patron-blocks-b42.9142.7-0.21
mod-configuration-b39.7639.73-0.03
mod-source-record-manager-b38.7138.7-0.01
mod-quick-marc-b36.9536.990.04
mod-pubsub-b36.1936.320.13
mod-entities-links-b30.5630.560
mod-inventory-storage-b30.5130.49-0.02
mod-patron-b30.1930.190
mod-circulation-storage-b28.9828.980
mod-authtoken-b27.3827.420.04
mod-circulation-b2525.010.01
edge-patron-b23.1623.160
nginx-okapi4.694.690
pub-okapi4.464.460
pub-edge4.354.350


MSK instance: kafka.m5.2xlarge

Image Added

MSK instance: kafka.m7g.2xlarge

Image Added

DB CPU Utilization

...

Appendix

Infrastructure

PTF -environment qcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

    NameMemory GIBvCPUsmax_connections

    db.r6g.xlarge

    		32 GiB4 vCPUs2731


  • MSK ptf-mobius-testing2
    • 2 m5.2xlarge brokers in 2 zones (total 2 brokers)

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=2
    • revision - 2
    • metadata mode - ZooKeeper
  • MSK ptf-KRaft-mode
    • 2 m7g.2xlarge brokers in 2 zones (total 2 brokers)

    • Apache Kafka version 3.7.x

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3
    • revision - 26
    • metadata mode - KRaft

...