Kafka Zookeeper Mode vs KRaft Mode MSK - instance type comparison

Kafka Zookeeper Mode vs KRaft Mode MSK - instance type comparison

Overview

This document contains the comparison results of testing workflows Check-in/Check-out and Data Import for MARC Bibliographic records in the Quesnelia release with different instance types: kafka.m7g.2xlarge zookeeper mode against kafka.m7g.2xlarge KRaft mode.

Tickets: https://folio-org.atlassian.net/browse/PERF-921VShttps://folio-org.atlassian.net/browse/PERF-936

Summary

  • Comparing kafka.m7g.2xlarge zookeeper metadata mode against kafka.m7g.2xlarge KRaft metadata mode

    • No difference in MSK resource utilization (CPU and Disk)  comparing two MSK clusters

    • In KRaft mode services utilized less memory in mod-inventory-b on 11%, mod-source-record-storage-b on 9%, and more memory in mod-data-import-b on 14%. For the rest modules the difference was less than 7%

    • No significant difference in service CPU utilization:

      • Service CPU utilization in Zookeeper mode: mod-inventory-b - 136%, mod-quick-marc-b - 96%, mod-di-converter-storage-b - 100%, nginx-okapi - 88% the rest of modules utilized less than 50%

      • Service CPU utilization in KRaft mode: mod-inventory-b - 133%, mod-quick-marc-b - 75%, mod-di-converter-storage-b - 103%, nginx-okapi - 75% the rest of modules utilized less than 50%

    • No difference in Check-in/Check-out response time

    • Data import durations fluctuate from test to test but work stable without issues

Test Runs 

Test #

MSK instance type

Scenario

Load level

Test #

MSK instance type

Scenario

Load level

1

kafka.m7g.2xlarge

zookeeper mode

CICO + DI MARC Bib Create

 8 users + 5K, 25K sequentially

2

DI MARC Bib Create

5K, 25K sequentially

3

CICO + DI MARC Bib Update

 8 users + 5K, 25K sequentially

4

DI MARC Bib Update

5K, 25K sequentially

5

kafka.m7g.2xlarge

KRaft mode

CICO + DI MARC Bib Create

 8 users + 5K, 25K sequentially

6

DI MARC Bib Create

5K, 25K sequentially

7

CICO + DI MARC Bib Update

 8 users + 5K, 25K sequentially

8

DI MARC Bib Update

5K, 25K sequentially

Test Results

This table shows results of Check-In/Check-out and Data Import create and update jobs.

MSK instance: kafka.m7g.2xlarge, metadata mode - ZooKeeper

Job profile

File size

DI Duration without CI/CO

DI Duration with CI/CO

CI with DI Average sec

CO with DI Average sec

PTF - Create 2

5k

00:03:05

00:02:39

0.707

1.104

 

25k

00:12:03

00:12:08

0.718

1.129

PTF - Updates Success - 6

5k

00:03:36

00:03:34

0.742

1.124

 

25k

00:17:05

00:17:33

0.756

1.148

MSK instance: kafka.m7g.2xlarge, metadata mode - KRaft

Job profile

File size

DI Duration without CI/CO

DI Duration with CI/CO

CI with DI Average sec

CO with DI Average sec

PTF - Create 2

5k

00:02:49

00:02:39

0.765

1.118

 

25k

00:13:31

00:12:04

0.777

1.186

PTF - Updates Success - 6

5k

00:04:36

00:04:31

0.706

1.095

 

25k

00:24:07

00:21:50

0.74

1.16

Check-in/Check-out without DI

Scenario

Load level

Request

Response time, sec
MSK instance: kafka.m7g.2xlarge Zookeeper mode

Response time, sec
MSK instance: kafka.m7g.2xlarge

KRaft mode

95 perc

average

95 perc

average

Circulation Check-in/Check-out (without Data import)

8 users

Check-in

0.72

0.606

0.695

0.583

Check-out

1.241

0.969

1.151

0.944

Comparison

Data Import durations and Check-In/Check-Out response time comparison

  • Response times of CI/CO with Data import do not differ in both MSK clusters

Job Profile

File size

DELTA, DI without CI/CO

DELTA, DI+CI/CO

DELTA, CI with DI

DELTA, CO with DI

PTF - Create 2

5k

00:00:16

00:00:00

-0.058

-0.014

25k

-00:01:28

00:00:04

-0.059

-0.057

PTF - Updates Success - 6

5k

-00:01:00

-00:00:57

0.036

0.029

25k

-00:07:02

-00:04:17

0.016

-0.012

Check-in/Check-out without DI

  • Check-in/Check-out perform the same in both MSK clusters. The difference of response times is so small that it can be neglected.

Scenario

Load level

Request

Response time, sec
MSK

instance: kafka.m7g.2xlarge

Zookeeper mode

Response time, sec
MSK

instance: kafka.m7g.2xlarge 

KRaft mode

 

Delta

95 perc

average

95 perc

average

Average

Circulation Check-in/Check-out (without Data import)

8 users

Check-in

0.72

0.606

0.695

0.583

-0.023

Check-out

1.241

0.969

1.151

0.944

-0.025

MSK resource utilization (CPU)

Load scenario

Brokers

MSK instance: 

kafka.m7g.2xlarge

Zookeeper mode

MSK instance: 

kafka.m7g.2xlarge 

KRaft mode

Delta, %

Load scenario

Brokers

MSK instance: 

kafka.m7g.2xlarge

Zookeeper mode

MSK instance: 

kafka.m7g.2xlarge 

KRaft mode

Delta, %

CICO

1

10

9

-1

2

10

9

-1

CICO+DI

1

31

32

1

2

32

30

-2

Response time

MSK instance: kafka.m7g.2xlarge Zookeeper mode

MSK instance: kafka.m7g.2xlarge Kraft mode

Service CPU Utilization

CPU utilization table

MSK instance: kafka.m7g.2xlarge Zookeeper mode MSK instance: kafka.m7g.2xlarge Kraft mode Module CPU (CICO + 25k Create) CPU (CICO + 25k Update) Module CPU (CICO + 25k Create) CPU (CICO + 25k Update) mod-inventory-b 115.21 136.94 mod-inventory-b 139.1 133.42 mod-quick-marc-b 95.15 96.4 mod-di-converter-storage-b 103.49 96.49 mod-di-converter-storage-b 81.26 100.43 mod-quick-marc-b 75.45 72.77 nginx-okapi 70.58 88.94 nginx-okapi 75.33 73.68 okapi-b 38.89 50.55 okapi-b 41.74 51.2 mod-source-record-storage-b 31.61 39.13 mod-source-record-storage-b 38.11 34.22 mod-users-b 23.6 22.12 mod-inventory-storage-b 23.13 26.1 mod-inventory-storage-b 21.37 19.9 mod-source-record-manager-b 17.16 16.33 mod-feesfines-b 18.28 9.11 mod-users-b 9.18 21.82 mod-configuration-b 17.6 10.52 mod-dcb-b 8.32 9.84 mod-source-record-manager-b 17.39 18.27 mod-search-b 7.19 8.53 mod-authtoken-b 17.04 13.37 mod-pubsub-b 4.32 5.66 mod-dcb-b 13.23 12.33 mod-configuration-b 3.24 10.32 mod-search-b 7.95 1.83 mod-oa-b 2.95 3.35 mod-pubsub-b 6.82 6.49 mod-patron-b 2.86 2.4 pub-okapi 3.56 3.64 mod-feesfines-b 2.51 9.08 mod-circulation-storage-b 3.35 2.7 mod-authtoken-b 2.17 12.71 mod-patron-b 2.72 2.79 mod-entities-links-b 2.15 1.81 mod-entities-links-b 2.24 2.23 mod-circulation-storage-b 2.01 2.9 mod-circulation-b 1.98 1.8 mod-data-import-b 1.6 1.58 mod-data-import-b 1.76 1.88 edge-patron-b 1.13 1.02 edge-patron-b 1.13 1.16 mod-users-bl-b 0.61 1.11 mod-patron-blocks-b 0.97 1 mod-circulation-b 0.55 2.09 mod-users-bl-b 0.68 0.68 mod-patron-blocks-b 0.41 0.95 pub-edge 0.06 0.06 pub-okapi 0.18 3.98 pub-edge 0.07 0.07 pub-edge 0.05 0.12

DI MARC BIB Create and Update + CICO

MSK instance: kafka.m7g.2xlarge Zookeeper mode

MSK instance: kafka.m7g.2xlarge Kraft mode

Service Memory Utilization

Service memory consumption for module deltas higher or close to 10%: In KRaft mode services utilized less memory in mod-inventory-b - 11%, mod-source-record-storage-b - 9%, and more in mod-data-import-b - 14% Module Memory MSK instance: kafka.m7g.2xlarge Zookeeper mode Memory MSK instance: kafka.m7g.2xlarge Kraft mode Delta mod-dcb-b 74.37 74.77 0.4 mod-inventory-b 70.81 59.63 -11.18 mod-users-b 50.37 53.1 2.73 mod-di-converter-storage-b 46.7 49.67 2.97 mod-feesfines-b 45.44 44.03 -1.41 mod-inventory-storage-b 33.4 30.49 -2.91 mod-source-record-storage-b 55.53 46.38 -9.15 okapi-b 42.5 49.68 7.18 mod-data-import-b 43.55 57.77 14.22 mod-patron-blocks-b 42.38 42.7 0.32 mod-search-b 45.58 48 2.42 mod-users-bl-b 45.36 45.82 0.46 mod-configuration-b 38.68 39.73 1.05 mod-source-record-manager-b 41.91 38.7 -3.21 mod-pubsub-b 35.94 36.32 0.38 mod-quick-marc-b 42.6 36.99 -5.61 mod-patron-b 30.52 30.19 -0.33 mod-entities-links-b 34.49 30.56 -3.93 mod-authtoken-b 27.32 27.42 0.1 mod-circulation-b 25.15 25.01 -0.14 edge-patron-b 22.38 23.16 0.78 mod-circulation-storage-b 22.34 28.98 6.64 nginx-okapi 4.58 4.69 0.11 pub-okapi 4.46 4.46 0 pub-edge 4.41 4.35 -0.06 pub-edge 4.35 4.35 0

MSK instance: kafka.m7g.2xlarge Zookeeper mode

MSK instance: kafka.m7g.2xlarge Kraft mode

DB CPU Utilization

Average DB CPU utilization is 90% during both create and update jobs tests with different MSK instance types. DB CPU utilized 15% during Check-In/Check-Out period without DI.

MSK instance: kafka.m7g.2xlarge Zookeeper mode

MSK instance: kafka.m7g.2xlarge Kraft mode

 

DB Connections

Average connection count is about 900 connections for create and 860 connections for update jobs with CI/CO. 770 connections for CI/CO without data import for tests with different MSK instance types.

MSK instance: kafka.m7g.2xlarge Zookeeper mode

MSK instance: kafka.m7g.2xlarge Kraft mode

MSK instance resource utilization

MSK resource utilization (CPU) No difference between two MSK clusters Load scenario Brokers MSK instance: kafka.m7g.2xlarge Zookeeper mode MSK instance: kafka.m7g.2xlarge Kraft mode Delta, % CICO 1 10 9 -1 2 10 9 -1 CICO+DI 1 31 32 1 2 32 30 -2 MSK resource utilization (DIsk) was growing gradually during tests with kafka.m5.2xlarge to 10%

Disk usage by broker

MSK instance: kafka.m7g.2xlarge Zookeeper mode

MSK instance: kafka.m7g.2xlarge Kraft mode

CPU (User) usage by broker

MSK instance: kafka.m7g.2xlarge Zookeeper mode

MSK instance: kafka.m7g.2xlarge Kraft mode

DB load

MSK instance: kafka.m7g.2xlarge Zookeeper mode

 

Top SQL-queries:

 

MSK instance: kafka.m7g.2xlarge Kraft mode

 

Top SQL-queries:

 

 

Appendix

Infrastructure

PTF -environment qcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

    Name

    Memory GIB

    vCPUs

    max_connections

    db.r6g.xlarge

    32 GiB

    4 vCPUs

    2731

  • MSK perf-921-g2

    • m7g.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.2.tiered

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=2

 

  • MSK ptf-KRaft-mode

    • m7g.2xlarge brokers in 2 zones (total 2 brokers)

    • Apache Kafka version 3.7.x

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

    • revision - 26

    • metadata mode - KRaft

    • Total topics: 1474

    • Total partitions: 11909

Task count for module mod-graphql set to 0 before test start.

Modules

Module Task Definition Revision Module Version Task Count Mem Hard Limit Mem Soft Limit CPU Units Xmx Metaspace Size Max Metaspace Size mod-remote-storage 5 mod-remote-storage:3.2.0 2 4920 4472 1024 3960 512 512 mod-ncip 5 mod-ncip:1.14.4 2 1024 896 128 768 88 128 mod-finance-storage 5 mod-finance-storage:8.6.0 2 1024 896 1024 700 88 128 mod-agreements 5 mod-agreements:7.0.0 2 1592 1488 128 0 0 0 mod-ebsconet 5 mod-ebsconet:2.2.0 2 1248 1024 128 700 128 256 mod-organizations 5 mod-organizations:1.9.0 2 1024 896 128 768 88 128 mod-consortia 3 mod-consortia:1.1.0 2 5136 4776 1024 4416 384 512 edge-sip2 3 edge-sip2:3.2.0-SNAPSHOT.209 2 1024 896 128 768 88 128 mod-serials-management 5 mod-serials-management:1.0.0 2 2480 2312 128 1792 384 512 mod-settings 5 mod-settings:1.0.3 2 1024 896 200 768 88 128 mod-data-import 8 mod-data-import:3.1.0 1 2048 1844 256 1292 384 512 edge-dematic 5 edge-dematic:2.2.0 1 1024 896 128 768 88 128 mod-search 5 mod-search:3.2.0 2 2592 2480 2048 1440 512 1024 mod-inn-reach 3 mod-inn-reach:3.2.0-SNAPSHOT.86 2 3600 3240 1024 2880 512 1024 mod-tags 5 mod-tags:2.2.0 2 1024 896 128 768 88 128 edge-courses 5 edge-courses:1.4.0 2 1024 896 128 768 88 128 mod-authtoken 6 mod-authtoken:2.15.1 2 1440 1152 512 922 88 128 mod-inventory-update 5 mod-inventory-update:3.3.0 2 1024 896 128 768 88 128 mod-notify 5 mod-notify:3.2.0 2 1024 896 128 768 88 128 mod-configuration 5 mod-configuration:5.10.0 2 1024 896 128 768 88 128 mod-orders-storage 5 mod-orders-storage:13.7.0 2 1024 896 512 700 88 128 edge-caiasoft 5 edge-caiasoft:2.2.0 2 1024 896 128 768 88 128 mod-login-saml 5 mod-login-saml:2.8.0 2 1024 896 128 768 88 128 mod-erm-usage-harvester 5 mod-erm-usage-harvester:4.5.0 2 1024 896 128 768 88 128 mod-licenses 5 mod-licenses:6.0.0 2 2480 2312 512 1792 384 512 mod-gobi 5 mod-gobi:2.8.0 2 1024 896 128 768 88 128 mod-password-validator 5 mod-password-validator:3.2.0 2 1440 1298 128 768 384 512 mod-bulk-operations 5 mod-bulk-operations:2.0.0 2 3072 2600 1024 1536 384 512 mod-fqm-manager 5 mod-fqm-manager:2.0.1 2 1024 896 128 768 88 128 edge-dcb 5 edge-dcb:1.1.0 2 1024 896 128 768 88 128 mod-graphql 6 mod-graphql:1.12.1 2 1024 896 128 768 88 128 mod-finance 5 mod-finance:4.9.0 2 1024 896 128 768 88 128 mod-erm-usage 5 mod-erm-usage:4.7.0 2 2800 2550 1024 1800 384 512 mod-batch-print 6 mod-batch-print:1.1.0 2 1024 896 128 768 88 128 mod-copycat 5 mod-copycat:1.6.0 2 1024 896 128 768 88 128 mod-lists 5 mod-lists:2.0.0 2 1024 896 128 768 88 128 mod-entities-links 6 mod-entities-links:3.0.0 2 2592 2480 400 1440 0 1024 mod-permissions 10 mod-permissions:6.5.0 2 1684 1544 512 1024 384 512 pub-edge 3 pub-edge:2023.06.14 2 1024 896 128 768 0 0 mod-orders 5 mod-orders:12.8.0 2 2048 1740 1024 1024 384 512 edge-patron 5 edge-patron:5.1.0 2 1024 896 256 768 88 128 edge-ncip 5 edge-ncip:1.9.2 2 1024 896 128 768 88 128 edge-inn-reach 3 edge-inn-reach:3.1.1-SNAPSHOT.45 2 1024 896 128 768 88 128 mod-users-bl 5 mod-users-bl:7.7.0 2 1440 1152 512 922 88 128 mod-oa 4 mod-oa:2.1.0-SNAPSHOT.62 2 1024 896 128 768 88 256 mod-inventory-storage 5 mod-inventory-storage:27.1.0 2 4096 3690 2048 3076 384 512 mod-invoice 6 mod-invoice:5.8.0 2 1440 1152 512 922 88 128 mod-user-import 5 mod-user-import:3.8.0 2 1024 896 128 768 88 128 mod-sender 6 mod-sender:1.12.0 2 1024 896 128 768 88 128 edge-oai-pmh 5 edge-oai-pmh:2.9.0 2 1512 1360 1024 1440 384 512 mod-data-export-worker 5 mod-data-export-worker:3.2.1 2 3072 2800 1024 2048 384 512 mod-rtac 5 mod-rtac:3.6.0 2 1024 896 128 768 88 128 mod-circulation-storage 5 mod-circulation-storage:17.2.0 2 2880 2592 1536 1814 384 512 mod-calendar 5 mod-calendar:3.1.0 2 1024 896 128 768 88 128 mod-source-record-storage 7 mod-source-record-storage:5.8.0 2 5600 5000 2048 3500 384 512 mod-event-config 5 mod-event-config:2.7.0 2 1024 896 128 768 88 128 mod-courses 5 mod-courses:1.4.10 2 1024 896 128 768 88 128 mod-circulation-item 5 mod-circulation-item:1.0.0 2 1024 896 128 768 88 128 mod-inventory 6 mod-inventory:20.2.0 2 4096 3688 1024 1814 384 512 mod-email 5 mod-email:1.17.0 2 2800 2550 512 1800 384 512 mod-pubsub 5 mod-pubsub:2.13.0 2 1536 1440 1024 922 384 512 mod-circulation 5 mod-circulation:24.2.0 2 2880 2592 1536 1814 384 512 mod-di-converter-storage 5 mod-di-converter-storage:2.2.0 2 1024 896 128 768 88 128 edge-rtac 5 edge-rtac:2.7.1 2 1024 896 128 768 88 128 edge-orders 5 edge-orders:3.0.0 2 1024 896 128 768 88 128 mod-users 6 mod-users:19.3.1 2 1024 896 128 768 88 128 mod-template-engine 5 mod-template-engine:1.20.0 2 1024 896 128 768 88 128 mod-patron-blocks 5 mod-patron-blocks:1.10.0 2 1024 896 1024 768 88 128 mod-audit 5 mod-audit:2.9.0 2 1024 896 128 768 88 128 edge-fqm 5 edge-fqm:2.0.0 2 1024 896 128 768 88 128 mod-source-record-manager 6 mod-source-record-manager:3.9.0-SNAPSHOT.330 2 5600 5000 2048 3500 384 512 nginx-edge 3 nginx-edge:2023.06.14 2 1024 896 128 0 0 0 mod-quick-marc 5 mod-quick-marc:5.1.0 1 2288 2176 128 1664 384 512 nginx-okapi 3 nginx-okapi:2023.06.14 2 1024 896 128 0 0 0 okapi-b 5 okapi:5.3.0 3 1684 1440 1024 922 384 512 mod-feesfines 5 mod-feesfines:19.1.0 2 1024 896 128 768 88 128 mod-invoice-storage 5 mod-invoice-storage:5.8.0 2 1872 1536 1024 1024 384 512 mod-dcb 6 mod-dcb:1.1.0 2 1024 896 128 768 88 128 mod-service-interaction 5 mod-service-interaction:4.0.1 2 2048 1844 256 1290 384 512 mod-data-export 17 mod-data-export:5.0.4 1 2048 1844 2048 0 0 0 mod-patron 5 mod-patron:6.1.0 2 1024 896 128 768 88 128 mod-oai-pmh 5 mod-oai-pmh:3.13.0 2 4096 3690 2048 3076 384 512 edge-connexion 5 edge-connexion:1.2.0 2 1024 896 128 768 88 128 mod-kb-ebsco-java 5 mod-kb-ebsco-java:4.0.0 2 1024 896 128 768 88 128 mod-notes 5 mod-notes:5.2.0 2 1024 896 128 952 384 512 mod-data-export-spring 5 mod-data-export-spring:3.2.0 1 2048 1844 256 1536 384 512 mod-organizations-storage 5 mod-organizations-storage:4.7.0 2 1024 896 128 768 88 128 mod-login 5 mod-login:7.11.0 2 1440 1298 1024 768 384 512 pub-okapi 3 pub-okapi:2023.06.14 2 1024 896 128 768 0 0 mod-eusage-reports 5 mod-eusage-reports:2.1.1 2 1024 896 128 768 88 128

Methodology/Approach

  • Compare two tests results