Overview This document contains the comparison results of testing workflows Check-in/Check-out and Data Import for MARC Bibliographic records in the Quesnelia release with different instance types: kafka.m7g.2xlarge zookeeper mode against kafka.m7g.2xlarge KRaft mode.
Tickets: https://folio-org.atlassian.net/browse/PERF-921 VShttps://folio-org.atlassian.net/browse/PERF-936
Summary Test Runs
1
CICO + DI MARC Bib Create
8 users + 5K, 25K sequentially
2
DI MARC Bib Create
5K, 25K sequentially
3
CICO + DI MARC Bib Update
8 users + 5K, 25K sequentially
4
DI MARC Bib Update
5K, 25K sequentially
5
CICO + DI MARC Bib Create
8 users + 5K, 25K sequentially
6
DI MARC Bib Create
5K, 25K sequentially
7
CICO + DI MARC Bib Update
8 users + 5K, 25K sequentially
8
DI MARC Bib Update
5K, 25K sequentially
Test Results This table shows results of Check-In/Check-out and Data Import create and update jobs.
MSK instance: kafka.m7g.2xlarge, metadata mode - ZooKeeper
Job profile
File size
DI Duration without CI/CO
DI Duration with CI/CO
CI with DI Average sec
CO with DI Average sec
PTF - Create 2
5k
00:03:05
00:02:39
0.707
1.104
25k
00:12:03
00:12:08
0.718
1.129
PTF - Updates Success - 6
5k
00:03:36
00:03:34
0.742
1.124
25k
00:17:05
00:17:33
0.756
1.148
MSK instance: kafka.m7g.2xlarge, metadata mode - KRaft
Job profile
File size
DI Duration without CI/CO
DI Duration with CI/CO
CI with DI Average sec
CO with DI Average sec
PTF - Create 2
5k
00:02:49
00:02:39
0.765
1.118
25k
00:13:31
00:12:04
0.777
1.186
PTF - Updates Success - 6
5k
00:04:36
00:04:31
0.706
1.095
25k
00:24:07
00:21:50
0.74
1.16
Check-in/Check-out without DI
Response time, sec MSK instance: kafka.m7g.2xlarge Zookeeper mode
Response time, sec MSK instance: kafka.m7g.2xlarge
95 perc
average
95 perc
average
Circulation Check-in/Check-out (without Data import)
8 users
Check-in
0.72
0.606
0.695
0.583
Check-out
1.241
0.969
1.151
0.944
Comparison Data Import durations and Check-In/Check-Out response time comparison
PTF - Create 2
5k
25k
PTF - Updates Success - 6
5k
25k
Check-in/Check-out without DI
instance: kafka.m7g.2xlarge
instance: kafka.m7g.2xlarge
95 perc
average
95 perc
average
Average
Circulation Check-in/Check-out (without Data import)
8 users
Check-in
0.72
0.606
0.695
0.583
-0.023
Check-out
1.241
0.969
1.151
0.944
-0.025
MSK resource utilization (CPU)
MSK instance:
kafka.m7g.2xlarge
KRaft mode
MSK instance:
kafka.m7g.2xlarge
KRaft mode
CICO
1
10
9
-1
2
10
9
-1
CICO+DI
1
31
32
1
2
32
30
-2
Response time MSK instance: kafka.m7g.2xlarge Zookeeper mode
MSK instance: kafka.m7g.2xlarge Kraft mode
Service CPU Utilization CPU utilization table
MSK instance: kafka.m5.2xlarge vs MSK instance: kafka.m7g.2xlarge MSK instance: kafka.m7g.2xlarge Zookeeper mode MSK instance: kafka.m7g.2xlarge Kraft mode Module CPU (CICO + 25k Create) CPU (CICO + 25k Update) Module CPU (CICO + 25k Create) CPU (CICO + 25k Update) mod-inventory-b 115.21 136.94 mod-inventory-b 139.1 133.42 mod-quick-marc-b 95.15 96.4 mod-di-converter-storage-b 103.49 96.49 mod-di-converter-storage-b 81.26 100.43 mod-quick-marc-b 75.45 72.77 nginx-okapi 70.58 88.94 nginx-okapi 75.33 73.68 okapi-b 38.89 50.55 okapi-b 41.74 51.2 mod-source-record-storage-b 31.61 39.13 mod-source-record-storage-b 38.11 34.22 mod-users-b 23.6 22.12 mod-inventory-storage-b 23.13 26.1 mod-inventory-storage-b 21.37 19.9 mod-source-record-manager-b 17.16 16.33 mod-feesfines-b 18.28 9.11 mod-users-b 9.18 21.82 mod-configuration-b 17.6 10.52 mod-dcb-b 8.32 9.84 mod-source-record-manager-b 17.39 18.27 mod-search-b 7.19 8.53 mod-authtoken-b 17.04 13.37 mod-pubsub-b 4.32 5.66 mod-dcb-b 13.23 12.33 mod-configuration-b 3.24 10.32 mod-search-b 7.95 1.83 mod-oa-b 2.95 3.35 mod-pubsub-b 6.82 6.49 mod-patron-b 2.86 2.4 pub-okapi 3.56 3.64 mod-feesfines-b 2.51 9.08 mod-circulation-storage-b 3.35 2.7 mod-authtoken-b 2.17 12.71 mod-patron-b 2.72 2.79 mod-entities-links-b 2.15 1.81 mod-entities-links-b 2.24 2.23 mod-circulation-storage-b 2.01 2.9 mod-circulation-b 1.98 1.8 mod-data-import-b 1.6 1.58 mod-data-import-b 1.76 1.88 edge-patron-b 1.13 1.02 edge-patron-b 1.13 1.16 mod-users-bl-b 0.61 1.11 mod-patron-blocks-b 0.97 1 mod-circulation-b 0.55 2.09 mod-users-bl-b 0.68 0.68 mod-patron-blocks-b 0.41 0.95 pub-edge 0.06 0.06 pub-okapi 0.18 3.98 pub-edge 0.07 0.07 pub-edge 0.05 0.12
DI MARC BIB Create and Update + CICO
MSK instance: kafka.m7g.2xlarge Zookeeper mode
MSK instance: kafka.m7g.2xlarge Kraft mode
Service Memory Utilization MSK instance: kafka.m5.2xlarge vs MSK instance: kafka.m7g.2xlarge Service memory consumption for module deltas higher or close to 10%: In KRaft mode services utilized less memory in mod-inventory-b - 11%, mod-source-record-storage-b - 9%, and more in mod-data-import-b - 14% Module Memory MSK instance: kafka.m7g.2xlarge Zookeeper mode Memory MSK instance: kafka.m7g.2xlarge Kraft mode Delta mod-dcb-b 74.37 74.77 0.4 mod-inventory-b 70.81 59.63 -11.18 mod-users-b 50.37 53.1 2.73 mod-di-converter-storage-b 46.7 49.67 2.97 mod-feesfines-b 45.44 44.03 -1.41 mod-inventory-storage-b 33.4 30.49 -2.91 mod-source-record-storage-b 55.53 46.38 -9.15 okapi-b 42.5 49.68 7.18 mod-data-import-b 43.55 57.77 14.22 mod-patron-blocks-b 42.38 42.7 0.32 mod-search-b 45.58 48 2.42 mod-users-bl-b 45.36 45.82 0.46 mod-configuration-b 38.68 39.73 1.05 mod-source-record-manager-b 41.91 38.7 -3.21 mod-pubsub-b 35.94 36.32 0.38 mod-quick-marc-b 42.6 36.99 -5.61 mod-patron-b 30.52 30.19 -0.33 mod-entities-links-b 34.49 30.56 -3.93 mod-authtoken-b 27.32 27.42 0.1 mod-circulation-b 25.15 25.01 -0.14 edge-patron-b 22.38 23.16 0.78 mod-circulation-storage-b 22.34 28.98 6.64 nginx-okapi 4.58 4.69 0.11 pub-okapi 4.46 4.46 0 pub-edge 4.41 4.35 -0.06 pub-edge 4.35 4.35 0
MSK instance: kafka.m7g.2xlarge Zookeeper mode
MSK instance: kafka.m7g.2xlarge Kraft mode
DB CPU Utilization Average DB CPU utilization is 90% during both create and update jobs tests with different MSK instance types. DB CPU utilized 15% during Check-In/Check-Out period without DI.
MSK instance: kafka.m7g.2xlarge Zookeeper mode MSK instance: kafka.m7g.2xlarge Kraft mode
DB Connections Average connection count is about 900 connections for create and 860 connections for update jobs with CI/CO. 770 connections for CI/CO without data import for tests with different MSK instance types.
MSK instance: kafka.m7g.2xlarge Zookeeper mode MSK instance: kafka.m7g.2xlarge Kraft mode MSK instance resource utilization MSK resources table MSK resource utilization (CPU) No difference between two MSK clusters Load scenario Brokers MSK instance: kafka.m7g.2xlarge Zookeeper mode MSK instance: kafka.m7g.2xlarge Kraft mode Delta, % CICO 1 10 9 -1 2 10 9 -1 CICO+DI 1 31 32 1 2 32 30 -2 MSK resource utilization (DIsk) was growing gradually during tests with kafka.m5.2xlarge to 10%
Disk usage by broker MSK instance: kafka.m7g.2xlarge Zookeeper mode MSK instance: kafka.m7g.2xlarge Kraft mode CPU (User) usage by broker MSK instance: kafka.m7g.2xlarge Zookeeper mode MSK instance: kafka.m7g.2xlarge Kraft mode DB load MSK instance: kafka.m7g.2xlarge Zookeeper mode
Top SQL-queries:
MSK instance: kafka.m7g.2xlarge Kraft mode
Top SQL-queries:
Appendix
Infrastructure PTF -environment qcp1
MSK ptf-KRaft-mode
2 m7g.2xlarge brokers in 2 zones (total 2 brokers)
Apache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
revision - 26
metadata mode - KRaft
Total topics: 1474
Total partitions: 11909
Task count for module mod-graphql set to 0 before test start.
Modules
All qcp1 modules Module Task Definition Revision Module Version Task Count Mem Hard Limit Mem Soft Limit CPU Units Xmx Metaspace Size Max Metaspace Size mod-remote-storage 5 mod-remote-storage:3.2.0 2 4920 4472 1024 3960 512 512 mod-ncip 5 mod-ncip:1.14.4 2 1024 896 128 768 88 128 mod-finance-storage 5 mod-finance-storage:8.6.0 2 1024 896 1024 700 88 128 mod-agreements 5 mod-agreements:7.0.0 2 1592 1488 128 0 0 0 mod-ebsconet 5 mod-ebsconet:2.2.0 2 1248 1024 128 700 128 256 mod-organizations 5 mod-organizations:1.9.0 2 1024 896 128 768 88 128 mod-consortia 3 mod-consortia:1.1.0 2 5136 4776 1024 4416 384 512 edge-sip2 3 edge-sip2:3.2.0-SNAPSHOT.209 2 1024 896 128 768 88 128 mod-serials-management 5 mod-serials-management:1.0.0 2 2480 2312 128 1792 384 512 mod-settings 5 mod-settings:1.0.3 2 1024 896 200 768 88 128 mod-data-import 8 mod-data-import:3.1.0 1 2048 1844 256 1292 384 512 edge-dematic 5 edge-dematic:2.2.0 1 1024 896 128 768 88 128 mod-search 5 mod-search:3.2.0 2 2592 2480 2048 1440 512 1024 mod-inn-reach 3 mod-inn-reach:3.2.0-SNAPSHOT.86 2 3600 3240 1024 2880 512 1024 mod-tags 5 mod-tags:2.2.0 2 1024 896 128 768 88 128 edge-courses 5 edge-courses:1.4.0 2 1024 896 128 768 88 128 mod-authtoken 6 mod-authtoken:2.15.1 2 1440 1152 512 922 88 128 mod-inventory-update 5 mod-inventory-update:3.3.0 2 1024 896 128 768 88 128 mod-notify 5 mod-notify:3.2.0 2 1024 896 128 768 88 128 mod-configuration 5 mod-configuration:5.10.0 2 1024 896 128 768 88 128 mod-orders-storage 5 mod-orders-storage:13.7.0 2 1024 896 512 700 88 128 edge-caiasoft 5 edge-caiasoft:2.2.0 2 1024 896 128 768 88 128 mod-login-saml 5 mod-login-saml:2.8.0 2 1024 896 128 768 88 128 mod-erm-usage-harvester 5 mod-erm-usage-harvester:4.5.0 2 1024 896 128 768 88 128 mod-licenses 5 mod-licenses:6.0.0 2 2480 2312 512 1792 384 512 mod-gobi 5 mod-gobi:2.8.0 2 1024 896 128 768 88 128 mod-password-validator 5 mod-password-validator:3.2.0 2 1440 1298 128 768 384 512 mod-bulk-operations 5 mod-bulk-operations:2.0.0 2 3072 2600 1024 1536 384 512 mod-fqm-manager 5 mod-fqm-manager:2.0.1 2 1024 896 128 768 88 128 edge-dcb 5 edge-dcb:1.1.0 2 1024 896 128 768 88 128 mod-graphql 6 mod-graphql:1.12.1 2 1024 896 128 768 88 128 mod-finance 5 mod-finance:4.9.0 2 1024 896 128 768 88 128 mod-erm-usage 5 mod-erm-usage:4.7.0 2 2800 2550 1024 1800 384 512 mod-batch-print 6 mod-batch-print:1.1.0 2 1024 896 128 768 88 128 mod-copycat 5 mod-copycat:1.6.0 2 1024 896 128 768 88 128 mod-lists 5 mod-lists:2.0.0 2 1024 896 128 768 88 128 mod-entities-links 6 mod-entities-links:3.0.0 2 2592 2480 400 1440 0 1024 mod-permissions 10 mod-permissions:6.5.0 2 1684 1544 512 1024 384 512 pub-edge 3 pub-edge:2023.06.14 2 1024 896 128 768 0 0 mod-orders 5 mod-orders:12.8.0 2 2048 1740 1024 1024 384 512 edge-patron 5 edge-patron:5.1.0 2 1024 896 256 768 88 128 edge-ncip 5 edge-ncip:1.9.2 2 1024 896 128 768 88 128 edge-inn-reach 3 edge-inn-reach:3.1.1-SNAPSHOT.45 2 1024 896 128 768 88 128 mod-users-bl 5 mod-users-bl:7.7.0 2 1440 1152 512 922 88 128 mod-oa 4 mod-oa:2.1.0-SNAPSHOT.62 2 1024 896 128 768 88 256 mod-inventory-storage 5 mod-inventory-storage:27.1.0 2 4096 3690 2048 3076 384 512 mod-invoice 6 mod-invoice:5.8.0 2 1440 1152 512 922 88 128 mod-user-import 5 mod-user-import:3.8.0 2 1024 896 128 768 88 128 mod-sender 6 mod-sender:1.12.0 2 1024 896 128 768 88 128 edge-oai-pmh 5 edge-oai-pmh:2.9.0 2 1512 1360 1024 1440 384 512 mod-data-export-worker 5 mod-data-export-worker:3.2.1 2 3072 2800 1024 2048 384 512 mod-rtac 5 mod-rtac:3.6.0 2 1024 896 128 768 88 128 mod-circulation-storage 5 mod-circulation-storage:17.2.0 2 2880 2592 1536 1814 384 512 mod-calendar 5 mod-calendar:3.1.0 2 1024 896 128 768 88 128 mod-source-record-storage 7 mod-source-record-storage:5.8.0 2 5600 5000 2048 3500 384 512 mod-event-config 5 mod-event-config:2.7.0 2 1024 896 128 768 88 128 mod-courses 5 mod-courses:1.4.10 2 1024 896 128 768 88 128 mod-circulation-item 5 mod-circulation-item:1.0.0 2 1024 896 128 768 88 128 mod-inventory 6 mod-inventory:20.2.0 2 4096 3688 1024 1814 384 512 mod-email 5 mod-email:1.17.0 2 2800 2550 512 1800 384 512 mod-pubsub 5 mod-pubsub:2.13.0 2 1536 1440 1024 922 384 512 mod-circulation 5 mod-circulation:24.2.0 2 2880 2592 1536 1814 384 512 mod-di-converter-storage 5 mod-di-converter-storage:2.2.0 2 1024 896 128 768 88 128 edge-rtac 5 edge-rtac:2.7.1 2 1024 896 128 768 88 128 edge-orders 5 edge-orders:3.0.0 2 1024 896 128 768 88 128 mod-users 6 mod-users:19.3.1 2 1024 896 128 768 88 128 mod-template-engine 5 mod-template-engine:1.20.0 2 1024 896 128 768 88 128 mod-patron-blocks 5 mod-patron-blocks:1.10.0 2 1024 896 1024 768 88 128 mod-audit 5 mod-audit:2.9.0 2 1024 896 128 768 88 128 edge-fqm 5 edge-fqm:2.0.0 2 1024 896 128 768 88 128 mod-source-record-manager 6 mod-source-record-manager:3.9.0-SNAPSHOT.330 2 5600 5000 2048 3500 384 512 nginx-edge 3 nginx-edge:2023.06.14 2 1024 896 128 0 0 0 mod-quick-marc 5 mod-quick-marc:5.1.0 1 2288 2176 128 1664 384 512 nginx-okapi 3 nginx-okapi:2023.06.14 2 1024 896 128 0 0 0 okapi-b 5 okapi:5.3.0 3 1684 1440 1024 922 384 512 mod-feesfines 5 mod-feesfines:19.1.0 2 1024 896 128 768 88 128 mod-invoice-storage 5 mod-invoice-storage:5.8.0 2 1872 1536 1024 1024 384 512 mod-dcb 6 mod-dcb:1.1.0 2 1024 896 128 768 88 128 mod-service-interaction 5 mod-service-interaction:4.0.1 2 2048 1844 256 1290 384 512 mod-data-export 17 mod-data-export:5.0.4 1 2048 1844 2048 0 0 0 mod-patron 5 mod-patron:6.1.0 2 1024 896 128 768 88 128 mod-oai-pmh 5 mod-oai-pmh:3.13.0 2 4096 3690 2048 3076 384 512 edge-connexion 5 edge-connexion:1.2.0 2 1024 896 128 768 88 128 mod-kb-ebsco-java 5 mod-kb-ebsco-java:4.0.0 2 1024 896 128 768 88 128 mod-notes 5 mod-notes:5.2.0 2 1024 896 128 952 384 512 mod-data-export-spring 5 mod-data-export-spring:3.2.0 1 2048 1844 256 1536 384 512 mod-organizations-storage 5 mod-organizations-storage:4.7.0 2 1024 896 128 768 88 128 mod-login 5 mod-login:7.11.0 2 1440 1298 1024 768 384 512 pub-okapi 3 pub-okapi:2023.06.14 2 1024 896 128 768 0 0 mod-eusage-reports 5 mod-eusage-reports:2.1.1 2 1024 896 128 768 88 128
Methodology/Approach