Overview

This document contains the results of testing workflows Check-out (Check-in had a functional issues at the moment) and Data Import for MARC Bibliographic records with PTF- Create-3 job profile in the Quesnelia release with MSK KRaft mode. Test were conducted on qcpt environment. The main idea is to see how number of topics and number of messages affects resource usage of MSK cluster and the main KPI for the testing flows. Tests should be performed with changed kafka broker number (2, 4, 6, 8) to achieve necessary load - 100.000 partitions in sync replicas.

Ticket: PERF-939 - Getting issue details... STATUS

Summary

• Defining capacity for MSK instance type kafka.m7g.xlarge changing number of topics and messages
  • Changing the number of brokers for the same number of topics and synced messages we observe best performance with 6 brokers - stable data imports and check-out average response time 1.1 seconds which is not longer than in baseline test results with mcpt cluster.
  • Test with additional load using the script generating messages on topics which are not involved in data import and check-out do not affect performance of these flows.
  • Test with 6 brokers and 200.000 synced messages failed. Kafka was not stable with more than 90% of CPU.
  • Test with 8 brokers and 200.000 synced messages was successful. So the cluster can handle even this doubled load.
  • mod-pubsub-b module play important role during testing and affect CPU utilization of MSK brokers.
• Resource utilization
  • 1'st test kafka.m7g.xlarge, 2 brokers, total topics: 6690, total partitions: 48062

    • Module CPU utilization: mod-pubsub-b - 82%, mod-inventory-b - 38%, mod-quick-marc-b - 35%, mod-dcb-b - 27%, okapi-b - 19%, mod-data-import-b - 17%

    • Module Memory: mod-inventory-b - 93%, mod-dcb-b - 88%, mod-permissions-b - 82%, mod-data-import-b - 63%, mod-quick-marc-b - 62%, mod-source-record-storage-b - 61%,  okapi-b- 60%, mod-search-b - 58%

  • 5'th test kafka.m7g.xlarge, 4 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748

    • Module CPU utilization: mod-inventory-b - 56%, mod-quick-marc-b - 38%, mod-dcb-b - 26%, mod-pubsub-b - 23%, mod-di-converter-storage-b - 16%, okapi-b -16%, mod-configuration-b - 15%, mod-users-b - 13%, mod-search-b - 12%

    • Module Memory: 
  • 6'th test kafka.m7g.xlarge, 6 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748
    • Module CPU utilization: spike of mod-data-import - 189%, mod-inventory - 86%, mod-quick-marc - 55%

    • Module Memory: mod-dcb-b - 104%, mod-inventory-b - 88%, mod-quick-marc-b - 62%, okapi-b - 60%, mod-search-b - 60%, mod-source-record-manager-b - 59%

Recommendations & Jiras

• Module mod-pubsub-b was not stable and stopped containers due to out of memory. Allocating more resources could resolve the problem. The last configuration for this module:  "cpu": 0, "memoryReservation": 2048 /-XX:MaxMetaspaceSize=512m -Xmx2500m

Test Runs 

Test Results

• 1'st test kafka.m7g.xlarge, 2 brokers, total topics: 6690, total partitions: 48062 - qcpt pointed to MSK cluster
  • MSK CPU utilization- 85%
  • MSK Disk usage - 0.5%
  • Check-out flow - 1.6 seconds without data import, 1.4 sec with data import
  • Data import Create job profile with 10k file on 1 tenant - 46 minutes completed successfully
  • Data import on 15 tenants concurrently - last longer than 4 hours and completed with errors (No actions reason)
• 2'nd test kafka.m7g.xlarge, 6 brokers, total topics: 7335, total partitions: 96426 - qcpt + qcon pointed to MSK cluster
  • MSK CPU utilization - 93% - not stable
  • Check-out flow - no possibility to start testing
  • DI - no possibility to start testing
• 3'rd test kafka.m7g.xlarge, 8 brokers, total topics: 7335, total partitions: 96426 - qcpt + qcon pointed to MSK cluster
  • MSK CPU utilization - 67% - 1,2 brokers, 45% - 3,4,5,6,7,8 brokers
  • Check-out flow - 1.1 seconds
  • Data import 1 single record 15 tenants duration - 55 seconds
• 4'th test (retest of test #3 after brokers reboot) kafka.m7g.xlarge, 8 brokers, total topics: 7335, total partitions: 96426 - qcpt + qcon pointed to MSK cluster
  • MSK CPU utilization - 90% - 1,2 brokers, 70% - 3,4,5,6,7,8 brokers - was unstable for an hour, after stabilization tests completed successfully
  • Check-out flow - 1.6 seconds
  • Data import 1 single record 15 tenants duration - 44 seconds
  • Data import Create job profile with 10k file on 1 tenant - 16 minutes completed successfully
• 5'th test kafka.m7g.xlarge, 4 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748 - qcpt pointed to MSK cluster
  • MSK CPU utilization - 79% 1,2 brokers, 62% - 9, 10 brokers
  • MSK CPU utilization with Data import 10k - 89% all brokers
  • Check-out flow -  1.1 seconds
  • Data import 1 single record 15 tenants duration - 1 minutes 20 seconds
  • Data import Create job profile with 10k file on 1 tenant - 10 minutes
  • Data import Create job profile with 10k file on 15 tenants- 2 hours 25 minutes
• 6'th test kafka.m7g.xlarge, 6 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748 - qcpt pointed to MSK cluster
  • MSK CPU utilization with DI single record - 70% 1,2 brokers, 60% - 9, 10, 11, 12 brokers
  • MSK CPU utilization with Data import 10k - 85% 
  • Check-out flow - 1,2 seconds
  • Data import 1 single record 15 tenants duration - 1 minutes 40 seconds
  • Data import Create job profile with 10k file on 1 tenant - 6 minutes 40 seconds
  • Data import Create job profile with 10k file on 15 tenants - 1 hour 30 minutes, completed successfully on all tenants
• 7'th test kafka.m7g.xlarge, 6 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748 - qcpt pointed to MSK cluster. Retest #6 with additional load by message creation script (./folio-topics-load-messages.sh  NUM_RECORDS=100000).
  • mod-pubsub-b module play important role during testing and affect CPU utilization of MSK brokers.
  • MSK CPU utilization with DI single record - 52% 1,2 brokers, 37% - 9, 10, 11, 12 brokers
  • MSK CPU utilization with Data import 10k - 70% 
  • Check-out flow -  1.1 seconds
  • Data import 1 single record 15 tenants duration - 1 minutes 10 seconds
  • Data import Create job profile with 10k file on 1 tenant - 6 min 20 seconds

Response time

Service CPU Utilization

DI MARC BIB Create and Update + CICO

5'th test (4 brokers, ~50k online partitions)

6'th test (6 brokers, ~50k online partitions)

Spikes:

• 10k on 1 tenant: mod-quick-marc - 134%, mod-inventory - 95%, mod-di-converter-storage - 57%

• 10k on 15 tenants: mod-data-import - 189%, mod-inventory - 86%, mod-quick-marc - 55%

Service Memory Utilization

5'th test (4 brokers, ~50k online partitions)

6'th test (6 brokers, ~50k online partitions)

DB CPU Utilization

5'th test (4 brokers, ~50k online partitions)

6'th test (6 brokers, ~50k online partitions)

DB Connections

5'th test (4 brokers, ~50k online partitions)

6'th test (6 brokers, ~50k online partitions)

6'th test (6 brokers, ~50k online partitions)

PTF -environment qcpt

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.4xlarge	GiB	vCPUs	-

• MSK ptf-KRaft-mode2
  • multiple configurations of m7g.xlarge brokers in 2 zones (1, 2, 3, 4 per zone)

  • Apache Kafka version 3.7.x, mode - KRaft

  • Cluster configuration name: fse-kafka-config revision 26

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=2 (the last test with 6 brokers - replication factor was 3)
• Open Search ptf-test
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Version: OpenSearch_2_7_R20240502
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3

Task count for modules mod-agreements set to 0 before test start.

Modules

Methodology/Approach

• Populate ptf-KRaft-mode2 cluster with topics from tenant cluster for qcpt
• Create the list of unconsolidated topics for all modules that are involved into data import and check-in/check-out
  •  The list of modules to unconsolidate

    1. mod-search
    2. mod-orders-storage
    3. mod-inn-reach
    4. mod-data-export-worker
    5. mod-data-import
    6. mod-bulk-operations
    7. mod-pubsub
    8. mod-dcb (doesn't exist yet)
    9. mod-quick-marc
    10. mod-inventory
    11. mod-invoice
    12. mod-search-ebsco
    13. mod-consortia
    14. mod-circulation-storage
    15. mod-remote-storage
    16. mod-source-record-manager
    17. mod-data-export-spring
    18. mod-audit
    19. mod-source-record-storage
    20. mod-linked-data
    21. mod-entities-links
    22. mod-circulation
    23. mod-settings
    24. mod-orders
    25. mod-inventory-storage
    26. mod-users
    For completeness sake, and for the future, here are the Eureka Kafka modules:
    1. mod-users-keycloak
    2. mgr-tenant-entitlements
    3. mgr-applications
    4. mod-consortia-keycloak

• Run smoke CICO for 20 minutes and single data import on random tenant involved in testing
• Run 1 hour CICO and 10k DI create job for 15 tenants concurrently
• Adjust the sh script to generate additional messages in topics which not related to tested flows
• Repeat testing
• Change brokers number (2, 4, 6, 8) - to troubleshoot possible issues reboot brokers after broker size changing
• Compare resource utilization of MSK and main KPI for CICO & DI

Additional/Files

Scripts may be found on S3 bucket: fse-ptf/capacity_testing/qcpt

Topics: