Table of Contents

Overview

This document contains the results of testing workflows Check-out (Check-in was unavailable to test because of had a functional issues at the moment) and Data Import for MARC Bibliographic records with PTF- Create-3 job profile in the Quesnelia release with MSK KRaft mode instance type. Test were conducted on qcpt environment. The main idea is to see how number of topics and number of messages affects resource usage of MSK cluster and the main KPI for the testing flows. Tests should be performed with changed kafka broker size number (2, 4, 6, 8) to achieve necessary load - 100.000 partitions in sync replicas.

Ticket:

...

• Defining capacity for MSK instance type kafka.m7g.xlarge changing number of topics and messages
  • Changing the number of brokers for the same number of topics and synced messages we observe best performance with 6 brokers - stable data imports and check-out average response time 1.1 seconds which is not longer than in baseline test results with mcpt cluster.
  • Test with additional load using the script generating messages on topics which are not involved in data import and check-out do not affect performance of these flows.
  • Test with 6 brokers and 200.000 synced messages failed. It Kafka was not stable with more than 90% of CPU.
  • Test with 8 brokers and 200.000 synced messages was successful. So the cluster can handle even this doubled load.
  • mod-pubsub-b module play important role during testing and affect CPU utilization of MSK brokers.
• Resource utilization
  • 1'st test kafka.m7g.xlarge, 2 brokers, total topics: 6690, total partitions: 48062

    • Module CPU utilization: mod-pubsub-b - 82%, mod-inventory-b - 38%, mod-quick-marc-b - 35%, mod-dcb-b - 27%, okapi-b - 19%, mod-data-import-b - 17%

    • Module Memory: mod-inventory-b - 93%, mod-dcb-b - 88%, mod-permissions-b - 82%, mod-data-import-b - 63%, mod-quick-marc-b - 62%, mod-source-record-storage-b - 61%,  okapi-b- 60%, mod-search-b - 58%

  • 5'th test kafka.m7g.xlarge, 4 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748

    • Module CPU utilization: mod-inventory-b - 56%, mod-quick-marc-b - 38%, mod-dcb-b - 26%, mod-pubsub-b - 23%, mod-di-converter-storage-b - 16%, okapi-b -16%, mod-configuration-b - 15%, mod-users-b - 13%, mod-search-b - 12%

    • Module Memory: 
  • 6'th test kafka.m7g.xlarge, 6 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748
    • Module CPU utilization: spike of mod-data-import - 189%, mod-inventory - 86%, mod-quick-marc - 55%

    • Module Memory: mod-dcb-b - 104%, mod-inventory-b - 88%, mod-quick-marc-b - 62%, okapi-b - 60%, mod-search-b - 60%, mod-source-record-manager-b - 59%

Recommendations & Jiras

• Module mod-pubsub-b was not stable and stopped containers due to out of memory. Allocating more resources could resolve the problem. The last configuration for this module:  "cpu":0, "memoryReservation":2048 /-XX:MaxMetaspaceSize=512m -Xmx2500m

...

• 1'st test kafka.m7g.xlarge, 2 brokers, total topics: 6690, total partitions: 48062 - qcpt pointed to MSK cluster
  • MSK CPU utilization- 85%
  • MSK Disk usage - 0.5%
  • Check-out flow - 1.6 seconds without data import, 1.4 sec with data import
  • Data import Create job profile with 10k file on 1 tenant - 46 minutes completed successfully
  • Data import on 15 tenants concurrently - last longer than 4 hours and completed with errors (No actions reason)
• 2'nd test kafka.m7g.xlarge, 6 brokers, total topics: 7335, total partitions: 96426 - qcpt + qcon pointed to MSK cluster
  • MSK CPU utilization - 93% - not stable
  • Check-out flow - no possibility to start testing
  • DI - no possibility to start testing
• 3'rd test kafka.m7g.xlarge, 8 brokers, total topics: 7335, total partitions: 96426 - qcpt + qcon pointed to MSK cluster
  • MSK CPU utilization - 67% - 1,2 brokers, 45% - 3,4,5,6,7,8 brokers
  • Check-out flow - 1.1 seconds
  • Data import 1 single record 15 tenants duration - 55 seconds
• 4'th test (retest of test #3 after brokers reboot) kafka.m7g.xlarge, 8 brokers, total topics: 7335, total partitions: 96426 - qcpt + qcon pointed to MSK cluster
  • MSK CPU utilization - 90% - 1,2 brokers, 70% - 3,4,5,6,7,8 brokers - was unstable for an hour, after stabilization tests completed successfully
  • Check-out flow - 1.6 seconds
  • Data import 1 single record 15 tenants duration - 44 seconds
  • Data import Create job profile with 10k file on 1 tenant - 16 minutes completed successfully
• 5'th test kafka.m7g.xlarge, 4 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748 - qcpt pointed to MSK cluster
  • MSK CPU utilization - 79% 1,2 brokers, 62% - 9, 10 brokers
  • MSK CPU utilization with Data import 10k - 89% all brokers
  • Check-out flow -  1.1 seconds
  • Data import 1 single record 15 tenants duration - 1 minutes 20 seconds
  • Data import Create job profile with 10k file on 1 tenant - 10 minutes
  • Data import Create job profile with 10k file on 15 tenants- 2 hours 25 minutes
• 6'th test kafka.m7g.xlarge, 6 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748 - qcpt pointed to MSK cluster
  • MSK CPU utilization with DI single record - 70% 1,2 brokers, 60% - 9, 10, 11, 12 brokers
  • MSK CPU utilization with Data import 10k - 85% 
  • Check-out flow - 1,2 seconds
  • Data import 1 single record 15 tenants duration - 1 minutes 40 seconds
  • Data import Create job profile with 10k file on 1 tenant - 6 minutes 40 seconds
  • Data import Create job profile with 10k file on 15 tenants - 1 hour 30 minutes, completed successfully on all tenants
• 7'th test kafka.m7g.xlarge, 6 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748 - qcpt pointed to MSK cluster. Retest #6 with additional load by message creation script (./folio-topics-load-messages.sh  NUM_RECORDS=100000).
  • mod-pubsub-b module play important role during testing and affect CPU utilization of MSK brokers.
  • MSK CPU utilization with DI single record - 52% 1,2 brokers, 37% - 9, 10, 11, 12 brokers
  • MSK CPU utilization with Data import 10k - 70% 
  • Check-out flow -  1.1 seconds
  • Data import 1 single record 15 tenants duration - 1 minutes 10 seconds
  • Data import Create job profile with 10k file on 1 tenant - 6 min 20 seconds

Response time

...

• 10k on 1 tenant: mod-quick-marc - 134%, mod-inventory - 95%, mod-di-converter-storage - 57%

• 10k on 15 tenants: mod-data-import - 189%, mod-inventory - 86%, mod-quick-marc - 55%

...

PTF -environment qcpt

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.4xlarge	GiB	vCPUs	-

  
  

• MSK ptf-KRaft-mode2
  • multiple configurations of m7g.xlarge brokers in 2 zones (1, 2, 3, 4 per zone)

  • Apache Kafka version 3.7.x, mode - KRaft

  • Cluster configuration name: fse-kafka-config revision 26

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=2 (the last test with 6 brokers - replication factor was 3)

Task count for modules mod-agreements set to 0 before test start.

Modules

...

• • log.retention.minutes=480
  • default.replication.factor=2 (the last test with 6 brokers - replication factor was 3)
• Open Search ptf-test
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Version: OpenSearch_2_7_R20240502
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3

Task count for modules mod-agreements set to 0 before test start.

Modules

Methodology/Approach

• Populate ptf-KRaft-mode2 cluster with topics from tenant cluster for qcpt
• Create the list of unconsolidated topics for all modules that are involved into data import and check-in/check-out

  • 
    

    Expand
    title The list of modules to unconsolidate
    1. mod-search 2. mod-orders-storage 3. mod-inn-reach 4. mod-data-export-worker 5. mod-data-import 6. mod-bulk-operations 7. mod-pubsub 8. mod-dcb (doesn't exist yet) 9. mod-quick-marc 10. mod-inventory 11. mod-invoice 12. mod-search-ebsco 13. mod-consortia 14. mod-circulation-storage 15. mod-remote-storage 16. mod-source-record-manager 17. mod-data-export-spring 18. mod-audit 19. mod-source-record-storage 20. mod-linked-data 21. mod-entities-links 22. mod-circulation 23. mod-settings 24. mod-orders 25. mod-inventory-storage 26. mod-users For completeness sake, and for the future, here are the Eureka Kafka modules: 1. mod-users-keycloak 2. mgr-tenant-entitlements 3. mgr-applications 4. mod-consortia-keycloak

    
    

• Run smoke CICO for 20 minutes and single data import on random tenant involved in testing
• Run 1 hour CICO and 10k DI create job for 15 tenants concurrently
• Adjust the sh script to generate additional messages in topics which not related to tested flows
• Repeat testing
• Change brokers number (2, 4, 6, 8) - to troubleshoot possible issues reboot brokers after broker size changing
• Compare resource utilization of MSK and main KPI for CICO & DI

Additional/Files

Scripts may be found on S3 bucket: fse-ptf/capacity_testing/qcpt

...