Overview

This document contains the results of testing workflows Check-out (Check-in had a functional issues at the moment) and Data Import for MARC Bibliographic records with PTF- Create-3 job profile in the Quesnelia release with MSK KRaft mode. Test were conducted on qcpt environment. The main idea is to see how number of topics and number of messages affects resource usage of MSK cluster and the main KPI for the testing flows. Tests should be performed with changed kafka broker number (2, 4, 6, 8) to achieve necessary load - 100.000 partitions in sync replicas.

Ticket: https://folio-org.atlassian.net/browse/PERF-939

Summary

• Defining capacity for MSK instance type kafka.m7g.xlarge changing number of topics and messages

  • Changing the number of brokers for the same number of topics and synced messages we observe best performance with 6 brokers - stable data imports and check-out average response time 1.1 seconds which is not longer than in baseline test results with mcpt cluster.

  • Test with additional load using the script generating messages on topics which are not involved in data import and check-out do not affect performance of these flows.

  • Test with 6 brokers and 200.000 synced messages failed. Kafka was not stable with more than 90% of CPU.

  • Test with 8 brokers and 200.000 synced messages was successful. So the cluster can handle even this doubled load.

  • mod-pubsub-b module play important role during testing and affect CPU utilization of MSK brokers.

• Resource utilization

  • 1'st test kafka.m7g.xlarge, 2 brokers, total topics: 6690, total partitions: 48062

    • Module CPU utilization: mod-pubsub-b - 82%, mod-inventory-b - 38%, mod-quick-marc-b - 35%, mod-dcb-b - 27%, okapi-b - 19%, mod-data-import-b - 17%

    • Module Memory: mod-inventory-b - 93%, mod-dcb-b - 88%, mod-permissions-b - 82%, mod-data-import-b - 63%, mod-quick-marc-b - 62%, mod-source-record-storage-b - 61%,  okapi-b- 60%, mod-search-b - 58%

  • 5'th test kafka.m7g.xlarge, 4 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748

    • Module CPU utilization: mod-inventory-b - 56%, mod-quick-marc-b - 38%, mod-dcb-b - 26%, mod-pubsub-b - 23%, mod-di-converter-storage-b - 16%, okapi-b -16%, mod-configuration-b - 15%, mod-users-b - 13%, mod-search-b - 12%

    • Module Memory: 

  • 6'th test kafka.m7g.xlarge, 6 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748

    • Module CPU utilization: spike of mod-data-import - 189%, mod-inventory - 86%, mod-quick-marc - 55%

    • Module Memory: mod-dcb-b - 104%, mod-inventory-b - 88%, mod-quick-marc-b - 62%, okapi-b - 60%, mod-search-b - 60%, mod-source-record-manager-b - 59%

Recommendations & Jiras

• Module mod-pubsub-b was not stable and stopped containers due to out of memory. Allocating more resources could resolve the problem. The last configuration for this module:  "cpu": 0, "memoryReservation": 2048 /-XX:MaxMetaspaceSize=512m -Xmx2500m

Test Runs 

Test Results

• 1'st test kafka.m7g.xlarge, 2 brokers, total topics: 6690, total partitions: 48062 - qcpt pointed to MSK cluster

  • MSK CPU utilization- 85%

  • MSK Disk usage - 0.5%

  • Check-out flow - 1.6 seconds without data import, 1.4 sec with data import

  • Data import Create job profile with 10k file on 1 tenant - 46 minutes completed successfully

  • Data import on 15 tenants concurrently - last longer than 4 hours and completed with errors (No actions reason)

• 2'nd test kafka.m7g.xlarge, 6 brokers, total topics: 7335, total partitions: 96426 - qcpt + qcon pointed to MSK cluster

  • MSK CPU utilization - 93% - not stable

  • Check-out flow - no possibility to start testing

  • DI - no possibility to start testing

• 3'rd test kafka.m7g.xlarge, 8 brokers, total topics: 7335, total partitions: 96426 - qcpt + qcon pointed to MSK cluster

  • MSK CPU utilization - 67% - 1,2 brokers, 45% - 3,4,5,6,7,8 brokers

  • Check-out flow - 1.1 seconds

  • Data import 1 single record 15 tenants duration - 55 seconds

• 4'th test (retest of test #3 after brokers reboot) kafka.m7g.xlarge, 8 brokers, total topics: 7335, total partitions: 96426 - qcpt + qcon pointed to MSK cluster

  • MSK CPU utilization - 90% - 1,2 brokers, 70% - 3,4,5,6,7,8 brokers - was unstable for an hour, after stabilization tests completed successfully

  • Check-out flow - 1.6 seconds

  • Data import 1 single record 15 tenants duration - 44 seconds

  • Data import Create job profile with 10k file on 1 tenant - 16 minutes completed successfully

• 5'th test kafka.m7g.xlarge, 4 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748 - qcpt pointed to MSK cluster

  • MSK CPU utilization - 79% 1,2 brokers, 62% - 9, 10 brokers

  • MSK CPU utilization with Data import 10k - 89% all brokers

  • Check-out flow -  1.1 seconds

  • Data import 1 single record 15 tenants duration - 1 minutes 20 seconds

  • Data import Create job profile with 10k file on 1 tenant - 10 minutes

  • Data import Create job profile with 10k file on 15 tenants- 2 hours 25 minutes

• 6'th test kafka.m7g.xlarge, 6 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748 - qcpt pointed to MSK cluster

  • MSK CPU utilization with DI single record - 70% 1,2 brokers, 60% - 9, 10, 11, 12 brokers

  • MSK CPU utilization with Data import 10k - 85% 

  • Check-out flow - 1,2 seconds

  • Data import 1 single record 15 tenants duration - 1 minutes 40 seconds

  • Data import Create job profile with 10k file on 1 tenant - 6 minutes 40 seconds

  • Data import Create job profile with 10k file on 15 tenants - 1 hour 30 minutes, completed successfully on all tenants

• 7'th test kafka.m7g.xlarge, 6 brokers, total topics: 6743, total partitions: online - 55872 In sync replicas - 111748 - qcpt pointed to MSK cluster. Retest #6 with additional load by message creation script (./folio-topics-load-messages.sh  NUM_RECORDS=100000).

  • mod-pubsub-b module play important role during testing and affect CPU utilization of MSK brokers.

  • MSK CPU utilization with DI single record - 52% 1,2 brokers, 37% - 9, 10, 11, 12 brokers

  • MSK CPU utilization with Data import 10k - 70% 

  • Check-out flow -  1.1 seconds

  • Data import 1 single record 15 tenants duration - 1 minutes 10 seconds

  • Data import Create job profile with 10k file on 1 tenant - 6 min 20 seconds

Response time