Overview

According to PERF-534 It's been observed that DI's performance was improved greatly when DI Kafka topics' partitions were increased to 2.

In this testing effort, we'd like to see if increasing mod-pubsub's Kafka topics partitions from 1 to two would have the same positive impact on the Check In Check Out workflow as many mod-pubsub's topics are related to circulation.  We will test CICO with R/W split enabled and disabled as well.

Summary

• It doesn’t looks like changing of mod-pub-sub partitions to 2 helping much. 

• For some tests response times were faster, and for some tests were slower (-20ms + 50ms).

• There is no big benefits when enabling Read/Write split on CICO in these standalone tests. However they may appear during real life usage and/or running several high DB load workflows (such as Data Import), as it will distribute load between DB nodes.

Recommendations

The only notable observation is that mod-pub-sub Kafka topics has naming pattern include mod-pub-sub version like: 

• ncp5.pub-sub.fs09000000.FEE_FINE_BALANCE_CHANGED.mod-pubsub-2.7.0

• ncp5.pub-sub.fs09000000.FEE_FINE_BALANCE_CHANGED.mod-pubsub-2.9.1

• ncp5.pub-sub.fs09000000.FEE_FINE_BALANCE_CHANGED.mod-pubsub-2.10.0-SNAPSHOT

*which is same topic for different mod-pubsub versions. 

If mod-pubsub gets updated frequently, then the old topics might still hang around and will accumulate unnecessarily. So possibly it's a good idea to exclude the version number from topic naming pattern.

Test Sets 

Results

Below listed response times (average (avg.) 75 percentile and 95 percentile) for tests (8,20,30,75 users) with 1 and 2 mod-pub-sub Kafka topic partitions.

Also there is comparison provided between 2 and 1 partitions. Number ⁺ⁿ mean that particular response time is slower by ⁿ ms comparing with appropriate number from 1 partition test. 

With Read Write split enabled

*Here we can see that there is no significant difference in response times between 1 and 2 partitions of mod-pub-sub kafka topics when R/W split is enabled. For some cases it's better and for some it's worse so we can conclude that it has no pattern and having 2 partitions have no benefits in response times. 

Read Write split disabled 

*Here we can see that there is no significant difference in response times between 1 and 2 partitions of mod-pub-sub kafka topics when R/W split is disabled. For some cases it's better and for some it's worse so we can conclude that it has no pattern and having 2 partitions have no benefits in response times. 

Comparisons

Comparison between RW/Split enabled/disabled with 1 and 2 partitions

Table below shows how many milliseconds will we save or miss if we'll enable Read/Write split on DB. (R/W split disabled response times are baseline numbers for comparisons)

Notable observations:

• As shown here - there is a big difference in CPU usage pattern with and without R/W split. For now I doesn't looks like it helping much, and in most cases it makes performance worse. However there possibly will be performance benefits during real life usage and/or running several high DB load workflows (such as Data Import), as it will distribute load between DB nodes.

• For now - no visible pattern to conclude if R/W split is working better with 1 mod-pub-sub partition or with 2.

Comparison between current result vs initial results

Initial results was made by measuring CICO performance on snapshot version of modules

As a base for current result we'll use CICO results with Read/Write split disabled and with one mod-pub-sub Kafka partition.(As it was also input conditions for initial testing).