Overview

This is an effort to retest MARC Authorities data import in the Lotus release on the lcp1 ECS cluster to compare the results of the Morning Glory release tests that were performed on the lcp2 ECS cluster.  The retests were done with comparable infrastructure (on the Morning Glory release) so that we could have an apples to apples comparison. 

Summary

1. Morning Glory's DI MARC Authorities performance is about the same as Lotus'
2. mod-source-record-manager's memory setting has a small impact on performance.  Doubling the memory allocation in some cases improves the speed of import.
3. DI performance improves when the number of partitions equals the number of topic consumers.
4. Having lots more partitions (50) without an equal number of topic consumers does not improve performance

Bugfest-like Settings

Tests were done with Morning-Glory's Bugfest-like setting (in order to do apples to apples comparison with Morning Glory's)

Kafka

All DI topics had 1 partition and all inventory topics (item | holdings-record | instance) also had 1 partition each.

Tests 1 Results (On lcp1)

Morning Glory (with lcp2's) Settings

Tests were done with PTF Morning-Glory environment's settings for apples to apples comparison.  Here only mod-srm's memory deviates from Bugfest's, and it is 50% less than Bugfest's.

Kafka

All DI topics had 1 partition and all inventory topics (item | holdings-record | instance) also had 1 partition each.

Tests 2 Results (On lcp2)

Tests 3 Results 

After making all DI topics having 2 partitions each.

By default all inventory topics (inventory.holdings-record, inventory.item, inventory.instance) have 50 partitions each after first time enabling the modules.  However, after Kafka automatically recreates the topics (if they had been deleted), these topics are defaulted to 1 partition, and such was the case in the PTF Kafka cluster.  The following tests were done after making all inventory topics have 50 partitions again.

Results of tests 5 and 6 were unexpected because the response time gets faster. We hypothesized that having 50 partitions would not improve performance and/or instead makes the performance worse because there isn't the same number of consumer available (only having 2 mod-inventory-storage tasks) to take advantage of the 50 "lanes" concurrency.  

Running Tests 7 after setting the inventory topics back to 1 partition to see if performance improve, and it does.

Here we picked Test 4 to compare against Test 7 and did not choose Tests 5 or 6 simply because we don't think they are consistent with our hypothesis. However, this needs more testing going forward to see if higher number of partitions helps or hurts performance.  Here we can tentatively conclude the folowing:

Tests 8: Lotus vs Morning Glory - comparison with different numbers of Kafka partitions.

Test 8 results show that the DI MARC Authority performance is about the same between the Lotus and Morning Glory releases.

Jobs comparison with different users' network connections for lcp2, 50 Kafka partitions (18/08/2022).

Observations:

Jobs overall time is the time from the beginning of file upload (which is strongly affected by user network speed with user interaction) to the actual finish of the import. To have more accurate results from testing it is better to run the import process right after the file upload process. For example, if after uploading the file a user leaves the computer for 5 minutes before coming back to kick off the import then the overall import time also includes the 5 idle minutes.