LCP performance testing

  PERF-232 - Getting issue details... STATUS  

currents software configuration


Summary

Several series of tests were performed. Last version of modules were deployed, and configuration of modules changed by recommendations of Folijet team. Details in the table

performance optimized configuration, the number of partitions in Kafka was also increased to 2. Import times were not significantly different, although CPU usage was reduced. Before configuration changes maxim CPU usage for SRM module was about 900% Inventory 700%

after changes SRM was maximum for CRM 300% from Inventory 250%.

BUT it should be noted that a general decrease in performance (import time) was found in all tests, even when returning to the state of the test on March 13

Аs possible reasons for the performance degradation, it is possible that Kafka or the database is not optimally configured.


Modules:


Moduleprevious versioncurrent version on lcp1
Data Import Modulemod-data-import-2.3.0-SNAPSHOT.224mod-data-import-2.4.1
Source Record Manager Modulemod-source-record-manager-3.3.0-SNAPSHOT.556mod-source-record-manager-3.3.8
Source Record Storage Modulemod-source-record-storage-5.3.0-SNAPSHOT.394mod-source-record-storage-5.3.2
Inventory Modulemod-inventory-18.1.0-SNAPSHOT.493mod-inventory-18.1.4
Inventory Storage Modulemod-inventory-storage-23.0.0-SNAPSHOT.657mod-inventory-storage-23.0.2
Invoice business logic modulemod-invoice-5.3.0-SNAPSHOT.284mod-invoice-5.3.0

Performance-optimized configuration:

ModuleCPU Initial

CPU Optimized

memory Initial MEMORY optimizedjava options (optimized, only in case other changes)task instances #
Data Import Module

128

1282048/10242048/1024
1
Source Record Manager Module5125122048/18441440/1296-XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -Xmx1292m -XX:+UseG1GC2
Source Record Storage Module

128

5121440/12961440/1296 -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=384m -Xmx908m -XX:+UseG1GC2
Inventory Module5125122048/18442880/2592-XX:MetaspaceSize=182m -XX:MaxMetaspaceSize=182m -Xmx1814m -XX:+UseG1GC2
Inventory Storage Module512512864/778846/778
2
Invoice business logic module1281281024/8001024/800
2
mod-data-import-converter-storage128512512/460512/460
2
okapi





kafka topics related to di - number of partitions raised to 2

lcp1.Default.fs09000000.DI_COMPLETED
lcp1.Default.fs09000000.DI_EDIFACT_RECORD_CREATED
lcp1.Default.fs09000000.DI_ERROR
lcp1.Default.fs09000000.DI_INVENTORY_AUTHORITY_CREATED_READY_FOR_POST_PROCESSING
lcp1.Default.fs09000000.DI_INVENTORY_HOLDING_CREATED
lcp1.Default.fs09000000.DI_INVENTORY_HOLDING_MATCHED
lcp1.Default.fs09000000.DI_INVENTORY_HOLDING_NOT_MATCHED
lcp1.Default.fs09000000.DI_INVENTORY_HOLDING_UPDATED
lcp1.Default.fs09000000.DI_INVENTORY_INSTANCE_CREATED
lcp1.Default.fs09000000.DI_INVENTORY_INSTANCE_CREATED_READY_FOR_POST_PROCESSING
lcp1.Default.fs09000000.DI_INVENTORY_ITEM_MATCHED
lcp1.Default.fs09000000.DI_LOG_SRS_MARC_AUTHORITY_RECORD_CREATED
lcp1.Default.fs09000000.DI_LOG_SRS_MARC_BIB_RECORD_CREATED
lcp1.Default.fs09000000.DI_MARC_FOR_UPDATE_RECEIVED
lcp1.Default.fs09000000.DI_PARSED_RECORDS_CHUNK_SAVED
lcp1.Default.fs09000000.DI_RAW_RECORDS_CHUNK_PARSED
lcp1.Default.fs09000000.DI_RAW_RECORDS_CHUNK_READ
lcp1.Default.fs09000000.DI_SRS_MARC_AUTHORITY_RECORD_CREATED
lcp1.Default.fs09000000.DI_SRS_MARC_BIB_INSTANCE_HRID_SET
lcp1.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_CREATED
lcp1.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_MATCHED
lcp1.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_MATCHED_READY_FOR_POST_PROCESSING
lcp1.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_MODIFIED
lcp1.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_MODIFIED_READY_FOR_POST_PROCESSING
lcp1.Default.fs09000000.DI_SRS_MARC_BIB_RECORD_NOT_MATCHED
lcp1.Default.fs09000000.DI_SRS_MARC_HOLDING_RECORD_CREATED

example change partition number if topic already exist

./kafka-topics.sh --alter --zookeeper\
 z-1.kafka280.xwt0mn.c17.kafka.us-east-1.amazonaws.com:2181,z-2.kafka280.xwt0mn.c17.kafka.us-east-1.amazonaws.com:2181,z-3.kafka280.xwt0mn.c17.kafka.us-east-1.amazonaws.com:2181\
 --topic lcp1.Default.fs09000000.DI_SRS_MARC_HOLDING_RECORD_CREATED --partitions 2

if topic not exist yet

 ./kafka-topics.sh --create --zookeeper\
 z-1.kafka280.xwt0mn.c17.kafka.us-east-1.amazonaws.com:2181,z-2.kafka280.xwt0mn.c17.kafka.us-east-1.amazonaws.com:2181,z-3.kafka280.xwt0mn.c17.kafka.us-east-1.amazonaws.com:2181\
 --topic lcp1.Default.fs09000000.DI_EDIFACT_RECORD_CREATED --partitions 2 --replication-factor 1

additional tests (*) - added CPU UNITS for module data-import-CS from 128 to 512

 (**) - cleaned tables DI (deleted records from 03 marth 2022)


DI durations

fileprofileduration

duration from Folijet - Lotus Snapshot Performance testing

(for comparison)

Data Import Test report (Lotus)

duration with revert configuration

just before 

2022-04-26

duration with revert configuration


2022-03-17

5KPTF - Create 2 204 min 04 sec~8 min3 min 54 s05 min 44 sec04 min 44 sec
5KUpdates Success - 105 min 26 sec~13 min4 min 12 s

10KPTF - Create 2

07 min 00 sec (8 mins)

00:09:55.36

~ 19 min6 min 45 s
08 min 50 sec
10KUpdates Success - 111 min 46 sec~ 25 min8 min 4 s
12 mon 03 sec
25KPTF - Create 2

25 min 48 sec (19:49.063*)

(17:33.521**)

~ 45 min16 min 8s38 min 34 sec22 min 52 sec
25KUpdates Success - 129 min 08 sec~ 1h 31min19 min 50s

50KPTF - Create 2FAILED



50KPTF - Create 240 min 00 (41 min)~ 1h 34min32 min 28 s

5K create charts

mod-data-import-converter-storage we see too much CPU usage