The following resources are used:
...
Previous Lotus testing performance results:
Lotus Snapshot Performance testing
Modules:
Data Import Module (mod-data-import-2.5.0-SNAPSHOT.231)
...
2 Partition for all DI Kafka topics
...
Please Notice: an environment should be configured in such a way that for every Kafka topic there are as many partitions as many instances created for a module connected to that topic
Examples:
| Code Block | ||
|---|---|---|
| ||
./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --delete --topic perf-eks-folijet.Default.fs09000000.DI_ERROR |
...
| Code Block | ||
|---|---|---|
| ||
./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --create --topic perf-eks-folijet.Default.fs09000000.DI_ERROR --partitions 2 --replication-factor 1 |
| Warning |
|---|
Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. |
| Code Block | ||
|---|---|---|
| ||
./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --describe --topic perf-eks-folijet.Default.fs09000000.DI_ERROR |
| Panel | ||
|---|---|---|
| ||
Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR PartitionCount: 2 ReplicationFactor: 1 Configs: min.insync.replicas=1,message.format.version=2.6-IV0,unclean.leader.election.enable=true |
JVM
...
mod-data-import: -XX:MaxRAMPercentage=85.0 -XX:+UseG1GC / cpu: 128m/192m | memory: 1Gi/1Gi
...
Results before flow control fix: MODSOURMAN-811
| env | profile | records number | time | time in Lotus | Kafka partition number | module instance number | CPU | description |
|---|---|---|---|---|---|---|---|---|
| MG Perf Rancher | PTF Create - 2 | 5000 | 7 min | 8 min | 2 | 2 | 512/1024 | mod-source-record-manager-3.4.0-SNAPSHOT.621 2022-05-27T12:58:30.331+00:00 2022-05-27T13:05:08.683+00:00 |
MG Perf Rancher | PTF Update - 1 | 5000 | 10 min | 13 min | 2 | 2 | 512/1024 | 2022-05-27T13:22:35.123+00:00 2022-05-27T13:32:35.344+00:00 |
MG Perf Rancher | PTF Create - 2 | 10`000 | 21 min | 27min | 19 min | 2 | 2 | 512/1024 | -Ddi.flow.control.enable=false 2022-05-30T09:51:13.876+00:00 | 2022-05-31T18:13:05.977+00:00 2022-05-30T10:12:33.982+00:00 | 2022-05-31T18:40:58.928+00:00 |
MG Perf Rancher | PTF Update - 1 | 10`000 | 30 min | 25 min | 2 | 2 | 512/1024 | -Ddi.flow.control.enable=false 2022-05-31T19:19:46.296+00:00 2022-05-31T19:49:59.651+00:00 |
MG Perf Rancher | PTF Create - 2 | 10`000 | 21 min | 19 min | 2 | 2 | 512/1024 | -Ddi.flow.control.enable=true 2022-05-31T20:02:06.368+00:00 2022-05-31T20:23:19.490+00:00 |
MG Perf Rancher | PTF Update - 1 | 10`000 | 31 min | 25 min | 2 | 2 | 512/1024 | -Ddi.flow.control.enable=true 2022-06-01T19:08:11.563+00:00 2022-06-01T19:39:58.803+00:00 |
MG Perf Rancher | PTF Create - 2 | 10`000 | 17 min | 19 min | 2 | 2 | 512/1024 | -Ddi.flow.control.enable=true 2022-06-03T09:20:07.654+00:00 2022-06-03T09:37:51.631+00:00 |
MG Perf Rancher | PTF Create - 2 | 30`000 | 1h 6 min | 45 min | 2 | 2 | 512/1024 | 2022-05-27T13:37:12.980+00:00 2022-05-27T14:31:52.595+00:00 |
MG Perf Rancher | PTF Update - 1 | 30`000 | 1h 26min | - | 2 | 2 | 512/1024 | 2022-05-27T15:37:33.580+00:00 2022-05-27T17:03:15.702+00:00 |
MG Perf Rancher | PTF Create - 2 | 50`000 | 2h 37 min | 1h 25min | 2 | 2 | 512/1024 | 3 errors: 2022-06-01T19:48:33.977+00:00 2022-06-01T22:25:59.700+00:00 |
...
Performance testing of update item scenario after removal of index (MODDATAIMP-697):
In the scope of FOLIO-3388 "item_status_name_idx_gin" index for the Item status field was removed from mod-inventory-storage. However, this index potentially can be used for matching by Item status during Item update.
...