The following resources are used:
...
2 items for all items (except mod-data-import)
2 partition for all DI kafka topics:
Examples:
delete old topic
# ./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --delete --topic perf-eks-folijet.Default.fs09000000.DI_ERROR
recreate topic with "--partitions 2 --replication-factor 1"
# ./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --create --topic perf-eks-folijet.Default.fs09000000.DI_ERROR --partitions 2 --replication-factor 1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic perf-eks-folijet.Default.fs09000000.DI_ERROR.
get topic info
# ./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --describe --topic perf-eks-folijet.Default.fs09000000.DI_ERROR
Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR PartitionCount: 2 ReplicationFactor: 1 Configs: min.insync.replicas=1,message.format.version=2.6-IV0,unclean.leader.election.enable=true
Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: perf-eks-folijet.Default.fs09000000.DI_ERROR Partition: 1 Leader: 2 Replicas: 2 Isr: 2
mod-data-import: -XX:MaxRAMPercentage=85.0 -XX:+UseG1GC / cpu: 128m/192m | memory: 1Gi/1Gi
...
| env | profile | records number | time in Morning Glory | time in Lotus | Kafka partition number | module instance number | CPU | description |
|---|---|---|---|---|---|---|---|---|
MG Perf Rancher | PTF Create - 2 | 5000 | 7 min | 8 min | 2 | 2 | 512/1024 | mod-source-record-manager-3.4.0-SNAPSHOT.659+ 2022-06-10T07:44:27.576+00:00 2022-06-10T07:51:11.140+00:00 |
MG Perf Rancher | PTF Create - 2 | 5000 | 7 min | 8 min | 2 | 2 | 512/1024 | mod-source-record-manager-3.4.0-SNAPSHOT.659+ -Ddi.flow.control.enable=false 2022-06-14T10:24:44.093+00:00 2022-06-14T10:31:54.725+00:00 |
MG Perf Rancher | PTF Update - 1 | 5000 | 11 min | 13 min | 2 | 2 | 512/1024 | mod-source-record-manager-3.4.0-SNAPSHOT.659+ 2022-06-20T06:18:46.748+00:00 2022-06-20T06:30:02.991+00:00 |
MG Perf Rancher | PTF Create - 2 | 10`000 | 16 min | 19 min | 2 | 2 | 512/1024 | mod-source-record-manager-3.4.0-SNAPSHOT.659+ 2022-06-10T07:54:23.720+00:00 2022-06-10T08:08:48.484+00:00 |
MG Perf Rancher | PTF Create - 2 | 10`000 | 16 min | 19 min | 2 | 2 | 512/1024 | mod-source-record-manager-3.4.0-SNAPSHOT.659+ 2022-06-14T10:36:41.482+00:00 2022-06-14T10:53:03.556+00:00 |
MG Perf Rancher | PTF Update - 1 | 10`000 | 22 min | 25 min | 2 | 2 | 512/1024 | mod-source-record-manager-3.4.0-SNAPSHOT.659+ 2022-06-20T07:07:00.594+00:00 2022-06-20T07:28:54.905+00:00 |
MG Perf Rancher | PTF Create - 2 | 50`000 | 59 min | 1h 25min | 2 | 2 | 512/1024 | mod-source-record-manager-3.4.0-SNAPSHOT.659+ 2022-06-10T08:12:29.178+00:00 2022-06-10T09:11:34.642+00:00 |
| MG Perf Rancher | PTF Update - 1 | 50`000 | 1h 42 min | 2h 17min | 2 | 2 | 512/1024 | mod-source-record-manager-3.4.0-SNAPSHOT.659+ 2022-06-20T09:11:41.701+00:00 2022-06-20T10:54:29.378+00:00 |
MG Perf Rancher | PTF Create - 2 | 100`000 | 2h 20min | 2h 24min (22 errors) | 2 | 2 | 512/1024 | mod-source-record-manager-3.4.0-SNAPSHOT.659+ 2022-06-13T09:30:35.574+00:00 2022-06-13T12:26:52.484+00:00 |
| MG Perf Rancher | PTF Update - 1 | 100`000 | 2h 49min | 4h 40min (tests were made for 1 instance number and partition number | 2 | 2 | 512/1024 | mod-source-record-manager-3.4.0-SNAPSHOT.659+ 2022-06-21T11:46:43.175+00:00 2022-06-21T14:36:05.532+00:00 57 errors Inventory/Inventory-storage errors: io.netty.channel.StacklessClosedChannelException, io.vertx.core.impl.NoStackTraceThrowable: Connection is not active now, current status: CLOSED io.vertx.core.impl.NoStackTraceThrowable: Timeout |
MG Perf Rancher | PTF Create - 2 | 500`000 | 14h 46min (60 errors) | 15h 37min (31 errors) | 2 | 2 | 512/1024 | mod-source-record-manager-3.4.0-SNAPSHOT.659+ 60 errors 2022-06-13T14:27:40.568+00:00 2022-06-14T05:14:27.458+00:00 |
| MG Bugfest | Default Marc Bib Create | 5000 | 4min | 2 | 2 | 512/1024 | mod-source-record-manager - Xmx = 2G "startedDate" : "2022-08-15T14:57:13.753+00:00", "completedDate" : "2022-08-15T15:01:29.365+00:00", | |
| MG Bugfest | Default Marc Bib Create | 10000 | 10min | 2 | 2 | 512/1024 | mod-source-record-manager - Xmx = 2G "startedDate" : "2022-08-15T15:03:07.364+00:00", "completedDate" : "2022-08-15T15:13:28.827+00:00" | |
| MG Bugfest | Create SRS MARC Authority | 5000 | 5min | 2 | 2 | 512/1024 | mod-source-record-manager - Xmx = 2G "startedDate" : "2022-08-16T00:15:55.396+00:00", "completedDate" : "2022-08-16T00:20:19.240+00:00", | |
| MG Bugfest | Create SRS MARC Authority | 10000 | 8 min | 2 | 2 | 512/1024 | mod-source-record-manager - Xmx = 2G "startedDate" : "2022-08-16T14:52:53.191+00:00", "completedDate" : "2022-08-16T15:00:12.723+00:00" | |
| MG Bugfest | Create SRS MARC Authority | 50000 | 34min | 2 | 2 | 512/1024 | mod-source-record-manager - Xmx = 2G "startedDate" : "2022-08-16T15:01:15.360+00:00", "completedDate" : "2022-08-16T15:35:37.028+00:00", |
Results before flow control fix: MODSOURMAN-811
...