Infrastructure

PTF -environment

10 m6i.2xlarge EC2 instances
2 instances of db.r6.xlarge database instances, one reader and one writer
MSK
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=120

#	modules	task definition	running tasks	CPU	memory	memoryReservation	maxMetaspaceSize	Xmx
#	modules	task definition	running tasks	CPU	memory	memoryReservation	maxMetaspaceSize	Xmx
1	mod-inventory-storage	10	2	1024	2208	1952	512m	1440m
2	okapi	3	3	1024	1684	1440	512m	922m
3	mod-invoice-storage	3	2	1024	1872	1536	512m	1024m
4	mod-orders	5	2	1024	2048	1440	512m	896m
5	mod-agreements	5	2	128	1592	1488	512m	968m
6	mod-data-export-worker	22	2	1024	3072	2800	612m	2128m
7	mod-data-export-spring	6	1	256	2048	1844	512m	1292m
8	mod-notes	3	2	128	1024	896	128m	768m
9	mod-orders-storage	4	2	128	1024	896	128m	768m
10	mod-inventory	4	2	1024	2880	2592	512m	1814m

Software Versions

Test #	Jobs	Orders	Success rate	Time
1	10	100	6/10	±10 sec
2	10	500	8/10	55-57 sec
3	10	1000	7/10	3 min
4	10	1000	9/10	2 min 17 s
5	10	1000	8/10	2 min 48 s
6	10	2000	10/10	±5 min

At this point we can conclude that more-or less safe limit is 8-10 jobs with 1000 orders per organisation
Most of jobs failed with Connection refused (Connection refused). However it doesn’t looks like it's the matter of FTP server. We did observe same behavior previously with 3-4 parallel job, and it was fixed with increasing of memory parameters on the module side. At same moment we can't see significant resource usage on FTP server side.
It looks like percentage of success if highly depended on orders number. The more orders number - the higher chance to complete all jobs successfully.