Overview

Per PERF-267, test EDIFACT PERF-270 exports of 10K records to understand the workflow behavior before and when the mod-data-export-worker task crashes, if it crashes at all. 

Infrastructure

• 10 m6i.2xlarge EC2 instances  (changed. In Lotus it was m5.xlarge)
• 2 instances of db.r6.xlarge database instances, one reader and one writer
• MSK. (topics being used [cluster].[tenant].data-export.job.command ,  [cluster].[tenant].data-export.job.update)
  • 4 m5.2xlarge brokers in 2 zones
  • auto.create-topics.enable = true
  • log.retention.minutes=120
  • 2 partitions per DI topics

Software Versions

• mod-data-export-worker v 1.4.4
• mod-data-export-spring v 1.4.4
• mod-agreements:5.2.1
• mod-notes:3.1.0
• mod-orders-storage:13.3.2
• mod-orders:12.4.3

Results

Summary 

System can handle from 6 concurrent jobs. However there is different succession rate each time. For instance: 

• 10 concurrent jobs 1000 order per organisation  - 6 successfully ended;
• 6 concurrent  jobs 1000 order per organisation - 4 successfully ended;
• 6 concurrent  jobs 1000 order per organisation - 5 successfully ended;
• 4 concurrent  jobs 1000 order per organisation - 2 successfully ended;
• 4 concurrent  jobs 500 order per organisation - 0 successfully ended.

Notable observations: 

Jobs are failing due to 

ERRORS analysis

All jobs failed with issue related to FTP server. Unfortunately we can't access there to check what exactly went wrong and check if it configurable or not:

09:45:41 [] [] [] [] ERROR AbstractStep Encountered an error executing step saveToFTPStep in job EDIFACT_ORDERS_EXPORTjava.lang.reflect.UndeclaredThrowableException: null

08:46:32 [] [] [] [] ERROR ectStorageRepository Error uploading filejava.io.IOException: Connection is not openat org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:514) ~[commons-net-3.6.jar!/:3.6]

Resources usage 

Resource usage showed for 4 concurrent jobs 1000 orders per organisation 

Here you can see CPU usage of related modules for EDIFACT export. Highest usage was on mod-orders-storage (85%) 

Memory usage is stable, however when exporting 10 organisation with 1000 orders each memory consumption on mod-data-export-worker growing 2-3% per each run. 

whole process consumes only 8% max of instance CPU.

DB CPU reached 35% max.