EDIFACT Order Export workflow [Nolana]
Overview
Retesting of EDIFACT for Nolana release in scope of PERF-311. EDIFACT is one of data-export-worker workflows that includes organisations and orders export to FTP server.
Infrastructure
Module | Version | Max Metaspace size MB | XMX MB | Soft Limit | Hard Limit |
|---|---|---|---|---|---|
mod-data-export-worker | 1.5.0-SNAPSHOT.76 | 612 | 2128 | 2600 | 3072 |
mod-data-export-spring | 1.5.0-SNAPSHOT.58 | 612 | 1536 | 1844 | 2048 |
mod-orgers | 12.5.0-SNAPSHOT.229 | 512 | 896 | 896 | 1024 |
mod-orgers-storage | 13.4.0-SNAPSHOT.154 | 128 | 768 | 896 | 1024 |
mod-organizations | 1.6.0-SNAPSHOT.32 | 128 | 768 | 896 | 1024 |
mod-organizations-storage | 4.3.1-SNAPSHOT.69 | 128 | 768 | 896 | 1024 |
mod-configuration | 5.9.0-SNAPSHOT.282 | 128 | 768 | 896 | 1024 |
Summary
EDIFACT is stable and it can successfully export up to 3000 orders if it's one job at the time;
For 3-10 jobs EDIFACT export is unstable due to FTP server issues;
We still can observe same behavior as it was in previous report EDIFACT Order Export workflow report 3 [Morning Glory]. The higher number of orders causes higher success rate. However for Nolana release we wasn't able to reach success rate higher than 60% due to FTP issues.
Duration of exports decreased in comparison to Morning Glory.
Most of resource usage decreased in comparison to Morning Glory release.
Ticket to investigate FTP server issue
MODEXPW-302: Investigate and fix FTP connection issuesClosed
Note: right now FTP server is located in US.EAST-1 region while our env is located in US.WEST-2 which might cause higher failure rate. In future we'll retest EDIFACT with new FTP server on US.WEST-2.
Summary after retest
EDIFACT after fixing
FTP issue is no longer reproducible;
No additional issues were found;
We still can observe small memory leak on mod-data-export-worker (1-2% per jobs set) We'll investigate it in scope of PERF-359
Test Results
Test # | Concurrent jobs | Orders per job | Success rate | Time | ERRORS |
|---|---|---|---|---|---|
1 | 3 | 1000 | 2/3 | 1min 17s | File upload failed. 450 PASV: data transfer in progress (FtpException) |
2 | 5 | 1000 | 2/5 | 1min 17s | null (NullPointerException) |
3 | 10 | 1000 | 2/10 | 1min 20s | (FtpException) File upload failed. 450 PASV: data transfer in progress (FtpException) File upload failed. 226 Transfer complete (FtpException) |
4 | 10 | 2000 | 6/10 | 2m 54 s | Connection is not open (IOException) null (NullPointerException) Socket closed (SocketException) |
5 | 10 | 3000 | 6/10 | 4min 30s | Socket closed (SocketException) Connection closed without indication. (FTPConnectionClosedException) Connection is not open (IOException) |
Resources usage
Note: In comparison with Morning Glory release modules CPU usage decreased. for comparison mod-configuration from 400 to 300 max (for corresponding tests) same for the rest of a modules.
Note: we can see slight increasing of memory usage on mod-data-export-worker (here more spikes of memory usage) and mod-data-export-spring (from 17% to 23%). However while memory usage do not increases from test to test we can say that it's working conditions of these modules.
Note: In comparison with Morning Glory release RDS CPU usage decreased from 70-80% to 30-35%
FTP Issues
Note: According to @Serhii_Nosko and his code analysis - there is no retry in mod-data-export-worker to ftp server, which mean that if it was not able to connect from the first time to the FTP server it will fail the whole process.
Also nullPointerException is related to FTP server issue as well.
Retest after bug fixes set #1
In scope of
Results
Test # | Concurrent jobs | Orders per job | Success rate | Time | ERRORS |
|---|---|---|---|---|---|
1 | 10 | 1000 | 10/10 | 1 min 30s | none |
2 | 10 | 1000 | 10/10 | 1 min 31s | none |
Resource usage
Note: We can observe growing of mod-data-export-worker memory usage, however after several additional tests we wasn't able to see growing.
Note: DB CPU usage is ±25% during both tests.
Retest after bug fixes set #2
Results
Test # | Concurrent jobs | Orders per job | Success rate | Time | ERRORS |
|---|---|---|---|---|---|
1 | 10 | 500 | 10/10 | 47 s | - |
2 | 10 | 1000 | 10/10 | 1min 52s | - |
3 | 10 | 2000 | 10/10 | 3min 9s | - |
4 | 10 | 3000 | 10/10 | 4 min 51s | - |
Resource usage
Note: for 2k orders and 3K orders 8 jobs started simultaneously, and 2 jobs have got scheduled status, after most of a jobs were finished - 2 with the scheduled status actually started.
We still can observe increasing of memory usage on mod-data-export-worker module. We'll investigate it in scope of PERF-359