EDIFACT Order Export workflow [Nolana]


Overview

Retesting of EDIFACT for Nolana release in scope of PERF-311. EDIFACT is one of data-export-worker workflows that includes organisations and orders export to FTP server. 


Infrastructure


ModuleVersionMax Metaspace size MBXMX MBSoft LimitHard Limit
mod-data-export-worker1.5.0-SNAPSHOT.76612212826003072
mod-data-export-spring1.5.0-SNAPSHOT.58612153618442048
mod-orgers12.5.0-SNAPSHOT.2295128968961024
mod-orgers-storage13.4.0-SNAPSHOT.154 1287688961024
mod-organizations1.6.0-SNAPSHOT.321287688961024
mod-organizations-storage4.3.1-SNAPSHOT.691287688961024
mod-configuration5.9.0-SNAPSHOT.2821287688961024



Summary

  • EDIFACT is stable and it can successfully export up to 3000 orders if it's one job at the time;
  • For 3-10 jobs EDIFACT export is unstable due to FTP server issues;
  • We still can observe same behavior as it was in previous report EDIFACT Order Export workflow report 3 [Morning Glory]. The higher number of orders causes higher success rate. However for Nolana release we wasn't able to reach success rate higher than 60% due to FTP issues. 
  • Duration of exports decreased in comparison to Morning Glory.
  • Most of resource usage decreased in comparison to Morning Glory release. 
  • Ticket to investigate FTP server issue MODEXPW-302 - Getting issue details... STATUS

Note: right now FTP server is located in US.EAST-1 region while our env is located in US.WEST-2 which might cause higher failure rate. In future we'll retest EDIFACT with new FTP server on US.WEST-2.

Summary after retest

  • EDIFACT after fixing  MODEXPW-302 - Getting issue details... STATUS is able to complete 3000 orders 10 jobs simultaneously;
  • FTP issue is no longer reproducible;
  • No additional issues were found;
  •  We still can observe small memory leak on mod-data-export-worker (1-2% per jobs set) We'll investigate it in scope of PERF-359



Test Results


Test #Concurrent jobsOrders per jobSuccess rateTimeERRORS

1

310002/31min 17s

File upload failed. 450 PASV: data transfer in progress (FtpException)

2510002/51min 17s

null (NullPointerException)


31010002/101min 20s

(FtpException)

File upload failed. 450 PASV: data transfer in progress (FtpException)

File upload failed. 226 Transfer complete (FtpException)


41020006/102m 54 s

Connection is not open (IOException)

null (NullPointerException)

Socket closed (SocketException)


51030006/104min 30s

Socket closed (SocketException)

Connection closed without indication. (FTPConnectionClosedException)

Connection is not open (IOException)


Resources usage


Note: In comparison with Morning Glory release modules CPU usage decreased. for comparison mod-configuration from 400 to 300 max (for corresponding tests) same for the rest of a modules. 


Note: we can see slight increasing of memory usage on mod-data-export-worker (here more spikes of memory usage) and mod-data-export-spring (from 17% to 23%). However while memory usage do not increases from test to test we can say that it's working conditions of these modules. 


Note: In comparison with Morning Glory release RDS CPU usage decreased from 70-80% to 30-35%



FTP Issues

Note: According to Serhii_Nosko and his code analysis - there is no retry in mod-data-export-worker to ftp server, which mean that if it was not able to connect from the first time to the FTP server it will fail the whole process. 

Also nullPointerException is related to FTP server issue as well. 








Retest after bug fixes set #1

In scope of MODEXPW-302 - Getting issue details... STATUS development team has created fix to the FTP server issue MODEXPW-302

Results


Test #Concurrent jobsOrders per jobSuccess rateTimeERRORS
110100010/101 min 30snone
210 100010/101 min 31snone


Resource usage




Note: We can observe growing of mod-data-export-worker memory usage, however after several additional tests we wasn't able to see growing. 


Note: DB CPU usage is ±25% during both tests. 


Retest after bug fixes set #2


Results


Test #Concurrent jobsOrders per jobSuccess rateTimeERRORS

1

1050010/1047 s

-

210100