Overview

Per PERF-267, test Eholdings exports (PERF-273) of 10K  records to understand the workflow behavior before and when the mod-data-export-worker task crashes, if it crashes at all. 

• How long does it take to export 10K records?
•  What happens to the job that is running, will it be able to resume and complete successfully when the new task is spun up? 
• Look for a memory trend and use it to decide on the number of concurrent jobs needed to reach the tipping point.  

Infrastructure

• 10 m6i.2xlarge EC2 instances  (changed. In Lotus it was m5.xlarge)
• 2 instances of db.r6.xlarge database instances, one reader and one writer
• MSK
  • 4 m5.2xlarge brokers in 2 zones
  • auto.create-topics.enable = true
  • log.retention.minutes=120
  • 2 partitions per DI topics

Software Versions

• mod-data-export-worker v 1.4.1 
• mod-data-export-spring v 1.4.1
• mod-agreements:5.2.0
• mod-notes:3.1.0

Results

Summary 

• This is initial test report for exporting Eholdings functionality. 
• Approximately 10K records can be exported in 30 minutes (we did test it with 9 631 titles and it takes 27 minutes package eholdings/packages/53-1094073).
• 2K export was completed in ±4 minutes;
• System is unstable and often fail during procedure with symptoms MODEXPW-170
  
  • Job is completed and file download link is active (like in job#000034 on the screen below) also you can download exported file with whole amount of data.
  • Start time and End time becomes equal;
  • In DB job status still in progress.
  • This status never changes and block other jobs after it (they getting status "scheduled") until you'll restart mod-data-export-worker, mod-data-export-spring and the explicitly change job status in DB to "FAILED".
• Memory trend: Memory is growing - however when container (mod-data-export-worker) starts the memory usage is on 15% rate. and after finishing 10K export it's on 29%. It's hard to determine memory trends while jobs usually stuck and you have to restart container to proceed. 
• Failover  The mod-data-export-worker's container was restarted to simulate a crash. The ongoing job got stuck with the result described above (link to download is active, but job is forever in progress).

On this screenshot three attempts of exporting. 

Notable observations

• There is no way to track exporting progress. Like how much records is transferred at this time? 
• there is no way to check how big file is exported. There is only jobId and Job time containing useful info. 

• UI does not updates by itself. only after page restart Which means additional calls on back end and more resource usage