The mod-data-export-spring and mod-data-export-worker projects are well suited for performing required tasks since they already contain configured mechanisms for saving files to s3 bucket, as well as integration with the spring batch library, which allows you to process large amounts of data quickly and comfortably. Also, the library provides very flexible "job" customization.
...
| Field from Response | Description | Question/assumption | Answer |
|---|---|---|---|
| Total | Count of initial input records | ||
| Processed | Count of processed records in bulk-edit | ||
| Matched | Count of matched (updated) records in bulk-edit |
Magda Zacharska could you please clarify if the User should see in 'matched records' the records that were updated on the last step of bulk-edit? | Denis the expected behavior was described in UIBULKED-46 and was supposed to be delivered in scope of MODEXPW-69 but the current implementation differs from what was expected. Matched records are displayed after the user uploads the file with identifiers or runs the query. After the bulk edit is completed the accordion should be named Preview of records changed as displayed on the following mockup: https://drive.google.com/file/d/1Uyr5_6zOLWnWEyukwJINBWFy2l8LjmMk/view?usp=sharing |
| Success | Count of successfully processed records in bulk-edit | ||
| Errors | Count of records that have errors in bulk-edit |
...
Mitigation: The Spring Batch supplies the AsyncItemProcessor and AsyncItemWriter which allows doing logic asynchronously instead of using a single thread for processing a chunk of the data. As well the spring batch Partitioner allows splitting the processing of a file into several threads. We have to just configure the ThreadPoolTaskExecutor for Processor and Writer or Partitioner Step.