Change logic to initialize job execution progress after reading file instead of processing the first chunk
RCA Group
Description
Environment
Potential Workaround
Attachments
- 30 May 2022, 07:15 AM
defines
Checklist
hideTestRail: Results
Activity
Ann-Marie Breaux May 30, 2022 at 8:28 AM
Hi @Serhii_Nosko Sounds good - let's close it.
@maksym_dryha please create a TestRail that uploads 3 large-ish files (ca. 500 records), uses the default Create Instance and SRS MARC Bib job profile, and confirms that the cards for the second and third files appear in the "Running" section of the data import page before the first file finishes. If you leave a comment here after you have created the TestRail, I'm happy to review it. It should be in the Extended group.
Serhii_Nosko May 30, 2022 at 7:17 AM
Hi @Ann-Marie Breaux, I've not yet tested on snapshot/snapshot-2 env earlier due to failures with this envs, that has been fixed now, but unfortunately master build for snapshot envs does not yet include my changes, we expect, that they will be deployed today to snapshot-2 env.
So in order to be able to close ticket in the current spring I've deployed these changes to our Racnher env and all looks good: https://folijet.ci.folio.org/data-import
Simultaneously I uploading couple of files and they all appearing quickly in the Running section now(whithout waiting until previous upload finishes). Here is screen from Rancher env with my testing, so I think we can close this ticket.
Ann-Marie Breaux May 18, 2022 at 1:18 PM
@Olamide Kolawole to discuss possibilities with BE devs and decide which approach is best. After it is implemented, test to ensure UI behavior is satisfactory
Details
Details
Assignee
Reporter
Priority
P2
Issue to resolve:
When importing 2 or more files simultaneously(meaning upload second file after first started but not completed) the second file does not appear in the Running section. It can confuse user when he uploads file and does not see any response from the system.
When and why issue reproduces:
It reproduces when Flow control feature is enabled starting from Morning Glory, because algorithm of processing raw chunks changed. Previously SRM consumes all raw chunks at the beginning of import, that allows to other jobs to consume first chunks and initialize job execution progress. When job execution progress initialized - UI will show corresponding jobs in Running section.
Necessary changes:
Need to change logic to initialize job execution progress after parsing file in mod-data-import and do not wait until first raw records chunk coming.
To initialize job execution progress we need job execution id and total number of records in the file. Mod-data-import already contains all this info.
There are 2 options how to implement this behaviour: (need decision from @Olamide Kolawole)
create new endpoint in srm that will initialize job execution progress. It will accept 2 params: jobExecutionId and totalCount of records. In this case mod-data-import should invoke this new endpoint to initialize job execution progress.
create new topic that will store job executions to be run. Mod-data-import should push messages that contains jobExecutionId and totalCount and mod-source-record-manager will consume them and initialize job execution progress
Acceptance criteria
Once the MOD change is made, test in UI with 2 large jobs started separately - to confirm that the second job's card appears in the "Running" area shortly after the first