POC: Make data-export horizontally scalable

Description

Overview
Currently as we are storing files locally before export job is triggered.If multiple instances of the module are running, It may happen that both the APIs are run by different instances, there by not finding the file. We need to be able to scale horizontally

Approach
The Ideal approach is to have a way to store the temporary files somewhere accessible to both running instances of module, which probably is needed at a FOLIO level.
APPROACH 1
One way to handle could be to store the uploaded file in S3. When export is triggered, file is looked in the local storage, if not found will be downloaded from S3 and export can continue as usual.

Disadvantages:
1. This method could introduce minor delays because the file needs to be uploaded to S3 and also downloaded before export can begin
2. File cleanup is another issue, because we run a periodic job to cleanup files, that would somehow need to know all the files to be cleaned up

APPROACH 2
This may be a major change ,we could be to load all the requested UUIDs to a database table and read from there, instead of maintaining a file in the local and clearing it up later.
This would need changes to be made to create a new table and also to upload endpoint and then reading would also change to read from the DB rather than from the file

Acceptance Criteria

  • Able to scale horizontally

Environment

None

Potential Workaround

None

relates to

Checklist

hide

TestRail: Results

Activity

Show:

Andrii Novytskyi September 18, 2020 at 1:55 PM

Andrii Novytskyi September 15, 2020 at 7:57 AM

Flag removed

Unblocked because there was a conversation between backend devs

Andrii Novytskyi September 8, 2020 at 1:55 PM

Flag added

Blocked until a discussion with the backend team about the other approach of saving the file with UUIDs that not breaking the scalability

Viktor Soroka August 26, 2020 at 2:47 PM

Discussed on grooming. Need to timebox/estimate.

Kruthi Vuppala April 8, 2020 at 2:35 PM

Done

Details

Assignee

Reporter

Priority

Story Points

Sprint

Development Team

Concorde

Fix versions

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs

Created March 25, 2020 at 2:50 PM
Updated October 14, 2020 at 12:48 PM
Resolved September 18, 2020 at 7:59 PM
TestRail: Cases
TestRail: Runs