Platform, DevOps and Release Management (UXPROD-1814)

[UXPROD-1826] APIs for batch uploads (imports) Created: 02/Apr/19  Updated: 16/Sep/20  Resolved: 19/Dec/19

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: Q4 2019
Parent: Platform, DevOps and Release Management

Type: New Feature Priority: P3
Reporter: Jakub Skoczen Assignee: Jakub Skoczen
Resolution: Done Votes: 0
Labels: cap-mvp, platform-backlog, po-mvp
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
is blocked by MODINVSTOR-353 implement batch upload for /instances... Closed
is blocked by CQLPG-88 Implement id > [uuid] for pagination Closed
is blocked by FOLIO-1774 Scalable stream support Closed
Relates
relates to FOLIO-2050 SPIKE Design batch create / update AP... Closed
relates to RMB-246 migrate to reactive postgres client (... Closed
relates to UXPROD-1928 APIs for batch downloads (exports) Closed
relates to DEBT-3 Slow or missing batch upload/download... In Review
relates to MODINV-119 Create endpoint to save collection of... Closed
relates to MODINVSTOR-294 streaming POST instance Closed
Epic Link: Platform, DevOps and Release Management
Back End Estimate: XXL < 30 days
Estimation Notes and Assumptions: Includes API design, RMB support and rollout to a selected module (PoC). Excludes rollout across all FOLIO modules.
Development Team: Core: Platform
PO Rank: 9
Rank: Chalmers (Impl Aut 2019): R4
Rank: Chicago (MVP Sum 2020): R1
Rank: Cornell (Full Sum 2021): R1
Rank: Duke (Full Sum 2021): R1
Rank: 5Colleges (Full Jul 2021): R1
Rank: GBV (MVP Sum 2020): R2
Rank: Lehigh (MVP Summer 2020): R1
Rank: MO State (MVP June 2020): R1
Rank: TAMU (MVP Jan 2021): R1
Rank: U of AL (MVP Oct 2020): R1

 Description   

Problem statement

FOLIO's API are designed for creating/updating (POST/PUT) single record at a time and while this process can be tuned through various HTTP optimisation techniques (keepalive, pipelining) it's generally tricky to ensure that all clients are configured properly.

To ensure good performance, it is much easier to provide dedicated batch APIs for creation/updates (download).

Transport protocol wise, batch uploads should support streaming of data directly from the HTTP connection into a DB connection (already supported by RMB)

Streaming upload



 Comments   
Comment by Jakub Skoczen [ 08/Apr/19 ]

Adam Dickmeiss Julian Ladisch We need tickets for rolling out streaming upload across platform-core backend modules.

Comment by patty.wanninger [ 27/Jun/19 ]

Jakub Skoczen As the PO for the migration sub group, is there a way that I can bring my groups' requirements forward for these stories?

Comment by Theodor Tolstoy (One-Group.se) [ 04/Jul/19 ]

NFR? Chalmers will need to do their migration without this, but they would like to have it later.

Comment by Marc Johnson [ 04/Jul/19 ]

Theodor Tolstoy (One-Group.se) Are you asking whether this is an NFR, or what NFR is short for?

Comment by Theodor Tolstoy (One-Group.se) [ 05/Jul/19 ]

The former!

Comment by Marc Johnson [ 06/Jul/19 ]

The former!

Theodor Tolstoy (One-Group.se) I only wanted to check. It wasn't a term I've seen used on FOLIO much. No offence intended :-/

Comment by Theodor Tolstoy (One-Group.se) [ 06/Jul/19 ]

None taken. I usually write a comment like that when this kind of jira emerge without the nfr tag. Adding the tag will make the jiras not show up in the go-live-ranking searches.

Comment by Cate Boerema (Inactive) [ 30/Jul/19 ]

Hi Jakub Skoczen, you are assigned to this feature as PO so you get to give it a PO ranking. That said, I wanted to pass on that Ian Walls (PO for Migration) would really like to see this ( FOLIO-2050 Closed in particular) highly ranked.

Comment by Jakub Skoczen [ 05/Aug/19 ]

I've split downloads (exports) and uploads (imports) into seperate features.

Comment by Jakub Skoczen [ 12/Sep/19 ]

Adam Dickmeiss Marc Johnson Ian Walls

Guys, I put together a couple slides that capture the existing Platform (RMB specifically) support for batch and streaming uploads and downloads. It captures missing functionality and open questions.
https://docs.google.com/presentation/d/1aHmxvWY-u6wjUHVq3BvWf0rsw8zcq6RXtqYDIzxvpIo/edit?usp=sharing

Comment by Jakub Skoczen [ 16/Sep/19 ]

Jakub Skoczen Adam Dickmeiss Oleksii Popov TODO: create an implementation ticket for adding streaming/batch insert method to PostgresClient

Comment by Brooks Travis [ 30/Jan/20 ]

I'm somewhat confused on the status of this feature. It's marked as closed/done, but is also showing as blocked by two issues.

Comment by Julian Ladisch [ 31/Jan/20 ]

I've changed FOLIO-2050 Closed and RMB-246 Closed from blocking issues to related issues.

Generated at Thu Feb 08 23:16:57 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.