Platform, DevOps and Release Management (UXPROD-1814)

[UXPROD-1928] APIs for batch downloads (exports) Created: 05/Aug/19  Updated: 16/Sep/20  Resolved: 23/Mar/20

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: Q1 2020
Parent: Platform, DevOps and Release Management

Type: New Feature Priority: P3
Reporter: Jakub Skoczen Assignee: Jakub Skoczen
Resolution: Done Votes: 0
Labels: cap-mvp, platform-backlog, po-mvp
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
is blocked by MODINVSTOR-288 roll out RMB support for streaming re... Open
is blocked by RMB-367 Documentation about downloads using G... Open
is blocked by CQLPG-88 Implement id > [uuid] for pagination Closed
is blocked by FOLIO-1774 Scalable stream support Closed
is blocked by MODUSERS-114 GET users as a stream Closed
Relates
relates to MODINVSTOR-452 Create size optimized bulk download e... Open
relates to MODINVSTOR-439 Create bulk download endpoint to save... Closed
relates to MODINVSTOR-295 streaming POST holdings Closed
relates to MODINVSTOR-296 streaming POST items Closed
relates to RMB-389 PgUtil.delete by CQL Closed
relates to FOLIO-1839 Enable incremental requests for all r... Open
relates to MODINV-119 Create endpoint to save collection of... Closed
relates to MODINVSTOR-294 streaming POST instance Closed
relates to UXPROD-1826 APIs for batch uploads (imports) Closed
Epic Link: Platform, DevOps and Release Management
Back End Estimate: XXL < 30 days
Estimation Notes and Assumptions: Includes API design, RMB support and rollout to a selected module (PoC). Excludes rollout across all FOLIO modules.
Development Team: Core: Platform
PO Rank: 9
Rank: Chalmers (Impl Aut 2019): R4
Rank: Chicago (MVP Sum 2020): R1
Rank: Cornell (Full Sum 2021): R1
Rank: Duke (Full Sum 2021): R4
Rank: 5Colleges (Full Jul 2021): R1
Rank: GBV (MVP Sum 2020): R2
Rank: Lehigh (MVP Summer 2020): R1
Rank: MO State (MVP June 2020): R1
Rank: TAMU (MVP Jan 2021): R1
Rank: U of AL (MVP Oct 2020): R1

 Description   

Problem statement

FOLIO's API are focused on fetching (GET) single record at a time and while this process can be tuned through various HTTP optimisation techniques (keepalive, pipelining) it's generally tricky to ensure that all clients are configured properly.

To ensure good performance, it is much easier to provide dedicated batch APIs for batch fetching (downloads).

Implementation wise, batch downloads should support streaming of data directly from the HTTP connection into a DB connection.

Streaming downloads

There is an example on how to implement streaming downloads with RMB in mod-users. Requires code review and maybe polishing.
https://github.com/folio-org/mod-users/compare/user-stream-example


Generated at Fri Feb 09 00:19:41 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.