Batch Importer (Bib/Acq) (UXPROD-47)

[UXPROD-2790] NFR: Make some revisions to Source Record Storage to improve performance for data migration Created: 26/Oct/20  Updated: 03/Jan/24  Resolved: 01/Feb/21

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: R1 2021
Parent: Batch Importer (Bib/Acq)

Type: New Feature Priority: P2
Reporter: Ann-Marie Breaux (Inactive) Assignee: Ann-Marie Breaux (Inactive)
Resolution: Done Votes: 0
Labels: NFR, data-import, epam-folijet, migration-load, r1-2021-stretch
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Defines
defines UXPROD-47 Batch Importer (Bib/Acq) Analysis Complete
is defined by MODSOURCE-193 source-records?updatedAfter API call ... Closed
is defined by MODSOURCE-201 SRS revision: Stream get source recor... Closed
is defined by MODSOURCE-202 SRS revision: Stream get records; ful... Closed
is defined by MODSOURCE-203 SRS revision: Default order by on pag... Closed
is defined by MODSOURCE-204 SRS revision: Refactor get records to... Closed
is defined by MODSOURCE-205 SRS revision: Refactor batch API impl... Closed
is defined by MODSOURCE-183 Add bulk query/streaming support to /... Closed
is defined by MODSOURCE-242 Use Testcontainers for tests Closed
Epic Link: Batch Importer (Bib/Acq)
Analysis Estimate: Small < 3 days
Analysis Estimator: Ann-Marie Breaux (Inactive)
Front End Estimate: Very Small (VS) < 1day
Front End Estimator: Ann-Marie Breaux (Inactive)
Development Team: Folijet
PO Rank: 90
Rank: TAMU (MVP Jan 2021): R3
Score: 14
Showstopper for Summer 2021 Implementers?: No
Showstopper December 11 Meeting Summary: Not a 'showstopper', so not discussed at meeting.

 Description   

Changes recommended by William Welling/TAMU, based on initial data migration needs; based on his familiarity with SRS, he feels that the performance issues are able to be resolved.

Separate stories created for each of the following, and linked to this feature:

  1. Stream get source records; latest generation record with only marc record
  2. Stream get records; full record with marc record, raw record, and error record if any
  3. Default order by on paging APIs to an appropriate indexed field
  4. Refactor get records to perform optimized database join over n+1 lookups
  5. Refactor batch post source records to process records and perform batch database inserts into relevant tables

Generated at Fri Feb 09 00:26:51 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.