Batch Importer (Bib/Acq) (UXPROD-47)

[UXPROD-3429] NFR: R2 2022 Morning Glory Data import Stability/Reliability/Performance work Created: 01/Dec/21  Updated: 26/Oct/22  Resolved: 27/Jun/22

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: Morning Glory (R2 2022)
Parent: Batch Importer (Bib/Acq)

Type: New Feature Priority: P2
Reporter: Taisiya Trunova Assignee: Ann-Marie Breaux (Inactive)
Resolution: Done Votes: 0
Labels: NFR, data-import, epam-folijet
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Continues
continues UXPROD-3210 NFR: R1 2022 Lotus Data import Stabil... Closed
continues UXPROD-3261 NFR: R1 2022 Lotus Data import perfor... Closed
Defines
is defined by MODSOURMAN-772 Timeout when importing MARC files - M... Closed
is defined by MODSOURMAN-792 Fix Race Conditions During Mapping Pa... Closed
is defined by MODSOURMAN-813 Remove JobExecutionCache to improve p... Closed
is defined by MODDATAIMP-639 Test DI performance - Morning Glory Closed
is defined by MODDICORE-254 Add logic to skip sending DI_ERROR fr... Closed
is defined by MODINV-664 Add logic to skip sending DI_ERROR fr... Closed
is defined by MODSOURCE-499 Alleviate Eventloop Blocking During B... Closed
is defined by MODSOURMAN-710 [100k-500k imports] Improve performan... Closed
is defined by MODSOURMAN-759 Improve sql query used by UI to know ... Closed
is defined by MODINVSTOR-894 Create Instance Optimization Closed
is defined by MODSOURMAN-791 Reduce Conversion of Parsed Content I... Closed
Relates
relates to UXPROD-3555 NFR: Data Import Vertx-to-Spring Migr... Open
relates to UXPROD-3471 NFR: R2 2022 Morning Glory: Implement... Closed
Release: Morning Glory (R2 2022)
Epic Link: Batch Importer (Bib/Acq)
Front End Estimate: Out of scope
Front-End Confidence factor: Medium
Back End Estimate: XXL < 30 days
Back End Estimator: Kateryna Senchenko
Development Team: Folijet
PO Rank: 115
Rank: Cornell (Full Sum 2021): R1
Rank: U of AL (MVP Oct 2020): R1

 Description   

Update the description once we know what will be left from Lotus UXPROD-3210 Closed and UXPROD-3261 Closed

Moving to Spring will help make the modules easier to maintain

Investigation required for:

  • Review folio-spring-base and how it can be used in mod-data-import
  • Define an approach for Spring transition for mod-source-record-manager and mod-source-record-storage (and the use of data-import-processing-core for Vertx-based and Spring-based modules)
  • mod-data-import stores input file in memory, limiting size of uploaded file and possibly having oom

Proposed solution/stories

  • Moving mod-data-import to use Spring
  • Improve logging to help in troubleshooting
  • Test performance after all the changes are applied

Questions



 Comments   
Comment by Ann-Marie Breaux (Inactive) [ 21/Jan/22 ]

Hi Kateryna Senchenko Could you review the BE T-shirt size for this? Is it really jumbo, or maybe smaller now?

Comment by Ann-Marie Breaux (Inactive) [ 11/Feb/22 ]

Hi Kateryna Senchenko I think I've got this one cleaned out now. Could you please:
1. Review the issues left in this feature. I pulled out all the non-mod-data-import Spring, and a few others we decided to push out. Anything else that could be pushed out to Nolana or later?
2. Do we need any stories for new or updated Karate tests?
3. Check the T-shirt size - can we reduce it from Jumbo?
4. Could you clean up the description? Or maybe we can work on it together next week

Comment by Kateryna Senchenko [ 11/Feb/22 ]

Hi Ann-Marie Breaux,

The list of tickets looks good to me, no Karate tests needed, but I'll add a story for performance testing. Put a rough estimate, and will update the description. Thank you!

Generated at Fri Feb 09 00:31:56 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.