Batch Importer (Bib/Acq) (UXPROD-47)

[UXPROD-3471] NFR: R2 2022 Morning Glory: Implement flow control for Data Import Created: 04/Jan/22  Updated: 29/Jun/22  Resolved: 14/Jun/22

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: Morning Glory (R2 2022)
Parent: Batch Importer (Bib/Acq)

Type: New Feature Priority: P2
Reporter: Ann-Marie Breaux (Inactive) Assignee: Ann-Marie Breaux (Inactive)
Resolution: Done Votes: 0
Labels: cornell-priority
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Defines
is defined by KAFKAWRAP-18 Add implementation pause/resume metho... Closed
is defined by MODDATAIMP-613 OCLC single record import takes exten... Closed
is defined by MODSOURMAN-661 Add new parameters to support flow co... Closed
is defined by MODSOURMAN-662 Implement flow control logic in SRM f... Closed
is defined by MODSOURMAN-663 Test Morning Glory DI performance aft... Closed
is defined by MODSOURMAN-690 SPIKE: Review impact on MDCPCT and UIIN Closed
is defined by MODSOURMAN-796 Change logic to initialize job execut... Closed
is defined by MODSOURMAN-811 Ensure proper work of flow control in... Closed
is defined by MODSOURMAN-693 Confirm how MARC Authority Deletes ar... Closed
Relates
relates to UXPROD-3210 NFR: R1 2022 Lotus Data import Stabil... Closed
relates to UXPROD-3429 NFR: R2 2022 Morning Glory Data impor... Closed
relates to UXPROD-3517 Delete MARC authority records via UI Closed
Release: Morning Glory (R2 2022)
Epic Link: Batch Importer (Bib/Acq)
Front End Estimate: Out of scope
Front-End Confidence factor: High
Back End Estimate: XL < 15 days
Back End Estimator: Kateryna Senchenko
Development Team: Folijet
PO Rank: 125

 Description   

Implement flow control as a way to ensure that Inventory Single Record Import jobs do not get stuck behind large import jobs.

For additional details, see ARCH-14 and https://folio-org.atlassian.net/wiki/display/FOLIJET/ARCH-14%3A+Investigate+opportunity+to+not+take+extended+time+for+OCLC+single+record+import+while+large+file+is+importing. Architectural review in Lotus, but implementation will not be until Morning Glory

NOTE Description of the Inventory Single Record Import process drafted in the scope of MODCPCT-38: https://docs.google.com/document/d/1LI3THKLH1C5ovrzvqzt3V6isP0Q1xTxwN0KKTh-hj8k/edit?usp=sharing

NOTE Spitfire implementing Delete MARC Authorities via UI in Morning Glory UXPROD-3517 Closed - those import jobs should also take priority over large import jobs



 Comments   
Comment by Ann-Marie Breaux (Inactive) [ 04/Jan/22 ]

Hi Kateryna Senchenko We need a high level BE estimate for this feature, which we've taken out of Lotus Stability/Reliability general feature and moved to Morning Glory. Could you provide ASAP?

Comment by Kateryna Senchenko [ 04/Jan/22 ]

Hi Ann-Marie Breaux, I put XL size for this feature taking into account all the necessary performance testing after implementation

Comment by Ann-Marie Breaux (Inactive) [ 04/Jan/22 ]

Sounds good - thanks, Kateryna Senchenko. Do you think we need any other stories or tasks for this feature, or can we consider it fully groomed at this point?

Comment by Kateryna Senchenko [ 04/Jan/22 ]

I'd consult with Former user once he is back from vacation, from my point of view we are fine, but we can miss something

Comment by Kateryna Senchenko [ 25/Jan/22 ]

Hi Ann-Marie Breaux, discussed this feature with Former user and BE guys and we consider this feature refined. No Karate tests required - existing basic DI functionality is already covered, and this change might only impact the performance (OCLC imports should be faster, while main DI import might slow down). To test the performance we have added MODSOURMAN-663 Closed . We don't have an approach to automate performance testing yet, therefore we only plan to test performance manually in scope that ticket.

Comment by Ann-Marie Breaux (Inactive) [ 25/Jan/22 ]

Thanks, Kateryna Senchenko That is great to hear. Our first refined MG feature!

I talked some more with Hkaplanian and Khalilah Gambrell about this. Would it be possible to add one more task? It might be an ARCH, or a task after the Folijet work is done. We'd like someone to review the MODCPCT and UIIN side of Single Record Import and advise if there are any additional changes there (in the process or in the UX) to improve the speed and reliability of the Single record imports, display of created/updated instances, and Inventory UI toasts. Would it be possible to add that? And let's make sure it's time-boxed to just a couple days, at most.

Comment by Jenn Colt [ 01/Feb/22 ]

Since our migration to Juniper we are seeing more instances of this from mod-copycat:

io.vertx.core.impl.NoStackTraceThrowable: Add record for job execution failed: POST http://pvt.lb.jcnl.folio-eis.us-east-1:9130/change-manager/jobExecutions/414d7f5e-3c47-41c2-85e3-9671c080a32d/records: Connection was closed

 

Will the changes made under this story help with this or does it need a separate bug? It seems to occur when we have a large job running and then someone tries to bring in a single record.

 

Comment by Ann-Marie Breaux (Inactive) [ 01/Feb/22 ]

Hi Jenn Colt Good question. Kateryna Senchenko or Vladimir Shalaev, could you review the above comment and respond? And please let me know if we should perhaps add a bug or task in this feature to check for this mod-copycat error once the dev work is done. Thank you!

Comment by Kateryna Senchenko [ 03/Feb/22 ]

Hi Ann-Marie Breaux, added MODSOURMAN-690 to check the mod-copycat and ui-inventory side after flow control is implemented.

As for the question from Jenn Colt - it looks like an unrelated issue and should be investigated separately.

Thank you!

Comment by Ann-Marie Breaux (Inactive) [ 11/Feb/22 ]

Hi Kateryna Senchenko One more question about this feature - will we need any new or adjusted Karate tests?

Comment by Kateryna Senchenko [ 11/Feb/22 ]

Hi Ann-Marie Breaux, I don't think we'll need additional Karate tests, checking the main functionality with existing tests should suffice. Thank you!

Comment by Ann-Marie Breaux (Inactive) [ 23/Mar/22 ]

Hi Kateryna Senchenko and Vladimir Shalaev Please see Khalilah Gambrell's comments on MODSOURCE-482 Closed . Is this feature still needed if the Inventory Single Record Import "creates" are faster now? I'm thinking probably yes, but I'm not sure. To make this feature unnecessary means that
1. We need to ensure that Inventory Single Record Imports (creates or updates) do not get stuck behind long-processing import jobs
2. We need to ensure that MARC Authority deletes (which are individual record deletes, triggered from the MARC Authority app UI) do not get stuck behind long-processing import jobs. These were not taken into account when the feature was first scoped, but may be necessary to provide a decent UX. See UXPROD-3517 Closed .

Comment by Vladimir Shalaev [ 23/Mar/22 ]

IMO this ticket is solving not the general performance of OCLC import issues, but parallel jobs - big data import and OCLC

Comment by Ann-Marie Breaux (Inactive) [ 01/Apr/22 ]

Good to know - thanks, Vladimir Shalaev.

Kateryna Senchenko Do we need to add a story to account for MARC Authority deletes? These are individual record deletes, triggered from the MARC Authority app UI, and we need to make sure that they do not get stuck behind long-processing import jobs. These were not taken into account when the feature was first scoped, but may be necessary to provide a decent UX. See UXPROD-3517 Closed for more details. Or maybe Spitfire should add a testing/review task to their UXPROD-3517 Closed ?

cc: Khalilah Gambrell

Generated at Fri Feb 09 00:32:16 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.