Batch Importer (Bib/Acq) (UXPROD-47)

[UXPROD-3261] NFR: R1 2022 Lotus Data import performance work Created: 31/Aug/21  Updated: 01/Mar/23  Resolved: 12/Apr/22

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: Lotus (R1 2022)
Parent: Batch Importer (Bib/Acq)

Type: New Feature Priority: P2
Reporter: Taisiya Trunova Assignee: Ann-Marie Breaux (Inactive)
Resolution: Done Votes: 0
Labels: NFR, cornell-priority, data-import, epam-folijet, performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Continues
continues UXPROD-3191 NFR: R3 2021 Kiwi Data import perform... Closed
is continued by UXPROD-3429 NFR: R2 2022 Morning Glory Data impor... Closed
Defines
defines UXPROD-47 Batch Importer (Bib/Acq) Analysis Complete
is defined by MODDICONV-217 Kiwi R3 2021 - Log4j vulnerability ve... Closed
is defined by MODDATAIMP-557 Close Vertx Kafka producers that have... Closed
is defined by MODDATAIMP-588 Spike: Investigate possibility of rem... Closed
is defined by MODDATAIMP-623 Remove Kafka cache initialization and... Closed
is defined by MODINV-444 Spike: CPU spikes related to events_c... Closed
is defined by MODINV-642 [Perf only] Inventory-storage could n... Closed
is defined by MODINVSTOR-878 Upgrade RMB/Vertx versions that conta... Closed
is defined by MODSOURCE-421 Add possibility to adjust parameter D... Closed
is defined by MODSOURCE-422 Upgrade Vertx from 4.0.0 to 4.1.2 (Ju... Closed
is defined by MODSOURCE-462 [Perf only] SRS container restarted d... Closed
is defined by MODSOURCE-464 [Perf only] SRM container restarted d... Closed
is defined by MODSOURMAN-616 Optimize DI landing page query Closed
is defined by MODSOURMAN-617 DI landing page: Push notifications i... Closed
is defined by MODSOURMAN-638 Remove Kafka cache by handling Constr... Closed
is defined by MODSOURMAN-639 Improve performance of saving journal... Closed
is defined by MODSOURMAN-640 Remove Kafka cache by handling Constr... Closed
is defined by MODSOURMAN-641 Remove Kafka cache by handling Constr... Closed
is defined by MODSOURMAN-668 Restructure job_execution_progress ta... Closed
is defined by MODSOURMAN-712 Timeout when importing MARC files - L... Closed
is defined by RMB-901 Add ability to set reconnectAttempts ... Closed
is defined by MODDATAIMP-580 SPIKE: Investigate Connection Pool Wa... Closed
is defined by MODDATAIMP-602 Investigate possibilities to remove K... Closed
is defined by MODDATAIMP-603 Investigate possibilities to remove K... Closed
is defined by MODDATAIMP-622 Test DI performance with Lotus improv... Closed
is defined by MODDATAIMP-644 Test DI performance with Lotus improv... Closed
is defined by MODDICORE-253 Add logic to skip sending DI_ERROR fr... Closed
is defined by MODINV-401 Spike: KafkaCache Memory not released... Closed
is defined by MODINV-598 Remove Kafka cache from update handlers Closed
is defined by MODINV-603 Remove Kafka cache by handling Constr... Closed
is defined by MODINV-646 Add logic to skip sending DI_ERROR fr... Closed
is defined by MODSOURCE-420 Add possibility to adjust parameter c... Closed
is defined by MODSOURCE-461 Upgrade RMB and Vertx versions that c... Closed
is defined by MODSOURCE-467 Reduce amount of logging of KafkaProd... Closed
is defined by MODSOURMAN-468 Create a new API and db table that sh... Closed
is defined by MODSOURMAN-469 Change data-import progress mechanism... Closed
is defined by MODSOURMAN-630 Test and merge PRs on changes for DI ... Closed
is defined by MODSOURMAN-631 Create a migration script for job_exe... Closed
is defined by MODSOURMAN-644 Add the required index: job_execution... Closed
is defined by MODSOURMAN-694 Improve sql query for retrieving job... Closed
is defined by MODSOURMAN-695 Upgrade RMB and Vertx versions that c... Closed
is defined by MODSOURMAN-699 Fix Can`t map 'RECORD' or/and 'MARC_... Closed
is defined by MODSOURMAN-700 SQL timeouts in SRM connected with ca... Closed
is defined by MODSOURMAN-751 Improve sql query used by UI to know ... Closed
is defined by UIDATIMP-918 Use new API for DataImport landing page Closed
is defined by UIDATIMP-1069 Use new API for DataImport ViewAll page Closed
is defined by UIPFIMP-45 Update 'source-manager-job-executions... Closed
Relates
relates to MODSOURMAN-622 DI Job marked as completed too soon? Closed
Release: Lotus R1 2022
Epic Link: Batch Importer (Bib/Acq)
Front End Estimate: Out of scope
Front-End Confidence factor: Medium
Back End Estimate: Jumbo: > 45 days
Development Team: Folijet
PO Rank: 124

 Description   

Team estimation - 45 days

UXPROD-3135 Closed was split into UXPROD-3193 Closed for stability and reliability and UXPROD-3191 Closed for performance; Ann-Marie Breaux to close UXPROD-3135 Closed once all issues moved from it to the new features

Current situation or problem:

1.Kafka producer closed after sending

2.WARN message when no handler found

3. Kafka cache resource consumption
4. Data import impacts other processes

5. High resource consumption to get job(s) status/progress

NOTE: Several stories in this feature are related to Spring; those may move to Morning Glory; need input from architects on whether Spring Kafka Template can be use in DI modules. Probably will need an architecture spike in Lotus, and then the Spring stories may move to Morning Glory

Proposed solution/stories
*1.*Create pool of active producers. Start pool on module launch, close on shutdown. Reuse connections. Add max/min pool sizes.
2. Do not subscribe to messages you're not going to process OR Lower log lever for this type of messages.

*3.*Remove Kafka cache. Modules that do not do persistent changes will sometimes (on duplicates read) do unnecessary calls. Can be optimized further upon adding distributed in-memory cache (ex hazelcast) (blocked by 6 PUT LINK TO FEATURE in p.6)
4. SPIKE REQ.: Need investigation (possible solution - configure rate limiter). Relates to High CPU/Memory consumption on modules
5. Add some kind of caching for progress tracking (database or in-memory)

Links to additional information:

Data Import Stabilization plan - Vladimir Shalaev - FOLIO Wiki

Questions



 Comments   
Comment by Ann-Marie Breaux (Inactive) [ 01/Dec/21 ]

Kateryna Senchenko please add an ARCH story here for Vladimir Shalaev, so that we can have the Spring details sorted in Lotus, and then perhaps work on the implementation for Morning Glory

Comment by Kateryna Senchenko [ 07/Dec/21 ]

Hi Ann-Marie Breaux, created ARCH-10 and blocked our implementations stories, do you mind if I move them to Morning Glory?

Comment by Ann-Marie Breaux (Inactive) [ 07/Dec/21 ]

Hi Kateryna Senchenko A few questions:

  1. Is there any risk/downside to moving this out to Morning Glory?
  2. Do we know this change will provide some benefits to users, or is that not clear yet?
  3. If we move it out, we'll need to create a Performance feature for Morning Glory, and put a T-shirt estimate on it - and... I just realized we already have that feature and it's linked above!
  4. And if we need an architecture decision or analysis before moving forward, we'll want those details finalized well before we start Morning Glory development. I'll add a note on ARCH-10 that we'd like it completed before mid-February.
Comment by Ann-Marie Breaux (Inactive) [ 12/Jan/22 ]

Hi Kateryna Senchenko Do we have a story or task to measure performance after all of the changes in this feature, and update the file size metrics and recommendations? If yes, what's the Jira number, and do we expect that PTF or Folijet will do that assessment work? If no, can we add a story or task? We have a couple blocked stories on this feature that may not be needed, but we have to have a solid understanding of the performance gains that have been made before we decide on doing them or closing them. And of course the community will also be eager to have any updated performance info. Thank you!

Comment by Kateryna Senchenko [ 14/Jan/22 ]

Hi Ann-Marie Breaux, that's a good point, we've been talking for a while now with tech guys regarding procedure of performance testing - we'd like to be able to test it ourselves using Rancher perf env and maybe writing some JMeter tests. As for Lotus improvements - I created MODDATAIMP-622 for now - we'll try to measure performance ourselves on Rancher perf env, and might involve PTF team as well. It would be nice though to start working on it right after code freeze, before the releases, so that we had some buffer for unexpected things (PTF usually starts testing at the time around bugfest).

Generated at Fri Feb 09 00:30:42 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.