Optimize Insert & Update of marc_records_lb (Orchid CSP 5 Clone)
Description
Environment
None
Potential Workaround
None
CSP Request Details
Nolana/Orchid CSP requested 21 June 2023
Approved 22 June 2023 by Khalilah, Mike G, Kristin M, Mark V, Debra H, Harry K
Changes reverted 14 July 2023 due to migration issues; reviewing and will re-submit
CSP Approved
Yes
CSP Rejection Details
None
blocks
clones
defines
improves
is blocked by
relates to
Checklist
hideTestRail: Results
Activity
Show:
Ann-Marie Breaux August 16, 2023 at 3:02 PM
Automated tests for Orchid all passed. Closing this issue.
Kateryna Senchenko August 10, 2023 at 4:29 PM
Manual QA effort is not required for this ticket
Kateryna Senchenko August 3, 2023 at 1:29 PM
Hi @Ann-Marie Breaux, yes - this is still the plan, Ruslan is currently working on https://folio-org.atlassian.net/browse/MODSOURCE-671#icft=MODSOURCE-671 to allow search on marc fields while migration is still in progress.
Ann-Marie Breaux August 1, 2023 at 4:11 PM
Hi @Olamide Kolawole and @Kateryna Senchenko My understanding is that we're trying to make this back-portable to Orchid (if we can keep it from requiring lots of downtime). Could you confirm if that is still the plan?
Khalilah Gambrell July 28, 2023 at 2:27 PM
Hey @Ann-Marie Breaux – Should this be closed as Won't Do for Orchid CSP?
Done
Details
Assignee
Ruslan LavrovRuslan LavrovReporter
Olamide KolawoleOlamide KolawolePriority
P2Story Points
0Development Team
FolijetFix versions
Release
Orchid (R1 2023) Service Patch #5TestRail: Cases
Open TestRail: CasesTestRail: Runs
Open TestRail: Runs
Details
Details
Assignee
Ruslan Lavrov
Ruslan LavrovReporter
Olamide Kolawole
Olamide KolawolePriority
Story Points
0
Development Team
Folijet
Fix versions
Release
Orchid (R1 2023) Service Patch #5
TestRail: Cases
Open TestRail: Cases
TestRail: Runs
Open TestRail: Runs
Created June 27, 2023 at 2:19 AM
Updated August 16, 2023 at 3:02 PM
Resolved August 11, 2023 at 1:38 PM
TestRail: Cases
TestRail: Runs
This is meant to help curb impact found in https://folio-org.atlassian.net/browse/MODSOURMAN-565#icft=MODSOURMAN-565. It was imperative to maintain the data integrity and interfaces so that this change has the potential to be deploy to various FOLIO versions and minimize potential for regressions.
ORCHID Critical service patch details
Describe issue impact on business: Process of saving records in SRS needs less resources than previously, and is now more stable with these infrastructure changes. This reduces significant portion of total processing time for an import
What institutions are affected? (field “Affected Institutions” in Jira to be populated): All institutions using SRS and Data Import
What is the workaround if exists? None; continue with existing Orchid SRS infrastructure
What areas will be impacted by fix (i.e. what areas need to be retested): Smoke; critical path Data import work as expected
Brief explanation of technical implementation and the level of effort (in workdays) and technical risk (low/medium/high): This was a significant set of changes made by @Olamide Kolawole, updating the process for saving records in SRS.
From the PR Request: Lots of resources are spent during insert and update of MARC_RECORDS_LB table. This is because there are triggers on the table that translate the JSONB content into rows and columns that are suitable for searching, MARC_INDEXERS. When the JSONB content is updated in MARC_RECORDS_LB, data in MARC_INDEXERS is deleted and re-inserted with the latest values.
The delete statement on MARC_INDEXERS is expensive for the database to plan because this table has 1000 partitions. Also, update of a single property, any property in the JSONB content of MARC_RECORDS_LB results in computing the almost the same data twice.
This change add the notion of "versions" to MARC_INDEXERS. Each update of MARC_RECORDS_LB would cause inserts into MARC_INDEXERS all the time. This bypasses needed to plan the expensive delete statement for every row. Deletion is now performed by a background job. This also means queries that utilize MARC_INDEXERS had to be updated to filter out older versions.
Technical risk: low, though good testing should be done.
Brief explanation of testing required and level of effort (in workdays). Provide test plan agreed with by QA Manager and PO: After the MODSOURCE and MODSOURMAN patches are applied, we need to retest the Smoke and Critical Path Data Import tests (most of which are automated), and perhaps selected Extended Manual tests. Manual testing across these MODSOURCE and MODSOURMAN changes are likely 3-5 days of work for manual QA, plus some input from PO.
What is the roll back plan in case the fix does not work? Revert to previous version