Optimize Insert & Update of marc_records_lb (Orchid CSP 5 Clone)

Description

This is meant to help curb impact found in https://folio-org.atlassian.net/browse/MODSOURMAN-565#icft=MODSOURMAN-565. It was imperative to maintain the data integrity and interfaces so that this change has the potential to be deploy to various FOLIO versions and minimize potential for regressions.

ORCHID Critical service patch details

  1. Describe issue impact on business: Process of saving records in SRS needs less resources than previously, and is now more stable with these infrastructure changes. This reduces significant portion of total processing time for an import

  2. What institutions are affected? (field “Affected Institutions” in Jira to be populated): All institutions using SRS and Data Import

  3. What is the workaround if exists? None; continue with existing Orchid SRS infrastructure

  4. What areas will be impacted by fix (i.e. what areas need to be retested): Smoke; critical path Data import work as expected

  5. Brief explanation of technical implementation and the level of effort (in workdays) and technical risk (low/medium/high): This was a significant set of changes made by , updating the process for saving records in SRS.
    From the PR Request: Lots of resources are spent during insert and update of MARC_RECORDS_LB table. This is because there are triggers on the table that translate the JSONB content into rows and columns that are suitable for searching, MARC_INDEXERS. When the JSONB content is updated in MARC_RECORDS_LB, data in MARC_INDEXERS is deleted and re-inserted with the latest values.
    The delete statement on MARC_INDEXERS is expensive for the database to plan because this table has 1000 partitions. Also, update of a single property, any property in the JSONB content of MARC_RECORDS_LB results in computing the almost the same data twice.
    This change add the notion of "versions" to MARC_INDEXERS. Each update of MARC_RECORDS_LB would cause inserts into MARC_INDEXERS all the time. This bypasses needed to plan the expensive delete statement for every row. Deletion is now performed by a background job. This also means queries that utilize MARC_INDEXERS had to be updated to filter out older versions.
    Technical risk: low, though good testing should be done.

  6. Brief explanation of testing required and level of effort (in workdays). Provide test plan agreed with by QA Manager and PO: After the MODSOURCE and MODSOURMAN patches are applied, we need to retest the Smoke and Critical Path Data Import tests (most of which are automated), and perhaps selected Extended Manual tests. Manual testing across these MODSOURCE and MODSOURMAN changes are likely 3-5 days of work for manual QA, plus some input from PO.

  7. What is the roll back plan in case the fix does not work? Revert to previous version

Environment

None

Potential Workaround

None

CSP Request Details

Nolana/Orchid CSP requested 21 June 2023 Approved 22 June 2023 by Khalilah, Mike G, Kristin M, Mark V, Debra H, Harry K Changes reverted 14 July 2023 due to migration issues; reviewing and will re-submit

CSP Approved

Yes

CSP Rejection Details

None

Confluence content

mentioned on

Checklist

hide

TestRail: Results

Activity

Show:

Ann-Marie Breaux August 16, 2023 at 3:02 PM

Automated tests for Orchid all passed. Closing this issue.

Kateryna Senchenko August 10, 2023 at 4:29 PM

Manual QA effort is not required for this ticket

Kateryna Senchenko August 3, 2023 at 1:29 PM

Hi , yes - this is still the plan, Ruslan is currently working on https://folio-org.atlassian.net/browse/MODSOURCE-671#icft=MODSOURCE-671 to allow search on marc fields while migration is still in progress.

Ann-Marie Breaux August 1, 2023 at 4:11 PM

Hi and My understanding is that we're trying to make this back-portable to Orchid (if we can keep it from requiring lots of downtime). Could you confirm if that is still the plan?

Khalilah Gambrell July 28, 2023 at 2:27 PM

Hey – Should this be closed as Won't Do for Orchid CSP?

Done

Details

Assignee

Reporter

Priority

Story Points

Development Team

Folijet

Fix versions

Release

Orchid (R1 2023) Service Patch #5

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs

Created June 27, 2023 at 2:19 AM
Updated August 16, 2023 at 3:02 PM
Resolved August 11, 2023 at 1:38 PM
TestRail: Cases
TestRail: Runs

Flag notifications