Change the way of saving instances ids for reducing the database load

Description

Purpose/Overview:
Currently, the "instances " table contains the "json" column that stores the JSON with some metadata like updated date and deletedRecordsSupport value and a couple of other fields.
From this JSON object, only an "id" and "suppressFromDiscovery " are required for business logic. The JSON file takes about 40% of all record size.
It is required to remove this column since id is available in "instance_id" column and the "suppressFromDiscovery" value can be moved to the new column.
This will reduce the request size by 37-39%% percent.

Requirements/Scope:

  1. The "json" column from instances table removed

  2. The "suppressFromDiscovery" field is stored in a new column with Boolean type

Acceptance criteria:

  • The "json" column removed from instances table

  • Migration scripts are provided

System testing:

need access to DB.

 

Regression testing:

MMZ: The initial harvest has been run three times in the kiwi-bugfest environment after the hotfix was deployed on March 4th, March 5th and March 6th. Each harvest takes ~16 hours and harvests 8,057,158 records as reported in the bugfest channel. No regression found. Performance and consistency has been improved.

For the smoke test please use tests that are less taxing on the system:
C13784 List records in marc21_withholdings format with start and end date

Environment

None

Potential Workaround

None

Attachments

2

Checklist

hide

TestRail: Results

Activity

Show:

Viachaslau Khandramai March 11, 2022 at 12:27 PM

Migration script was applied because new version of module is enabled on Kiwi bugfest. Schema changes were applied.

Denis March 1, 2022 at 5:37 PM

Deployed to Kiwi Bugfest. Please proceed with verification

Aliaksei Harbuz January 13, 2022 at 7:13 AM

The mod-oai-pmh module responses after changes for database schema added. Verification at folio-testing env :

1) Import the 20150109.mrc with create "Default - Create instance and SRS MARC Bib" profile

2) Execute request: 

The response is provided: 

 

Done

Details

Assignee

Reporter

Priority

Story Points

Sprint

Development Team

Firebird

Fix versions

Release

Kiwi (R3 2021) Hot Fix #2

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs
Created December 10, 2021 at 12:12 PM
Updated March 11, 2022 at 12:44 PM
Resolved January 20, 2022 at 10:17 PM
TestRail: Cases
TestRail: Runs