Review the Spitfire app that batch updates MARC Authority records based on default mapping updates: FOLIO-3659

Start with Spike FOLIO-3681 to assess the work to develop and test, and create the required stories

Would replace script 3 on https://folio-org.atlassian.net/wiki/display/FOLIOtips/Scripts+for+Inventory%2C+Source+Record+Storage%2C+and+Data+Import+Cleanup, which works, but is too time-consuming

Tenant could run the script to adjust Instances whenever the default MARC Bib-to-Inventory Instance mapping rules are updated. We could test it on Bugfest, since many istances have data that diverges from current BF default mappings.

Approach
Create a java app that has to do the update work:
1. Create data-import job profile for MARC-to-MARC matching by 999ff$s subfield and update bibliographic record. Job/Match/Action/Mapping profiles will be hidden

• Job Profile: Release Upgrade - Migrate MARC bibliographic records 

• Action Profile: Release Upgrade - Migrate MARC bibliographic records 

• Match Profile: Release Upgrade - Migrate MARC bibliographic records 

• Mapping Profile: Release Upgrade - Migrate MARC bibliographic records

2. Load bibliographic records by pages of 50k (should be configurable) from SRS (GET /source-storage/records?recordType=MARC_BIB&state=ACTUAL&offset=<P>&limit=<N>

3. Verify that createdDate & updatedDate of records are older that time of script launch (If new records was detected - update totalRecords)
4. Prepare mrc file or JSON payload (README)
5. Initialize data-import job
6. Wait until the data-import job finished (GET /change-manager/jobExecutions/<id> and check status of job)
7. Load the next page and repeat 3-6 until there are no authority records left.
8. Delete the job profile created in 1st step
9. Logging should exist to indicate N of batch that is in progress now.

Validate that a user can view Job status

• job status is shown on the Data Import Logs UI

• For each record update, user can view SRS/MOD-Inventory-Storage output

Documentation 

• Instructions must be provided to Hosting providers/System administrators for using the standalone application 

  • Must consider that some libraries have already upgraded to Morning Glory weeks/months before this implementation. 

  • Include a note that this should be run off-hours

• Release notes for Morning Glory and Nolana should be updated and include link to Instructions 

Testing - MORE details to discuss

• Need a story for PTF

• Need an environment(s) to test 

  • Upgrade Lotus > Morning Glory   

    • Work with FSE? 

  • Upgrade Morning Glory > Nolana

    • Work with FSE?   

• Custom mapping profiles need to be accounted for in testing

Discussed 20 Feb 2023 in context of Poppy development planning:

• Spitfire is testing the tool for updating MARC Authority records after mapping updates on different clients

• All blame the performance of DI, but PTF env – able to run in 50K chunks, which is not performant in production; that led to stuck imports and errors in production envs; reduced chunks to 20K; now recommending 5K chunks (running sequentially); better to run in smaller chunks

• Spitfire will include Kate and Olamide in future meetings about the Authority refresh

• Make sure there is good documentation from Spitfire

• Don’t implement for Bibs until it’s more performant; consider for Quesnelia