Scripts for Inventory, Source Record Storage, and Data Import Cleanup

Owned by Ann-Marie Breaux (Deactivated)
Last updated: May 27, 2024 by Ruslan Lavrov
4 min read

 The following scripts may be used for identifying and cleaning up records in FOLIO's Source Record Storage and Inventory. More technical details are available via the title hotlink for each script.

  1. Retrieving Inventory Holdings Records and Update/Cleanup then if needed
    1. Release: Kiwi or later
    2. Release notes: Kiwi
    3. Use cases:
      1. Inventory Holdings source is anything other than FOLIO or MARC 
      2. If the Inventory Holdings source = MARC, check to see if there is a linked SRS MARC Holdings record
        1. If yes, then keep the Holdings source as MARC, and ensure that View source and Edit via QM in the Actions menu work for the Inventory Holdings record.
        2. If no, then change the Holdings source to FOLIO, and ensure that View source/Edit in QM do not display in the Inventory Holdings Action menu.
    4. Timing: On demand
    5. Related Jira: MODSOURMAN-627: Data integrity issues with instances/holdings records
    6. NOTE: These scripts do not update the Inventory Holdings. They only retrieve holdings that should be reviewed and then possibly subsequently updated
  2. Scripts for identifying and fixing invalid instanceIDs in SRS
    1. Release: Lotus or later
    2. Release notes: Lotus
    3. Use case: Inventory instance source = MARC and "View source" is not available but there is MARC Bibliographic record that has id of this instance at 999 ff $i subfield and has not in "records_lb.external_id" column
    4. Timing: On demand
    5. Related Jira: MODSOURCE-300: Create script for fixing invalid SRS instanceIDs
  3. Script to refresh Instances against an updated MARC Bib-to-Inventory Instance map
    1. Release: Kiwi or later
    2. Release notes: Lotus
    3. Use case: To refresh some or all Inventory Instances, after the library's default MARC Bib-to Inventory Instance map has been updated
    4. Timing: On demand
    5. Related Jira: MODDATAIMP-567: Script to refresh Instances against an updated MARC Bib-to-Inventory Instance map
  4. Script to delete duplicate mapping rules for each record type
    1. Release: Kiwi or later
    2. Release notes: Kiwi
    3. Use case: To clear duplicate mapping_rules records in the db 
    4. Timing: On demand
    5. Related Jira: MODSOURMAN-632: Create a script to delete duplicate mapping rules for each record type
  5. Script to identify "bad URLs" (broken links, etc.) in records linking to electronic items
    1. Release: Juniper or later (and maybe earlier... tested on Juniper)
    2. Use cases:
      1. To detect Page Not Found (404) and any other unsuccessful status code
      2. To detect permanent redirects that could be updated to the new URL to improve link stability and loading time
      3. To detect insecure (non-https) links and upgrade them to https if possible
    3. Timing: On demand
  6. Script for setting the SRS record "order" field so that Inventory Single Record Imports do not display blank log information
    1. Release: Kiwi or later
    2. Release notes: Kiwi, Lotus
    3. Use case: To set the "order" field in SRS records, so that Inventory Single Record imports will display proper information in the log
    4. Timing: On demand
    5. Related Jiras: 
      1. MODSOURMAN-422: Add file name info and record sequence number for records posted direct via API
      2. UIDATIMP-1136: No log displaying for (some? all?) of the single record imports
  7. Script to remove identifiers with invalid identifier type from instance records
    1. Release: Kiwi or later
    2. Release notes: Kiwi, Lotus
    3. Use case: Remove identifiers with invalid identifier type from instance records
    4. Timing: On demand
    5. Related Jiras:
      1. BF-318 : Remove identifiers with invalid identifier type from instance records
  8. Script to change the labels of Inventory reference values
    1. Release: Kiwi or later
    2. Use case: To change the labels (names) of reference values in the Inventory domain and keep the value's existing UUID.
    3. Timing: On demand
  9. Script to identify item records with invalid itemLevelCallNumberTypeIds
    1. Release: Nolana or later
    2. Use case: Identify records with invalid call number types so that they can be cleaned up
    3. Timing: On Demand
  10. Script to remove SRS MARC Bib records with no links to InstanceIds
    1. Release: prior to Orchid
    2. Use case: Multiple match errors for MARC-to-MARC match by 0XX or 9XX fields, except 001 or 999 ff
    3. Timing: On Demand
    4. Related Jiras:
      1. MODSOURCE-585 Data import matching takes incorrect SRS records into consideration
  11. Script to remove child and parent elements from data import profiles
    1. Release: prior to Orchid
    2. Use case: slow performance of mod-data-import-converter-storage, OOM crashes of mod-source-record-manager, RecordTooLargeException from Kafka
    3. Timing: On Demand
    4. Related Jiras:
      1. MODDICONV-293 Add validation for creating profiles to prevent saving child and parent elements
      2. UIDATIMP-1369 "parentProfiles" and "childProfiles" properties are populated on profile update
  12. Scripts to permanently delete irrelevant Data Import profiles
    1. Release: prior to Poppy
    2. Use case: a particular profile cannot be deleted from the UI, even though the associated Job Profile was deleted
    3. Timing: On Demand
    4. Related Jiras:
      1. MODDICONV-335 Action Profile can not be deleted
  13. Scripts to detect profiles with mirroring associations
    1. Release: prior to Poppy
    2. Use case: profiles with mirroring associations (invalid profiles, for which details cannot be displayed on UI) will fail to migrate to use wrappers
    3. Timing: On Demand
    4. Related Jiras:
      1. MODDICONV-331 Create a script to detect mirroring associations to run before migration to Poppy
  14. Clean up records with inconsistent matched id values
    1. Release: prior to Poppy
    2. Use case: If more than 1 ACTUAL record with the same matched_id - set status OLD to all the other records.
    3. Timing: On Demand
    4. Related Jiras:
      1. MODSOURCE-666 Create script to clean up records with inconsistent matched id values
  15. Scripts for receiving id's and wrapper id's for selected jobProfile
    1. Release: Poppy
    2. Use case: to get job profile information in case it migrated with issues
    3. Timing: After migration, on demand
    4. Related Jiras:
      1. MODDICONV-360 Investigate result of migration profiles and profiles snapshots to Poppy
  16. Script to identify invalid associations between Match profile linked to Modify Action profile
    1. Release: Quesnelia
    2. Use case: to get the unique job profile IDs in case it has a Match Profile linked to the Modify Action Profile.
    3. Timing: After migration, on demand
    4. Related Jiras:
      1. MODDICONV-375 Create script to identify invalid associations.
      2. MODDATAIMP-1008 Document scenarios for Modify used with Update action and verify the behavior
  17. Scripts to populate fund code in imported order lines and existing invoice lines
    1. Release: Orchid or later
    2. Use case: fund code is not shown in the imported order lines list, created invoice lines list, and in their CSV export
    3. Timing: On Demand
    4. Related Jiras:
      1. MODDATAIMP-1057 Create scripts to populate fund code in imported order lines and invoice lines
      2. MODORDERS-1102 Order lines search results do not display fund code for orders created by data import
      3. MODINVOSTO-173 [RRT] Invoices do not display fund codes
      4. MODDATAIMP-1045 [RRT] Fund code not appearing in invoice CSV export