2022-03-09 Data Import Subgroup meeting

Owned by Ann-Marie Breaux (Deactivated)
Last updated: Mar 16, 2022
5 min read

Recordings are posted Here (2022+) and Here (pre-2022)                   Slack channel for Q&A, discussion between meetings

Requirements details Here                                                                    Additional discussion topics in Subgroup parking lot


Attendees: Ann-Marie Breaux (Deactivated)   leeda.adkins@duke.edu  Monica Arnold Nick Cappadona Jennifer Eustis Christie Thomas Jenn Colt Lisa McColl Raegan Wiechert Taylor Smith Timothy Watters 

Lotus

Morning Glory

Agenda topics: Lotus Bugs

    • Lotus Bugfest:
      • Sign up for tests this week
      • Ann-Marie still retiring tests that have been replaced by Automated end-to-end tests
      • Ann-Marie still writing additional tests that will be available for signup this week and next week
      • Set up your own logon
    • Create jobs without match profiles that include records with 999 ff fields cause errors in the srs-instance relationship
      • What is preferred action? Option 1
        • Wipe out the 999 ff and replace with a new one with new Instance and SRS UUIDs? And maybe move the 999 ff to some other field?
          • First step - wipe out the 999 ff's in general?
            • If update, the 001 is protected
            • If create, incoming 001 is moved down to 035 and assign new HRID
        • Fail the import since there is no match action (and advice the user to remove the 999 ff field, then try importing again)
          • Only fail the affected records, since the 999 ff may not be in every record in the file
          • Have a clear error message that explains there's a create action, but there's a 999 (regardless of whether there is a match or not)
            • Instruct user to adjust profile or file
          • Clarify - will the incoming record still be parked in SRS (but not marked as the current version?) - if so, what state are they in? Need an example from the devs

    • HRID Settings: is there ever a use case to change these after initial migration?

      • Yes: define the specific use case, test it in Lotus BF, update the bug
      • No: close the bug as won’t do; retire the manual TestRail
        • No ongoing use case identified, except when new inventory record types become available
        • Already has a separate permission from all other Inventory settings; important to keep the setting locked down for most users
        • Obsolesce the TestRail case
        • Make sure this doesn't limit the ability of multiple libraries working within the same tenant (which should be controlled more with user settings, like acq group) than with numeric runs

    • Completed job appears at top of log, then disappears, then reappears

      • Is this still happening in Kiwi production environments?
      • Jenn Colt/Lisa M: Kiwi - it has stopped, but only because the query to retrieve the logs for the landing page has slowed down so much, that it's hard to know it it's fix 
        • Yes: document and assign the bug to Morning Glory
          • See if we can reproduce in PTF or Folijet Perf env, with a mix of big and little imports, EDIFACT and MARC
        • No: close the bug

    • MARC field protections apply to MARC modifications when they should not

      • Agree on preferred action - only applies to MARC records that have existing SRS (since field protections have no effect on newly created MARC records)

      • If modifications to the incoming MARC record (add or change or remove some fields)
        • Take those modifications into account, and then apply the field protections (MARC updates)
        • Take the field protections into account (MARC updates), and then apply the modifications - doesn't make sense, since modifications act on the incoming record, whereas field protections are focused on the existing record first, and then comparing the incoming record
        • Vary the logic depending on whether the MARC modification or MARC update (field protection) is first in the job profile sequence
        • What about when the modifications are after the update? Will not be used as frequently as modifications at the beginning? Should we have a test case/job profile that covers these
      • Review use cases in the comments - build a few draft tests in Lotus BugFest and have SMEs add the details?
  • START HERE NEXT WEEK 

    • Extend MatchValueLoader implementations to allow filtering according to Qualifiers and MatchCriteria:

    • Identifier matching should allow for qualifiers, compare part, and match criteria

      • Are there any specific match use cases that you want to use that you cannot (NOT MARC-MARC right now; that's next)

      • Any qualifier/begins/contains matches that are not working but that are needed?

    • MARC-MARC matching
      • Lotus: Allows for any field in a MARC record except
      • Are these needed in Morning Glory?
        • Matching for 100-899 fields? (I think they work, but not heavily tested yet)
        • Repeatable fields (e.g. 024, 035)
          • Incoming record: Only first version of the field is considered (doublechecking with the dev on whether it's the first field that has the requested indicator(s) and/or subfield, or just the first field, regardless of indicators/subfield)
          • If it takes Ind 1, Ind 2, Subfield into account (in addition to the data)
            • Does FOLIO need to check all incoming 024s against all 024s in the existing SRS records? Or just the first?
        • Wildcards for Ind 1, Ind 2, Subfield (repeatable or non-repeatable fields)
          • Needed?
    • Create a script for libraries to refresh Instances against an updated MARC-Instance map
      • What kind of result do we expect after script? (current->expected)
      • What should we do about different releases installed for different libraries?
      • Do we need to upgrade script that Ian Walls wrote or we will need to write another one to update many records at a time?
      • What is the scope of the script? All the instances (even updated ones) or just not updated ones?


Chat: 

From Christie Thomas (she/her) to Everyone 01:28 PM
+1 Jenn!

From Jenn Colt to Everyone 01:36 PM
It started in juniper

From Lisa McColl to Everyone 01:47 PM
Same here

From Leeda Adkins to Everyone 01:54 PM
Both of those are very good, common scenarios: modification on create to prepend the proxy, and modification on update to delete incoming extra 856s

From Jenn Colt to Everyone 01:56 PM
We want to move our mods into folio

From Jennifer Eustis (she/her) to Everyone 01:56 PM
We do both - before Aleph and then also in Aleph
One common after modification is that we remove the 856 with our proxy that we put in the holdings record

From Jenn Colt to Everyone 01:58 PM
That second set would be updates not modifications I think?

From Jennifer Eustis (she/her) to Everyone 01:58 PM
You're right Jenn. Removing the 856 from the bib is a global bulk update that we do to the bib records

From Christie Thomas (she/her) to Everyone 01:59 PM
but is it the incoming marc file or the existing marc file?

From Jenn Colt to Everyone 01:59 PM
*documentation wishes*

From Lloyd (Marmot Library Network) to Everyone 01:59 PM
I have to go to another meeting.

From Jennifer Eustis (she/her) to Everyone 02:00 PM
For me here, the Aleph global update is on the existing record that was either created or updated

From Christie Thomas (she/her) to Everyone 02:05 PM
I can add the 856 use case.

From Lisa McColl to Everyone 02:06 PM
slack channel?