2022-03-16 Data Import Subgroup meeting

Owned by Ann-Marie Breaux (Deactivated)
Last updated: Mar 23, 2022
5 min read

Recordings are posted Here (2022+) and Here (pre-2022)                   Slack channel for Q&A, discussion between meetings

Requirements details Here                                                                    Additional discussion topics in Subgroup parking lot


Attendees: Ann-Marie Breaux (Deactivated)  Timothy Watters  leeda.adkins@duke.edu Jennifer Eustis Christie Thomas Lisa McColl Monica Arnold Lloyd Chittenden 

Lotus

Morning Glory

Agenda topics:

Environment for Import/Export/quickMARC Labs - Confirm the following

  • Keep tied to the most recent released version? (e.g. Kiwi with HF1-2)
  • Environment will not exceed ca. 50K instances in Inventory
  • Purging will happen on-demand. When purging, purge Inventory and SRS data, Import/Export logs, but DO NOT purge any reference data (including all Import/Export profiles)
  • Include all the default ref data (like the organizations, funds, etc)
  • See if we can include the flower and which HF version on the homepage (see Chicago homepage screenshot)
  • If SMEs have any issues, start with Ann-Marie Breaux (Deactivated). Dev team managing the environment is Kitfox.

Lotus Bugs

    • Lotus Bugfest:
      • Testing began this week; any issues? 
      • Ann-Marie still writing and decommissioning additional tests that will be available this week and next week
      • Set up your own logon

      • MARC field protections apply to MARC modifications when they should not

        • Ann-Marie: build a few draft tests in Lotus BugFest and have SMEs add the details

        • Job 1
          • Incoming MARC Bib has 981$d and updated cataloging data 
          • Modify SRS MARC to edit something
          • Update MARC Bib (protect Chicago's existing 856)
          • Match on POL
            • Update Instance with cat date, status
          • Match on POL
            • Update Holdings
          • Match on POL
            • Update Item
          • Update MARC Bib to override protections (only if field/subfield to be deleted is in your protections)
          • Modify SRS MARC to delete 981$d
        • Job 2
          • Incoming MARC Bib has 981$d and updated cataloging data 
          • Update MARC Bib (protect Chicago's existing 856)
          • Modify SRS MARC to edit something
          • Match on POL
            • Update Instance with cat date, status
          • Match on POL
            • Update Holdings
          • Match on POL
            • Update Item
          • Update MARC Bib to override protections (only if field/subfield to be deleted is in your protections)
          • Modify SRS MARC to delete 981$d
    • Create a script for libraries to refresh Instances against an updated MARC-Instance map
      • First iteration this week; get it to do the right thing, then work on speed
      • Script would be deployed on demand
      • Would act on all instances with Source = MARC
      • Release version would not matter
      • Any other questions or requirements?
    • MARC Field protection:
      • Can we assume that LDR, 002-009 fields would never need field protection
      • If any do need field protection (maybe 006 or 007), can we always assume * for data? (Ind1, Ind2, Subfield n/a)
      • Talk with MM SIG about it tomorrow
  • START HERE NEXT WEEK 
    • Record matches are not decreased when additional match conditions are added to a job profile
      • See example in bug; Devs need additional examples of multi-tiered matches
      • Make first match as specific as possible (retrieve no more than 90 results), then secondary match to narrow that further
      • Kiwi release notes: add as known issue
      • Lotus release notes: add that this is partially addressed in Lotus fix, will aim to change/remove restriction in Morning Glory

    • Extend MatchValueLoader implementations to allow filtering according to Qualifiers and MatchCriteria:

    • Identifier matching should allow for qualifiers, compare part, and match criteria

      • Are there any specific match use cases that you want to use that you cannot (NOT MARC-MARC right now; that's next)

      • Any qualifier/begins/contains matches that are not working but that are needed?

    • MARC-MARC matching
      • Lotus: Allows for any field in a MARC record except
      • Are these needed in Morning Glory?
        • Matching for 100-899 fields? (I think they work, but not heavily tested yet)
        • Repeatable fields (e.g. 024, 035)
          • Incoming record: Only first version of the field is considered (doublechecking with the dev on whether it's the first field that has the requested indicator(s) and/or subfield, or just the first field, regardless of indicators/subfield)
          • If it takes Ind 1, Ind 2, Subfield into account (in addition to the data)
          • Does FOLIO need to check all incoming 024s against all 024s in the existing SRS records? Or just the first?
          • Wildcards for Ind 1, Ind 2, Subfield (repeatable or non-repeatable fields)
            • Needed?
      • Additional info from A-M/Igor:

      • Let's pretend that these fields are in an incoming record: (Field Ind1 Ind2 Subfield)

        • 024 _ _ $a 12345
          024 1 1 $a 45678
          024 1 _ $x 67890
          024 2 2 $x 67890

      • And the fields in the existing SRS record are

        • 024 2 2 $x 67890
          024 _ _ $a 12345
          024 1 _ $x 13579
          024 1 1 $a 45678
          024 1 _ $x 67890

      • I understand that for repeatable fields, FOLIO Lotus only pays attention to the first incoming field, not the rest, but compares to any matching fields in the existing record.

      • Now - setting up different match profiles, I want to be sure I understand the logic that is in place now:

      • If the match profile is 024 _ _ $a: 

        • Matches, because the incoming first 024 looks for an existing 024 with blank indicators and $a and the same value (even though that is the second 024 in the existing record)


        If the match profile is 024 1 1 $a:

        • Matches, because the first incoming 024 with indicators 11 and $a (which is the second 024 in the incoming file) looks for an existing 024 with indicators 11 and $a and the same value (which is the fourth 024 in the existing record)


        If the match profile is 024 1 _ $x:

        • Matches, because the first incoming 024 with indicators 1_ and $x (which is the third 024 in the incoming file) looks for an existing 024 with indicators 1_ and $x and the same value (which is the fifth 024 in the existing record)


        If the match profile is 024 2 2  $x:

        • Matches, because the first incoming 024 with indicators 22 and $x (which is the fourth 024 in the incoming file) looks for an existing 024 with indicators 22 and $x and the same value (which is the first 024 in the existing record)
      • However!
        Let’s pretend the incoming record looks like this:
        • 024 1 1 $a 12345
          024 1 1 $a 45678
      • And the existing SRS record is
        • 024 1 1 $a 45678
      • If the match profile is 024 1 1 $a, SRS does not match, even though “024 1 1 $a 45678” is present in both incoming and existing records.
        SRS starts searching a field, that is specified in match profile, scrolling the incoming record from the very beginning, as usual, and takes the first occurrence of <024 1 1 $a>. The first occurrence is “024 1 1 $a 12345". So, SRS takes “024 1 1 $a 12345” and can’t find it in the existing record 

From Christie Thomas (she/her) to Everyone 01:20 PM
I just posted a screen shot in our slack channel.

From Christie Thomas (she/her) to Everyone 01:27 PM
So, after our meeting last week I thought of a scenario where we may want to use a marc modification after the create / update actions. Not to complicate things, but it may be something to come back to.

From Jennifer Eustis (she/her) to Everyone 01:30 PM
Flow descriptions for DI: Flow descriptions

From Jennifer Eustis (she/her) to Everyone 01:41 PM
Here at 5C, 856's are protected because one school wants them in the instance whereas UMass removes them from the instance.

From Christie Thomas (she/her) to Everyone 01:42 PM
And override during batch!

From Jennifer Eustis (she/her) to Everyone 02:02 PM
these