Skip to end of banner
Go to start of banner

2022-03-23 Data Import Subgroup meeting

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Recordings are posted Here (2022+) and Here (pre-2022)                   Slack channel for Q&A, discussion between meetings

Requirements details Here                                                                    Additional discussion topics in Subgroup parking lot


Attendees: Ann-Marie Breaux (Deactivated) Timothy Watters 

Lotus

Morning Glory

  • Morning Glory Folijet and Spitfire planning: dashboard where you can see the current scope and status of Data Import work for Morning Glory
  • Started work on E2E automated tests (smoke tests for Inventory)
  • Will soon be starting log refinements (deleting logs)

Agenda topics:

Lotus Bugs

  • MARC Field protection:
    • Can we assume that LDR, 002-009 fields would never need field protection
    • Will allow user-assigned field protection for 006 and 007, but not LDR, 001-005, 008-009
    • Field protections apply to MARC Bibs and Authorities, but not Holdings 
    • Should we add any text at the top of the field protection settings that explains the 1) restrictions on control fields or 2) types of MARC records or 3) variations in handling for non-repeatable vs repeatable fields?
  • Record matches are not decreased when additional match conditions are added to a job profile
    • See example in bug; Devs need additional examples of multi-tiered matches
    • Make first match as specific as possible (retrieve no more than 90 results), then secondary match to narrow that further
    • Kiwi release notes: add as known issue
    • Lotus release notes: add that this is partially addressed in Lotus fix, will aim to change/remove restriction in Morning Glory
  • Deleting job logs
    • Will be possible from the UI on the Data Import landing page and the View all page, plus by directly hitting an endpoint in the backend
    • OK that the logs are delete and cannot be restored?
    • OK that (for now) deleting logs will be part of the Data Import: All permissions?

  • Extend MatchValueLoader implementations to allow filtering according to Qualifiers and MatchCriteria:

  • Identifier matching should allow for qualifiers, compare part, and match criteria

    • Are there any specific match use cases that you want to use that you cannot (NOT MARC-MARC right now; that's next)

    • Any qualifier/begins/contains matches that are not working but that are needed?

  • MARC-MARC matching
    • Lotus: Allows for any field in a MARC record except
    • Are these needed in Morning Glory?
      • Matching for 100-899 fields? (I think they work, but not heavily tested yet)
      • Repeatable fields (e.g. 024, 035)
        • Incoming record: Only first version of the field is considered (doublechecking with the dev on whether it's the first field that has the requested indicator(s) and/or subfield, or just the first field, regardless of indicators/subfield)
        • If it takes Ind 1, Ind 2, Subfield into account (in addition to the data)
        • Does FOLIO need to check all incoming 024s against all 024s in the existing SRS records? Or just the first?
        • Wildcards for Ind 1, Ind 2, Subfield (repeatable or non-repeatable fields)
          • Needed?
    • Additional info from A-M/Igor:

      • Let's pretend that these fields are in an incoming record: (Field Ind1 Ind2 Subfield)

        • 024 _ _ $a 12345
          024 1 1 $a 45678
          024 1 _ $x 67890
          024 2 2 $x 67890

      • And the fields in the existing SRS record are

        • 024 2 2 $x 67890
          024 _ _ $a 12345
          024 1 _ $x 13579
          024 1 1 $a 45678
          024 1 _ $x 67890

      • I understand that for repeatable fields, FOLIO Lotus only pays attention to the first incoming field, not the rest, but compares to any matching fields in the existing record.

      • Now - setting up different match profiles, I want to be sure I understand the logic that is in place now:

      • If the match profile is 024 _ _ $a: 

        • Matches, because the incoming first 024 looks for an existing 024 with blank indicators and $a and the same value (even though that is the second 024 in the existing record)


        If the match profile is 024 1 1 $a:

        • Matches, because the first incoming 024 with indicators 11 and $a (which is the second 024 in the incoming file) looks for an existing 024 with indicators 11 and $a and the same value (which is the fourth 024 in the existing record)


        If the match profile is 024 1 _ $x:

        • Matches, because the first incoming 024 with indicators 1_ and $x (which is the third 024 in the incoming file) looks for an existing 024 with indicators 1_ and $x and the same value (which is the fifth 024 in the existing record)


        If the match profile is 024 2 2  $x:

        • Matches, because the first incoming 024 with indicators 22 and $x (which is the fourth 024 in the incoming file) looks for an existing 024 with indicators 22 and $x and the same value (which is the first 024 in the existing record)
      • However!
        Let’s pretend the incoming record looks like this:
        • 024 1 1 $a 12345
          024 1 1 $a 45678
      • And the existing SRS record is
        • 024 1 1 $a 45678
      • If the match profile is 024 1 1 $a, SRS does not match, even though “024 1 1 $a 45678” is present in both incoming and existing records.
        SRS starts searching a field, that is specified in match profile, scrolling the incoming record from the very beginning, as usual, and takes the first occurrence of <024 1 1 $a>. The first occurrence is “024 1 1 $a 12345". So, SRS takes “024 1 1 $a 12345” and can’t find it in the existing record 


  • No labels