2022-03-23 Data Import Subgroup meeting
Recordings are posted Here (2022+) and Here (pre-2022) Slack channel for Q&A, discussion between meetings
Requirements details Here Additional discussion topics in Subgroup parking lot
Attendees: Ann-Marie Breaux (Deactivated) Timothy Watters Jennifer Eustis Autumn Faulkner Christie Thomas Jenn Colt Lloyd Chittenden Raegan Wiechert Taylor Smith
Lotus
- Lotus Folijet planning: dashboard where you can see the current scope and status of Data Import work for Lotus
- Current Data Import feature development dashboard and bugfix support
- Main remaining Lotus BF work: field protections
Morning Glory
- Morning Glory Folijet and Spitfire planning: dashboard where you can see the current scope and status of Data Import work for Morning Glory
- Started work on E2E automated tests (smoke tests for Inventory)
- Will soon be starting log refinements (deleting logs)
Agenda topics:
Lotus Bugs
- MARC Field protection:
- Can we assume that LDR, 002-009 fields would never need field protection
- Will allow user-assigned field protection for 006 and 007, but not LDR, 001-005, 008-009
- Since 006 is sometimes used to generate format icons for discovery, maybe protect it?
- If 006 is protected, should the 008 also be protected? No
- Field protections apply to MARC Bibs and Authorities, but not Holdings
- A-M: Check with Brooks - will this have any implications for InnReach actions? 24 March 2022: No, all should be fine; all triggers based on Inventory Instance record, and then info in the corresponding SRS MARC; no triggers based on MARC Holdings records. Some triggers based on Inventory Holdings and Inventory Item records.
- Should we add any text at the top of the field protection settings that explains the 1) restrictions on control fields or 2) types of MARC records or 3) variations in handling for non-repeatable vs repeatable fields?
- No, add Info icon that leads to the field protections wiki page
- A-M: Add story for info icon
- A-M: finish cleaning up UI restrictions for the control fields
- A-M: Add info in the field protection document and confirm with devs that fields are not protected in the incoming record unless the MARC Updates action is sequenced before the MARC Modification action in the job profile. Field protections should not happen on the incoming record, because field protections apply to the existing record, not the incoming record.
- Record matches are not decreased when additional match conditions are added to a job profile
- See example in bug; Devs need additional examples of multi-tiered matches
- A-M: Doublecheck the match profile in Kiwi BF - why is the MARC data not showing?
- Make first match as specific as possible (retrieve no more than 90 results), then secondary match to narrow that further
- Kiwi release notes: add as known issue
- Lotus release notes: add that this is partially addressed in Lotus fix
- Will park any additional changes until users try the Lotus changes
START HERE
- Deleting job logs
- Will be possible from the UI on the Data Import landing page and the View all page, plus by directly hitting an endpoint in the backend
- OK that the logs are delete and cannot be restored?
- OK that (for now) deleting logs will be part of the Data Import: All permissions?
Extend MatchValueLoader implementations to allow filtering according to Qualifiers and MatchCriteria:
Identifier matching should allow for qualifiers, compare part, and match criteria
Are there any specific match use cases that you want to use that you cannot (NOT MARC-MARC right now; that's next)
Any qualifier/begins/contains matches that are not working but that are needed?
- MARC-MARC matching
- Lotus: Allows for any field in a MARC record except
- Are these needed in Morning Glory?
- Matching for 100-899 fields? (I think they work, but not heavily tested yet)
- Repeatable fields (e.g. 024, 035)
- Incoming record: Only first version of the field is considered (doublechecking with the dev on whether it's the first field that has the requested indicator(s) and/or subfield, or just the first field, regardless of indicators/subfield)
- If it takes Ind 1, Ind 2, Subfield into account (in addition to the data)
- Does FOLIO need to check all incoming 024s against all 024s in the existing SRS records? Or just the first?
- Wildcards for Ind 1, Ind 2, Subfield (repeatable or non-repeatable fields)
- Needed?
- Additional info from A-M/Igor:
Let's pretend that these fields are in an incoming record: (Field Ind1 Ind2 Subfield)
- 024 _ _ $a 12345
024 1 1 $a 45678
024 1 _ $x 67890
024 2 2 $x 67890
- 024 _ _ $a 12345
And the fields in the existing SRS record are
024 2 2 $x 67890
024 _ _ $a 12345
024 1 _ $x 13579
024 1 1 $a 45678
024 1 _ $x 67890
I understand that for repeatable fields, FOLIO Lotus only pays attention to the first incoming field, not the rest, but compares to any matching fields in the existing record.
Now - setting up different match profiles, I want to be sure I understand the logic that is in place now:
If the match profile is 024 _ _ $a:
- Matches, because the incoming first 024 looks for an existing 024 with blank indicators and $a and the same value (even though that is the second 024 in the existing record)
If the match profile is 024 1 1 $a:- Matches, because the first incoming 024 with indicators 11 and $a (which is the second 024 in the incoming file) looks for an existing 024 with indicators 11 and $a and the same value (which is the fourth 024 in the existing record)
If the match profile is 024 1 _ $x:- Matches, because the first incoming 024 with indicators 1_ and $x (which is the third 024 in the incoming file) looks for an existing 024 with indicators 1_ and $x and the same value (which is the fifth 024 in the existing record)
If the match profile is 024 2 2 $x:- Matches, because the first incoming 024 with indicators 22 and $x (which is the fourth 024 in the incoming file) looks for an existing 024 with indicators 22 and $x and the same value (which is the first 024 in the existing record)
- However!
Let’s pretend the incoming record looks like this:- 024 1 1 $a 12345
024 1 1 $a 45678
- 024 1 1 $a 12345
- And the existing SRS record is
- 024 1 1 $a 45678
- If the match profile is 024 1 1 $a, SRS does not match, even though “024 1 1 $a 45678” is present in both incoming and existing records.
SRS starts searching a field, that is specified in match profile, scrolling the incoming record from the very beginning, as usual, and takes the first occurrence of <024 1 1 $a>. The first occurrence is “024 1 1 $a 12345". So, SRS takes “024 1 1 $a 12345” and can’t find it in the existing record