2021-06-16 Data Import Subgroup meeting

Owned by Ann-Marie Breaux (Deactivated)
Last updated: Jun 17, 2021 by leeda.adkins@duke.edu

 Recordings are posted Here

Slack channel for Q&A, discussion between meetings

Additional discussion topics in Subgroup parking lot

Attendees: Ann-Marie Breaux (Deactivated) , Nick Cappadona , Jennifer Eustis leeda.adkins@duke.edu Jenn Colt, Christie Thomas, Timothy Watters, Lloyd Chittenden, Meghan Bergin

Agenda topics:

  1. Current development: Sprint 116:
    1. Finalizing Juniper releases, Iris Hotfix 2 releases
    2. Juniper bugfix release next week related to MARC Authority work, to distinguish types of MARC records when necessary and leave them generic when not
    3. Prep for Juniper Bugfest (which starts on 28 June)
      1. Work on updating the tests next week; Ann-Marie Breaux (Deactivated)to send out info about creating/editing tests and reviewing existing one; talk with Anton Emelianov (Deactivated)about lots of folks being on vacation during the Bugfest period
      2. Need to tighten up the tests; maybe combine some of the smaller ones to have one test with multiple checks; the simpler tests would be found as part of a larger test, e.g. create a new profile
      3. Will the automated tests (end-to-end and API tests) that are being developed in Kiwi mean that we can retire some of the manual tests?
      4. Include the blank tests to be adopted
      5. Include examples of what we've been seeing this whole release cycle
        1. How many updates can you make to a record
        2. Having action steps in certain sequence versus a different sequence
        3. Updating via QM and then doing an update via Data import (with the DI updates for Instance and SRS MARC in different sequence on the record)
    4. Iris hotfix updates
      1. Hotfix 2 bug fixes will be available on Iris Bugfest on 21 June; aiming for general release on 28 June
      2. Hotfix 3 TBD
  2. Field protection fixes: review examples. 
    1. How did testing go?
      1. leeda.adkins@duke.edu did testing on Iris bugfest
      2. Matching bib-bib on 999$i and no instance update; per leeda.adkins@duke.edu , logic is working properly for repeatable fields 
      3. Matching bib-bib on 999$i and instance update; per leeda.adkins@duke.edu , no matter whether the instance action or marc action is first, the new repeatable fields are not being added
      4. When instance action is introduced, depending on the sequence, sometimes quickMARC edits or subsequent SRS MARC edits are not possible (MODSOURCE-282 and MODQM-122 should fix some of these; fixes already available on folio-snapshot, and will be on Iris bugfest next week)
      5. Could you use an 035/Instance identifier match and then only include MARC update action? TBD; Edit by leeda.adkins@duke.edu : match tested in Honeysuckle for updating an EOCR to full cat with instance update only; haven't tested with marc update
      6. Better to do larger updates to make it more clear on whether the updates and field protections really worked
      7. Jenn Coltadded field protections to Cornell's local tenant and then did the OCLC update, and the protections worked!
      8. leeda.adkins@duke.eduhas been adding results in the Data import Slack channel; will add/link a doc with test results on the field protections wiki page
      9. Add Bugfest tests to cover these various scenarios (Ann-Marie Breaux (Deactivated))
      10. Summary
        1. Logic for repeatable fields seems to be working as expected, when it is invoked (what about non-repeatable fields?)
        2. If MARC update included in the job profile, the MARC protections are honored
        3. If only instance update included in the job profile, the MARC protections are not honored
        4. If MARC and instance updates both in the same job profile, results are inconsistent (can we make it so that field protections are honored without explicit MARC update action in the profile, and only need the explicit action when overriding the protections?)
    2. Reviewed the constraints on the field protections page
      1. See if we can make the field protections work no matter if the MARC update action is in the job or not, which was the original intent. If it is not too risky for a hotfix for bugfix, include in Iris and Juniper; if too risky, include in Kiwi
      2. Ensure that the protections are case-insensitive; if already working, Ann-Marie Breaux (Deactivated)to add TestRail test(s); if not working, Ann-Marie Breaux (Deactivated)to add bugfix/hotfix
      3. One master list of field protections for all MARC types; Ann-Marie Breaux (Deactivated)to ensure that devs know this, when implementing additional functionality for other MARC record types
      4. Being able to add additional field protections at at the job profile level; Ann-Marie Breaux (Deactivated)to add to Jira as part of a Field protection enhancement feature 
      5. Being able to use wildcards, begins, ends, contains for data specified in a field protection; Ann-Marie Breaux (Deactivated)to add to Jira as part of a Field protection enhancement feature 
    3. After this, follow-on work to override field protections (once they are working properly) and update individual SRS fields
    4. Will not be done by Iris Hotfix 2, needed in an Iris Hotfix 3?
  3. START HERE NEXT WEEK
  4. Has anyone created successful MARC to Instance Identifier matches for 010, 020, 022, 024, 035, 9xx fields? If so, it would be great to see examples. Developers still need to work on some identifier matching bug fixes (see examples in homework below)
    1. Has the MARC work from Juniper made it so that the other MARC-MARC matches can be unblocked? 
  5. Bug when creating Instances instead of matching/updating them: MODDATAIMP-427
    1. Review the bug and confirm preferred action
    2. Not planned as an Iris hotfix since it's an edge case. Needed as a Juniper bugfix? Or can it wait until Kiwi release?

HOMEWORK from last meeting:

  1. If you include field protection in a job profile explicitly (Update MARC/Updates - with no field protection overrides), is it working as expected? (other than having to include it explicitly)?
  2. Has anyone created successful MARC to Instance Identifier matches for 010, 020, 022, 024, 035, 9xx fields? If so, it would be great to see examples. Developers still need to work on some identifier matching bug fixes

    • Lisa (since I won't be able to make the meeting today, putting notes here): In Iris bugfest I was able to overlay with exact matches on the 010$a, 022$a, 024$a, and 035$a. The 020 to ISBN did not work I think because it appears the 020 in bugfest was not mapped to the identifier of ISBN in the Instance record. Example: HRID: in871746

      Inventory ViewView in quickMARC

    • Jennifer: I tested this about 2 weeks ago about for the match on 010, 020, 035 only. And I was lucky that there weren't multiple OCLC matches. This seemed to work and here's the video: https://umass-amherst.zoom.us/rec/share/yeOUyWSxxgRdf3YZielNYFZw-uskf0nGA3SmrMpcF3fYAc_G3YqYaxIQtgKd5wZZ.N83MYbSHPlqF9TqC. Note: The video will only 90 days online. Ann-Marie Breaux (Deactivated)to see if we can download the video and move to Data Import Google drive

To do's (plus ones above)

  • Confirm Rancher environment for Data Import/quickMARC testing (Khalilah and Ann-Marie) - IN PROCESS
  • Data Import FAQ updates (Ann-Marie) - IN PROCESS
  • Make videos confirming the following scenarios
    • For import profiles that match on Holdings or Item IDs (and doesn't need to affect the Instance), make a video showing what happens to SRS MARC (Ann-Marie)
    • For a brief MARC updating the suppress status on a group of instances. What happens with the SRS MARC (A-M)
  • Talk with Kimie and see if ideas for updating the UI for implicit (A-M)
  • If a profile has update instance (to update cat date) which has an implicit update MARC, and also an update MARC (with implicit update instance) that has field protections, will the field protections be honored by the update instance? (A-M to check with devs)
  • Most important part is 1) making sure this is clearly documented, and 2) preferably, consistent, and 3) preferably, that implicit actions are acknowledged somehow in the import profile (A-M confirm with devs and document)
    1. QM edit: increments
    2. Import MARC Modify: increments, we think
    3. Import MARC Update: does not increment
    4. Import Instance Update (implicit MARC update): increments