2022-07-06 Data Import Subgroup meeting

Owned by Ann-Marie Breaux (Deactivated)
Last updated: Jul 07, 2022
5 min read

Recordings are posted Here (2022+) and Here (pre-2022)                   Slack channel for Q&A, discussion between meetings

Requirements details Here                                                                    Additional discussion topics in Subgroup parking lot


Attendees: Ann-Marie Breaux (Deactivated) Christie Thomas Heather MacFarlane (Deactivated) Jennifer Eustis Lynne Fors Monica Arnold Taylor Smith Timothy Watters Lloyd Chittenden 

Morning Glory

Agenda topics:

  • Lotus Potential Hotfixes:
    • Key factors: regressions or not, risk level, data integrity, any workarounds
    • Quota Exceeded error UIDATIMP-1205 - Getting issue details... STATUS
      • Will be included in Lotus HF2
      • Awaiting deployment to Lotus BF, so that it can be tested
    • Holdings source problem, when Holdings created by opening an order is updated by Data Import MODORDERS-717 - Getting issue details... STATUS
      • Will be included in Lotus HF2
      • Is not locking down fields, but changes the Source from FOLIO to MARC
      • Fix is ready and is awaiting deployment to Lotus Bugfest for testing
      • Will add a note in the Lotus release notes about related scripts for fixing existing Inventory Holdings with Source = MARC, but no underlying MARC Holdings in SRS
    • 035 Matching regression MODSOURCE-521 - Getting issue details... STATUS
      • May be included in Lotus HF3
      • Related to MODSOURCE-509 - Getting issue details... STATUS
      • Also have to be competing 035s to happen, so if clean all of them out, it's OK
      • Also can pre-construct the new 035, and that helps
      • Affects MARC-to-Instance, which we were not trying to change
      • Almost done with the development work, then need to test heavily and release in Lotus HF3 
      • Dev team's recommendation is to only release the regression fix, not all the work from both Jiras - is that OK?
      • Hopefully extensive testing in Lotus BF by SMEs
    • Protected fields being removed from the record on import MODDICORE-272 - Getting issue details... STATUS
      • The 2 scenarios actually look like 2 different issues
        • Scenario 1: (field number): already fixed in MG, but low-ish risk to backport to Lotus
        • Scenario 2: ($5 value on any field):
          • Per Christie, it was definitely working in Kiwi - doublecheck
          • Still looking at whether it ever worked in Kiwi or not
          • It is working in Morning Glory
          • It is risky to backport to Lotus
          • Is it OK to add a "known issue" release note for Kiwi and Lotus, as long as it's fixed in MG? Need to revisit this once more testing and discussion with devs

Change default mapping for relator term (Nolana)

  • MODDATAIMP-700 - Getting issue details... STATUS
  • Similar to how the 336 mapping was changed - OK?
  • Yes - still TBD if the Instance field will change to multi-select; if not, only the first relator term is visible in the Instance, but all are still in the SRS MARC underneath

Mark Instances for deletion

  • Confirm Data Import requirements
    • Cross-SIG Slack channel for discussion: https://folio-project.slack.com/archives/C03KPLWJBLM
    • Need to be 100% clear on the scope for Nolana, and how various apps that interact with Instances will be affected; still being determined, but it definitely will not be complete deletion of Instances and related SRS MARC
    • For Data Import, the main concern is Instances with Source = MARC, not Instances with Source = FOLIO
    • Current workarounds in use by FOLIO libraries:
      • Statistical code
      • Labels
      • Suppress from discovery
      • Chicago: Stat code+910+instance note (with special note type of Deletion note)
        • Chicago testing a profile that adds stat code, 910, and instance note, but does not change Leader byte
      • Is there value in trying to standardize these workarounds (which is what "mark for deletion kind of feels like" across FOLIO if FOLIO does not allow for actual deletion of Instances?
      • Does taking this type of step set us up to accomplish the larger Instance/SRS MARC deletion effort more easily in the future, or is it development effort that could be used for something else?
    • Import that marks an instance for deletion - still TBD, but probably 
    • Import that unmarks an instance for deletion - still TBD, but probably 
    • Requirements for Staff suppress, Suppress from Discovery - do they need to be coordinated with Mark for deletion? Maybe? Not sure
    • Is it important to change the Leader byte in the SRS MARC record? 
      • Yes for OAI-PMH, and for VuFind (see Zoom comments below)
      • Maybe would need to be a tenant-level setting, instead of automatic
      • This could not be only an import profile-driven value, since the user would often be setting the value in the Instance, rather than via Data Import (similar to how the Suppress from Discovery value is set currently; Data import can affect it, but the source of truth for the value is the Inventory Instance, and an underlying SRS MARC gets a replacement value from Inventory whenever the value is changed in the Instance, which is the opposite flow of almost all the other MARC-Instance data elements.
    • Is it important to restrict editing via QM or DI? Maybe no. Why should it be treated differently from Staff suppress or Suppress from discovery? Plus it would likely be significant work. Is it worth the effort to lock down editing if these records are not likely to be acted upon again after they have been marked for deletion?
    • Requirements for SRS/OAI-PMH
      • Needs to be removed from OAI-PMH outputs, but that already happens for Suppress from discovery, so is this additional value/status/flag really needed?
    • Out of scope:
      • SRS MARC Bib reacting to an Instance being marked for deletion, if that Instance is not Source = MARC
      • Disconnecting an SRS MARC Bib from an Inventory Instance (and the Instance changing back to Source = FOLIO)

Next week:

  • Finalize requirements for Data Import creating Orders from MARC Bibs (to be delivered in Nolana)


Slack chat:

From Christie Thomas (she/her) to Everyone 01:15 PM
It was working in Kiwi.

From Jennifer Eustis (she/her) to Everyone 01:16 PM
Did you test in your sandbox or in the FOLIO kiwi environment?

From Christie Thomas (she/her) to Everyone 01:25 PM
Field protections scenario 2 appears to not be a regression according to Kiwi bugfest. I just tested it there and the bug is present there as well.

From Lynne Fors to Everyone 01:39 PM
Yes, does need to not be sent out by OAI-PMH

From Christie Thomas (she/her) to Everyone 01:46 PM
Well, I think that using the leader byte should not be used as a part of the deletion workflow since not all records will have a marc source.

From Christie Thomas (she/her) to Everyone 01:52 PM
I think that we at Chicago are okay with our current workflow for "marking things for deletion" which is marking it suppressed from discovery, adding a stat code of delete, and adding a deletion note (locally created instance note). I don't think that we need any other functionality.

From Christie Thomas (she/her) to Everyone 01:57 PM
Unless it was deleted by accident. But we usually just create a new instance.

From Lynne Fors to Everyone 01:58 PM
I am not in favor of locking down quickMarc when marked for deletion.  We change LDR 05 to d, but we sometimes go in and change it back to an active record if we get a replacement copy.  We were told that VuFind prefers to use LDR 05 = d to assist in managing what records display and what don't outside of the mark for suppression box

From Heather MacFarlane to Everyone 01:59 PM
I need to jump to another meeting, thank you Ann-Marie!

From Lloyd Chittenden to Everyone 02:00 PM
I think they would very rarely flip back.

From Lynne Fors to Everyone 02:00 PM
We would purge the LDR 05 = d incrementally.

From Taylor Smith to Everyone 02:04 PM
have a good one everybody