2021-09-01 Data Import Subgroup meeting

Owned by Ann-Marie Breaux (Deactivated)
Last updated: Sep 02, 2021

 Recordings are posted Here

Slack channel for Q&A, discussion between meetings

Additional discussion topics in Subgroup parking lot

Attendees: Ann-Marie Breaux (Deactivated) Jennifer Eustis Monica Arnold , Tim Watters, Lisa McColl, leeda.adkins@duke.edu, Khalilah Gambrell Christie Thomas, Lloyd Chittenden,  Jose Alexander

Development update: 

  • Kiwi planning - dashboard where you can see the current scope and status of Data Import work for Kiwi
  • Current Data Import feature development dashboard and bugfix support
  • Sprint demo for sprints 120-121: Recording
    • Folijet demo'd the fix for cat date not respecting the tenant time zone, and UI unit test work
  • Juniper hotfix
    • Necessitated by some libraries having MARC records with multiple 001s in SRS (not via Data Import UI, but via direct insertion into SRS, bypassing UI and business logic)
    • Juniper migration script assumed each SRS MARC only has one 001 field; multiples cause the migration script to fail, so some libraries cannot upgrade to Juniper
    • Updated migration script and Multiple 001 check script to deal with this
    • Will add more info in Iris and Juniper release notes
    • Will also create a wiki page listing SRS cleanup scripts that can be run periodically to check for invalid data

    • From Lloyd Chittenden: About the multiple 001.  Will the hotfix allow them in the future, or is this just a tool to make it easier to get rid of them before the upgrade?

      From A-M: SRS does not disallow them, and we're not changing that, so you could load a record with multiple 001s in the future, if you go direct to SRS and bypass the UI and business logic. But unsure what would happen in Data Import or in Data Export or in Inventory UI, where the system is expecting to only find one 001 field.

Agenda topics: 

  • System updates and validation rules for MARC Authority records: current plans. Any concerns or questions? (development will be handled by the Spitfire dev team)  
    • See additional Zoom chat below
    • Jennifer E:
      • inconsistent with MARC Bibs/Holdings
      • Having the manipulation and being able to add prefixes (the standard 001/003/035 manipulation) is helpful
    • Christie:
      • Would have to do pre-processing of all MARC authorities before they were loaded; would either need to delete them or move them somewhere else; if deleted, then might or might not replace the 001 with something else
      • if 001 is left as-is, might have accidental duplication if multiple authority vendors
      • How to identify different sources, if receiving records from multiple sources
      • A little strange that work is being done in the quickMARC group, without more consultation of the Entity Management group
      • If this was temporary, then it might be OK, but would need to have more options/become more robust in the future
      • If not going to manipulate, and is duplicate to 010 or other 0xx number field, maybe remove the 001/003?
    • Leeda: 003 has the source of the 001
      • 010 or 016 is Duke's update match point
      • Also interested in revisiting the HRID/001 autopopulation for other types of MARC records
    • Khalilah:
      • QM group has talked about being able to add default data to distinguish different sources (which can be accomplished with MARC modifications in the DI field mapping profile)
      • When get to the phase of linking the authority records to correct bib/instance holdings, then will definitely involve the Entity Management group
        • Per Christie: when that linking happens, there need to be options on whether an authority record controls headings in linked bib records
    • A-M:
      • 001 is an often used matchpoint
      • 001 as FOLIO HRID - give you a count/sense of size of records
    • Khalilah: 010 and UUID most often cited as numbers/identifiers to search on
    • Jennifer E: use the 001 for matchpoints for ETL; stromg preference is for consistency with handling of HRIDs in all 3 MARC record types
    • 001/003 Options
      • No system manipulation of 001/003 (unless library uses MARC modification to add/change the 003 or 9xx for source)
        • Quickest, but may be tricky to retrofit if different decision in the future
      • Follow the Bib/Holdings 001/003/035 manipulation
        • Would need to add a new settings screen
        • Not the preferred by the QM Subgroup, since HRIDs are not as necessary because they already have a value, and not showing in Inventory (yet)
      • Add the 001/003/035 manipulation at a later date
        • Cleaning up existing records 
      • Make it optional
        • The most work, since would require a new Setting, plus logic to make it optional
        • Would be consistent with what happens to Bibs/Instances and Holdings
    • Next steps: Khalilah and A-M to review and come back to both groups with recommendation
    • May also need to revisit the whole 001/003/035 manipulation and refine it (remove duplicates, make it optional) A-M add to backlog
    • A-M: Add bug for 003-009 validation in MARC modification field mapping, to remove validation for subfields and indicators
  • Handling Inventory optimistic locking:  Data Import support work
  • New identifier types being added, will affect the default MARC-to-Instance map: MODDICORE-172
  • Need to return to the scenarios for 4 August 2021 of trying to match 
    • Another reason for doing a match after an action - marc mod to add an 003, then match, then take further actions
    • For field protections - be able to use the qualifier to determine if the field is protected
  • Also will soon provide details for Shared Rancher (sandbox) environment for labs and Import/Export/quickMARC/OAI-PMH noodling

Zoom chat (mostly about MARC Authority 001)

From Leeda Adkins to Everyone:  01:27 PM
Currently our authority records come with an 001 that is a duplicate of the 010$a
Our 003 comes in with DLC

From Christie Thomas to Everyone:  01:33 PM
It would be good to under the scope of quickmarc.
Maybe we should have entity management representation on quickmarc and it is no longer about editing marc records be they bib records or authority records.
That should be understand, not just under …

From Khalilah Gambrell to Everyone:  01:34 PM
quickMARC subgroup contains folks who were on the entity management working group.

From Christie Thomas to Everyone:  01:35 PM
But maybe there are those of us who would want to be a part of the discussions if we knew that the scope had changed and it is about managing authority records now.

From Khalilah Gambrell to Everyone:  01:35 PM
We do discuss the areas that lean more towards entity management work.

From Leeda Adkins to Everyone:  01:37 PM
We do pre-processing for our MeSH records coming from MARCIVE, moving the 001 into an 016

From Christie Thomas to Everyone:  01:42 PM
Leeda: Do you have scripts to do that or do you use MarcEdit?

From Leeda Adkins to Everyone:  01:42 PM
MarcEdit


From Khalilah Gambrell to Everyone:  01:44 PM
Christie, I appreciate you getting us all on the same page. There are some many things happening in this project that it is always good to ask.

From Leeda Adkins to Everyone:  01:48 PM
Our plan is not to migrate our authority records from Aleph; rather we are going to load a refresh/replacement file from our authority vendor

From Jennifer Eustis (she/her) to Everyone:  01:52 PM
For us here, that's our match point on our system generated number

From Leeda Adkins to Everyone:  01:53 PM
We use the 010 or 016 as our update match point, so the 001 for authorities isn't used

From Christie Thomas to Everyone:  01:56 PM
UUID could give you a count.