2021-08-18 Data Import Subgroup meeting

Owned by Ann-Marie Breaux (Deactivated)
Last updated: Aug 18, 2021

 Recordings are posted Here

Slack channel for Q&A, discussion between meetings

Additional discussion topics in Subgroup parking lot

Attendees: Ann-Marie Breaux (Deactivated) Timothy Watters Jennifer Eustis Ann Kardos leeda.adkins@duke.edu Lisa McColl Christie Thomas Monica Arnold  Lloyd Chittenden, Alex 

Development update: 

  • Kiwi planning - dashboard where you can see the current scope and status of Data Import work for Kiwi
  • Current Data Import feature development dashboard and bugfix support
  • Iris and Juniper release note updates
    • EDIFACT invoices:
      • If copying the default field mapping profile, DO NOT remove the mapping for the Vendor Reference Number field, since it will cause the import to fail. Will be fixed in Kiwi. 
      • Invoice files must have suffix of .edi or .inv (case insensitive). Starting in Kiwi, they should be able to have any file extension
    • PTF analysis and Recommended Maximum File Sizes and Configuration
  • New wiki info:
    • Recommended Maximum File Sizes and Configuration (as of Iris Hotfix 3 and Juniper)
      • Biggest issues to continue to work on:
        • Decreasing the processing size and increasing recommended file size for Inventory updates
        • Once authority records are importing, those will be large files
        • Larger loads for updates are important, but other priorities are higher (log enhancement, and especially update multiple holdings-items from 1 record, per Jennifer Eustis and Christie Thomas )
          • A-M to review outstanding reliability/performance work at end of Kiwi to see what if any big wins that could be done quickly in Lotus, and if any could be parked in favor of additional functionality
        • Ensuring that check-in/check-out time is less affected by imports
    • Match profiles: Most commonly used
    • MARC field mapping syntax table added to Field Mapping profiles: MARC
      • Ability to do if/then based on content, e.g. vocabulary in $2 of a 650 field (in the default MARC-to-Instance map)
      • Ability to do if/then in the field mapping profile (e.g. 856 indicator)
      • A-M to find or create draft features so that we can start gathering use cases

Agenda topics: 

  • Instance, Holdings, Item matching: Match profiles: Most commonly used
    • Add a story for matching by stat code (what happens if there are multiples?)
      • Match:
        • If stat code is ?, then stop
        • If stat code is (something), then create a new bib
    • Check the matching for locations (by name, code, and name (code))
  • Adding hotlinks to Import log
  • Need to return to the scenarios for 4 August 2021 of trying to match 
    • Another reason for doing a match after an action - marc mod to add an 003, then match, then take further actions
    • For field protections - be able to use the qualifier to determine if the field is protected
  • Also will soon provide details for Shared Rancher (sandbox) environment for labs and Import/Export/quickMARC noodling

To do's

  • Finish follow-up from 4 Aug 2021 DI mtg (A-M)
  • Confirm Rancher environment for Data Import/quickMARC testing (Khalilah and Ann-Marie) - IN PROCESS
  • Data Import FAQ updates (Ann-Marie) - IN PROCESS
  • Make videos confirming the following scenarios
    • For import profiles that match on Holdings or Item IDs (and doesn't need to affect the Instance), make a video showing what happens to SRS MARC (Ann-Marie)
    • For a brief MARC updating the suppress status on a group of instances. What happens with the SRS MARC (A-M)
  • Talk with Kimie and see if ideas for updating the UI for implicit (A-M)
  • If a profile has update instance (to update cat date) which has an implicit update MARC, and also an update MARC (with implicit update instance) that has field protections, will the field protections be honored by the update instance? (A-M to check with devs)
  • Most important part is 1) making sure this is clearly documented, and 2) preferably, consistent, and 3) preferably, that implicit actions are acknowledged somehow in the import profile (A-M confirm with devs and document)
    1. QM edit: increments
    2. Import MARC Modify: increments, we think
    3. Import MARC Update: does not increment
    4. Import Instance Update (implicit MARC update): increments