2021-12-01 Data Import Subgroup meeting

Attendees: @Ann-Marie Breaux (Deactivated) @leeda.adkins@duke.edu  @Jennifer Eustis @Timothy Watters @Christie Thomas @Jenn Colt @Monica Arnold @Nick Cappadona @Raegan Wiechert @Lloyd Chittenden 

Sequencing of MARC fields in View source:

• @Monica Arnold will send sample to A-M

• Check what happens to 0XX fields when imported

• And when updated in quickMARC

• And during migration (are they being put in numerical order by a process outside of DI? If yes, hat would be concerning for 5xx, 6xx, 7xx fields also

• @Ann-Marie Breaux (Deactivated) will add a Jira and investigate

Kiwi Bugfix work: Done for DI

• Location match regressions fixed

• Importing large invoices

• Data Import profiles with data from multiple EDIFACT fields added to a single Invoice or Invoice line field

• Still updating PTF stats for size of files and timing of import; should have updated recommendations in the next couple of weeks

• Fixed the 00X field protection bug

Juniper Hotfix 4 work: Done for DI

• Importing large invoices

Lotus

• Lotus Folijet planning: dashboard where you can see the current scope and status of Data Import work for Lotus

  • Matching Inventory records on POL and VRN: work has been finalized 

  • Per Jenn, in addition to 9xx, see if we can match on 035

  • Duke sometimes has two 901s, and they will have different vendor reference numbers; would want to check both 901s when matching

  • Christie: non-OCLC 035s are being matched to the Instance System Control Number

  • Need MARC-MARC matching to do field protection; can't do that if match MARC-Instance

  • Any sample files you would like the devs to use for testing? Will need the preliminary file and final file (since these are updating source = FOLIO Instances and/or SRS MARC Bibs created at point of order)

  • We need better documentation of the DI flow

    • the implicit/explicit

    • also making sure that we finish features that are partially done and displaying in the UI

    • make sure that the 001/999 ff matching is on the matching documentation wiki page

    • make sure there's a clear explanation of what each action does on the actions page

• Current Data Import feature development dashboard and bugfix support

Agenda topics: 

• UUIDs for Inventory and SRS

  • From Honeysuckle HF 3 through Kiwi, FOLIO has been creating Instances and MARC Bibs with the same UUID if the Instance and SRS MARC Bib are created at the same time, and putting that UUID in 999 ff $i and $s

  • UUID handling for Inventory Holdings and SRS MARC Holdings, and Inventory Authorities and SRS MARC Authorities is the same, but has not been happening as long

  • In Lotus, will change the UUID assignment so that different UUIDs are assigned for Inventory and SRS versions of records, and put those UUIDs in 999 ff $i and $s

  • Preference for the existing records with the same UUIDs is to NOT try to update them to have different UUIDs

    • Why? Seems risky, especially if libraries have exported  FOLIO records to other organizations  and are expecting the UUDs to stay the same

    • Seems difficult

    • Is that an OK decision? Any issues or questions?

    • Jenn: don't try to change the existing ones

    • A-M to talk to devs

      • Ask the devs: for records that have the same UUIDs in 999 $i and $s, especially in the future, when this time period will have been a small blip

      • Ask the devs: why are we changing it? Is it because something is broken because we have the same UUID for 2 different record types, or is it something more serious? If it's important to have different UUIDs, then will it be a problem if we leave some records (or even all, going forward) with the same UUIDs?

      • Make sure any architectural decisions like this are documented on a wiki page and in the code

      • Do the tech leads or TC have a place where architectural decisions are documented?

    • In the Import lab 2 Dec 2021

      • Are the 999 ff $i and $s really the same for all new Instance/SRS creates at the same time? DI, Inv Single Rec Import, quickMARC updates, quickMARC derives

      • Play with 999s in lab on 2 Dec 2021 and see what happens when they are created at the same time or different times, if the incoming record has 999s or not

      • Jenn: 999s should be trustworthy for matching, but not necessarily for reporting

      • Could libraries change on their own, instead of a centralized cleanup script?

      • Recommendation for users: Don't build a process that assumes $i and $s are always the same, or are always different

• Data Import log improvements: brainstorming session

  • Is there additional info on the backend that is not surfacing in the UI?

  • Filtering by errors is helpful

  • Not all completed with errors jobs give enough information, especially if it's status Failed

  • DI will create the Instance, but no SRS created, will Complete with errors, but not have any error messages

  • Having error messages be more understandable - see Lisa's InstanceType example (seems to be related to the 008)

  • If SRS is discarded, need to have asterisk and explanation of errors

  • The summary page showing the overall results of the job has not yet been developed, and it's really hard to find the errors, especially for large jobs - counts of what succeeded and what failed

  • There are some records that cannot be loaded, and no idea why

  • Is it important to log the single record imports? Yes, everyone agrees

  • START HERE NEXT MEETING

  • Is it important to keep DI logs online forever? (Kiwi BF has 7k+, Cornell has 20K+) challenging for the View all screen, since it's trying to keep track of so much data

    • What if they were automatically purged after x days?

    • What if the log results could be exported?