2021-12-15 Data Import Subgroup meeting

Owned by Ann-Marie Breaux (Deactivated)
Last updated: Dec 15, 2021
4 min read

 Recordings are posted Here

Slack channel for Q&A, discussion between meetings

Additional discussion topics in Subgroup parking lot

Attendees: Ann-Marie Breaux (Deactivated) leeda.adkins@duke.edu Lisa McColl  Jennifer Eustis Timothy Watters , Christie Thomas Monica Arnold Lloyd Chittenden Raegan Wiechert 

Lotus

Agenda topics: 

  • Last meeting of 2021!
  • Importing MARC Authority record UPDATES
    • Dev meeting tomorrow to finalize the development details
    • Spitfire will be doing the development, with Khalilah as PO
  • Kiwi Bug: Identifier matching should match on Identifier and Identifier type
    • https://folio-org.atlassian.net/browse/MODDICORE-231 
    • Preparing a Kiwi fix now
    • Should be on the hosted reference envs tomorrow
    • Did not change the logic for qualifier, only compare part of value, or type of match
    • Can anyone help with testing?
    • Test scenario we're using as a starting point
      • Records
        • Instance 1 has UPC with data 1 and OCLC number with data 2
        • Instance 2 has Invalid UPC with data 1 and System control number with data 2
      • Update scenarios:
        • Match on UPC, then update: should update Instance 1, but not Instance 2
        • Match on Invalid UPC, then update: should update Instance 2, but not Instance 1
        • Match on data 2 as OCLC number, with qualifier (OCoLC), numerics only: should update Instance 1, but not Instance 2
        • Match on data 2 as System control number, with qualifier (AMB): should update Instance 1, but not Instance 2 
    • Other test scenarios we should use?
      • Maybe some of the ones from Anne H's spreadsheet; A-M to review and add
      • Please test on Thurs Fri in snapshot-load if possible and report any results (good or bad) to A-M ASAP
    • Will consider a Juniper hotfix, but only after the Kiwi release (which should happen next week)
      • Juniper hotfix: probably needed, since migration to Kiwi won't be a few more weeks
    • Looking at surfacing no match/multiple matches in the UI log
  • Also looking at https://folio-org.atlassian.net/browse/MODDATAIMP-438: Ensure Inventory Single Record Imports are not delayed or queued behind large import jobs
    • Will plan to resolve in Lotus; too early for a decision on whether it will be a fix for Kiwi or not
    • Some comments A-M to add to Jira and discuss with developers
      • Leeda: In current system, single record imports have a separate queue from regular imports; similarly, EDIFACT and MARC Imports run in separate areas, MARC Authority runs separate from MARC Bibs
      • Jennifer E: single record imports: in current system, they get the toast right away; on FOLIO, for regular imports, have to refresh the regular import screen to see it's running
      • Jennifer E: Would there be value in turning off indexing, partial indexing, or allow indexing as a way to speed up the imports (and in which modules: Inventory, SRS, Invoices?)
      • Would be helpful to have clarity on how parallel or sequential the processing for imports are, when several are started at the same time by 1 user or by multiple users
      • Folijet also talking to Thor about the Success/Fail toasts in Inventory and timing them so that they are more in sync with whatever is happening to the Instance (create or update or problem)
  • Data Import Log Enhancements
    • Will try to surface messages for 
      • Further processing stopped because there was a match/no-match to 1 hit, and then no further action specified in the job profile
      • Further processing stopped because there the match resulted in multiple hits, so no further action taken
      • A-M to talk with devs and add an issue; probably Morning Glory
    • Discussion topic: Is there value in purging logs after X days?
      • Could we start with a pre-defined value for X that is the same for all FOLIO libraries? If so, what is that value?
      • Christie: Purge logs for certain types of jobs, but not for all jobs; if it's a large import, may not find the weird data until months later; use the logs to see what the process was and who ran it and other possible affected records and what external tools were used to process the records. System person reviews on a regular basis and manually deletes based on date and job type criteria
      • Leeda: +1 to Christie; can see the log detail for up to 4 months of jobs; beyond that you can see the summary (equivalent of the 1 line summary), but not the details for each record; would be helpful 
      • Jennifer E: purge every 90 days
      • Lisa M: +1 to Christie
      • Possible purge filters
        • Single record imports vs Regular imports
        • Date
        • Job profile
        • Scheduler
        • Tags (a certain job profile?)
      • Ann-Marie Breaux (Deactivated) will talk with the designer to see what ideas she has
      • If logs are purged, would something else be needed? Either at point of import or point of purge?
        • This is our starting question for our next meeting, in 2022!
        • See you then!

Chat: 

From Lisa McColl to Me (Direct Message) 01:00 PM
"You cannot edit the Full Name or Email fields as you are not using Confluence to manage users" - my error message

(discovered that they can be edited in Jira, and then that fixes the name in Confluence)From Jennifer Eustis (she/her) to Everyone 01:04 PM
There's no hawk emoji reaction :(
From Christie Thomas (she/her) to Everyone 01:09 PM
I thought this was a known issue! Or is this a different bug from the one where an order will match and inventory instance on any product id -> any identifier type issue.

(different bug. The POL one is related to Inventory lookup and is in the Orders app/Dennis. This one is match profiles in the Data Import app
This has been happening for a while.

From Jennifer Eustis (she/her) to Everyone 01:33 PM
The implementers group might

From Christie Thomas (she/her) to Everyone 01:34 PM
Tech Council is considering other upgrading issues - such as how reference data upgrades.

From Jennifer Eustis (she/her) to Everyone 01:52 PM
We purge every 90 days.

From Lisa McColl to Everyone 01:53 PM
+1 to what Christie is saying
esp. because FOLIO is not quite stable at this point

From Christie Thomas (she/her) to Everyone 01:58 PM
That  would be nice. (some sort of export option)