2021-12-15 Data Import Subgroup meeting
Recordings are posted Here
Slack channel for Q&A, discussion between meetings
Additional discussion topics in Subgroup parking lot
Attendees: @Ann-Marie Breaux (Deactivated) @leeda.adkins@duke.edu @Lisa McColl @Jennifer Eustis @Timothy Watters , @Christie Thomas @Monica Arnold @Lloyd Chittenden @Raegan Wiechert
Lotus
Lotus Folijet planning: dashboard where you can see the current scope and status of Data Import work for Lotus
Current Data Import feature development dashboard and bugfix support
Agenda topics:
Last meeting of 2021!
Importing MARC Authority record UPDATES
Dev meeting tomorrow to finalize the development details
Spitfire will be doing the development, with Khalilah as PO
Kiwi Bug: Identifier matching should match on Identifier and Identifier type
Preparing a Kiwi fix now
Should be on the hosted reference envs tomorrow
Did not change the logic for qualifier, only compare part of value, or type of match
Can anyone help with testing?
Test scenario we're using as a starting point
Records
Instance 1 has UPC with data 1 and OCLC number with data 2
Instance 2 has Invalid UPC with data 1 and System control number with data 2
Update scenarios:
Match on UPC, then update: should update Instance 1, but not Instance 2
Match on Invalid UPC, then update: should update Instance 2, but not Instance 1
Match on data 2 as OCLC number, with qualifier (OCoLC), numerics only: should update Instance 1, but not Instance 2
Match on data 2 as System control number, with qualifier (AMB): should update Instance 1, but not Instance 2
Other test scenarios we should use?
Maybe some of the ones from Anne H's spreadsheet; A-M to review and add
Please test on Thurs Fri in snapshot-load if possible and report any results (good or bad) to A-M ASAP
Will consider a Juniper hotfix, but only after the Kiwi release (which should happen next week)
Juniper hotfix: probably needed, since migration to Kiwi won't be a few more weeks
Looking at surfacing no match/multiple matches in the UI log
Also looking at https://folio-org.atlassian.net/browse/MODDATAIMP-438: Ensure Inventory Single Record Imports are not delayed or queued behind large import jobs
Will plan to resolve in Lotus; too early for a decision on whether it will be a fix for Kiwi or not
Some comments A-M to add to Jira and discuss with developers
Leeda: In current system, single record imports have a separate queue from regular imports; similarly, EDIFACT and MARC Imports run in separate areas, MARC Authority runs separate from MARC Bibs
Jennifer E: single record imports: in current system, they get the toast right away; on FOLIO, for regular imports, have to refresh the regular import screen to see it's running
Jennifer E: Would there be value in turning off indexing, partial indexing, or allow indexing as a way to speed up the imports (and in which modules: Inventory, SRS, Invoices?)
Would be helpful to have clarity on how parallel or sequential the processing for imports are, when several are started at the same time by 1 user or by multiple users
Folijet also talking to Thor about the Success/Fail toasts in Inventory and timing them so that they are more in sync with whatever is happening to the Instance (create or update or problem)
Data Import Log Enhancements
Will try to surface messages for
Further processing stopped because there was a match/no-match to 1 hit, and then no further action specified in the job profile
Further processing stopped because there the match resulted in multiple hits, so no further action taken
A-M to talk with devs and add an issue; probably Morning Glory
Discussion topic: Is there value in purging logs after X days?
Could we start with a pre-defined value for X that is the same for all FOLIO libraries? If so, what is that value?
Christie: Purge logs for certain types of jobs, but not for all jobs; if it's a large import, may not find the weird data until months later; use the logs to see what the process was and who ran it and other possible affected records and what external tools were used to process the records. System person reviews on a regular basis and manually deletes based on date and job type criteria
Leeda: +1 to Christie; can see the log detail for up to 4 months of jobs; beyond that you can see the summary (equivalent of the 1 line summary), but not the details for each record; would be helpful
Jennifer E: purge every 90 days
Lisa M: +1 to Christie
Possible purge filters
Single record imports vs Regular imports
Date
Job profile
Scheduler
Tags (a certain job profile?)
@Ann-Marie Breaux (Deactivated) will talk with the designer to see what ideas she has
If logs are purged, would something else be needed? Either at point of import or point of purge?
This is our starting question for our next meeting, in 2022!
See you then!
Chat:
From Lisa McColl to Me (Direct Message) 01:00 PM
"You cannot edit the Full Name or Email fields as you are not using Confluence to manage users" - my error message
(discovered that they can be edited in Jira, and then that fixes the name in Confluence)From Jennifer Eustis (she/her) to Everyone 01:04 PM
There's no hawk emoji reaction :(
From Christie Thomas (she/her) to Everyone 01:09 PM
I thought this was a known issue! Or is this a different bug from the one where an order will match and inventory instance on any product id -> any identifier type issue.
(different bug. The POL one is related to Inventory lookup and is in the Orders app/Dennis. This one is match profiles in the Data Import app
This has been happening for a while.
From Jennifer Eustis (she/her) to Everyone 01:33 PM
The implementers group might
From Christie Thomas (she/her) to Everyone 01:34 PM
Tech Council is considering other upgrading issues - such as how reference data upgrades.
From Jennifer Eustis (she/her) to Everyone 01:52 PM
We purge every 90 days.
From Lisa McColl to Everyone 01:53 PM
+1 to what Christie is saying
esp. because FOLIO is not quite stable at this point
From Christie Thomas (she/her) to Everyone 01:58 PM
That would be nice. (some sort of export option)