2021-05-12 Data Import Subgroup meeting
Recordings are posted Here
Attendees: Ann-Marie Breaux (Deactivated) Jennifer E, Leeda A, Magda Zacharska, Mark Arnold, Lloyd, Nick Cappadona, Christie T, Jenn C, Alex
Agenda topics:
- Moving agenda and notes to the wiki, instead of a separate attached document - is that OK with everyone? Yes
- Iris hotfixes dashboard
- To track the hotfixes that are being considered for Data Import, and their status with the Cap Planning Team
- Juniper work mainly focused on stabilization and reliability, not additional features
- Are the Hotfix 1/Hotfix 2 dates listed on wiki? Hotfix 1 = May 28, Hotfix 2 = June 28
- Lisa/Lehigh out of 50K that tried to update - tested on Honeysuckle env, broken into groups of 1K; 10,384 did not update properly according to the LDP; DI log said successful; did not create an error in the log
- Lisa: please try to retest on Iris, and if reproduceable, add a bug; may be related to MODINV-400 or something completely different
- Mark/MSU: sometimes have situations in production Honeysuckle where the log says updated with errors, but actually all seemed to complete fine; 60 files of updates, 750 records per file, 1st 2 showed errors, and the rest didn't; everything looked OK - could not find any errors
- Jenn C/ worried about 999 matching, OCLC updates, testing doesn't cover chain of events (Inventory create, then update in QM, then update via DI, etc.); updates based on job profiles based updating MARC versus updating instance profiles
- Can see different titles between Instance and MARC Bib in snapshot-load: tested yesterday with the OCLC and it was working, then tested based updates today, and ended up with a mismatch - will send details in Slack
- We'll continue to update the FAQ and work on confirming that all processes are acting on SRS records the same way. It's hard to track right now given all the changes that are being made
- Yes, per Christie and Lisa - huge issue for data integrity, records get touched so many times, and it has to be reliable
- What sequences of actions are reliable, avoidable?
- Maybe include the generation in the View Source?
- Problem with 035$z's showing and then not
- Make it clear what is happening with the generation - when it updates and when it doesn't, and make sure it matches the design and is consistent across different import practices. We think...
- QM edits increment the generation and give it a new record ID
- Instance update: gives it gen 0 and new record ID and family
- MARC update: doesn't give it a new gen or a new record ID
- Is Single record update working properly? MODDICORE-136 - was working properly yesterday per Jenn, but then another fix today may have broken it? We will keep trying one fix at a time, and keep asking SMEs to help with testing
- What is a good next step?
- Per Nick - start with the admin info in the FAQ; finish filling it out
- Confirm what happens with a create (Inventory single record or DI)
- Confirm what happens with an update (QM, Inventory single record, DI update with Instance, DI update with MARC)
- Is it consistent or is it different? Should be consistent
- Example: MODSOURMAN-461 - 999$i being used as the Instance ID, but 999$i cannot be trusted(?), and some libraries remove the 999$i before importing
- Document what the SRS metadata is, what should happen in various scenarios, and whether it is in line with expectations
- Are there unit tests that check to make sure that those expectations are being matched? Yes
- Have an env where data will last (OK to use Rancher for post-Iris?) For Iris, bugfest and folio-iris would be the longest lived envs
- Start with fresh records if using Bugfest - Nick and Jenn can help if we need testers
- A-M talk with PM and Dev Mgt about next steps
- A-M will also check with devs about SME access to Rancher for testing, so that data and profiles are not blown away every night, as happens with the hosted ref envs
=================== Next Meeting ===================
- Data import setup checklist
- Data Import FAQ
- Surfacing errors in log summaries
- Fix column sorting
- Mockups
- Export – only errors? All?
- EDIFACT Invoices
Today's Zoom Chat:
From Christie Thomas (she/her) to Everyone: 01:28 PM
I agree, Jenn. I am concerned about what updates we will be able to reliably do in Iris when we go live, which will be on hot fix 1.
Protected fields also impacts our ability to update records.
I had not heard about the data in the title being different than the data in the instance, and that is also concerning.
From Jenn to Everyone: 01:32 PM
https://folio-snapshot-load.dev.folio.org/inventory/view/02e172e2-eaa4-4b31-acb0-55c9b17700de?query=business&sort=title
From Jennifer Eustis (she/her) to Everyone: 01:35 PM
Thanks for bringing that up Jenn about how updates do different things with the json admin data.
From Nick Cappadona to Everyone: 01:38 PM
https://folio-org.atlassian.net/browse/MODDICORE-136?focusedCommentId=103827&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-103827
From Christie Thomas (she/her) to Everyone: 01:45 PM
Thank you so much for leading on this very detailed troubleshooting!
Yes! Thank you, Nick!
From Jennifer Eustis (she/her) to Everyone: 01:46 PM
+++! to Nick!
From Lisa McColl to Everyone: 01:46 PM
+1
From Jennifer Eustis (she/her) to Everyone: 01:47 PM
Especially when we know that there is a push for bulk edit. If we know this, then this will help bulk edit. Knowing this also helps with marc reporting
From Christie Thomas (she/her) to Everyone: 01:47 PM
Good point, Jennifer.
This is not just data import / quick marc, but global updates as well.
From Christie Thomas (she/her) to Everyone: 01:58 PM
Thank you! I have to run to another meeting and need a short break between. Let me know if I can help test anything.