2024-1-31 Data Import Subgroup meeting

Recordings are posted Here (2022+) and Here (pre-2022) Slack channel for Q&A, discussion between meetings

Requirements details Here Additional discussion topics in Subgroup parking lot

Attendees: @Ryan Taylor @Taylor Smith @Aaron Neslin @Raegan Wiechert @Corrie Hutchinson (Unlicensed) @Autumn Faulkner @Peter Martinez @Yael Hod @Lynne Fors @Jeanette Kalchik @Sara Colglazier @Jennifer Eustis @Kim Wiljanen Tess Amran

Notetaker: Jennifer Eustis, Corrie Hutchinson, Christie Thomas

Links:

Agenda:

Topic	Who	Meeting Notes	Related Jira	Decisions and Actions

Topic	Who	Meeting Notes	Related Jira	Decisions and Actions
Announcements	Ryan	Quesnelia Planning Working with the team next week, about 1/2 way through Will bring any testing needs to the group soon as appropriate Ramsons Planning Planning to have a list of features/issues/bugs for review by subgroup for feedback and prioritization Likely use a spreadsheet as was done for Quesnelia There will be a CSP for Poppy which will include Folijet tickets. Timeline not yet determined.	N/A	N/A
Missing Action Profiles in Job Profile after Poppy migration: As called out in Poppy Release Notes, there is a known issue that's been observed in which some links to reusable Action Profiles might be missing from Job Profiles after Poppy migration. Release notes recommend the following: ^{After migration, review existing Job Profiles to verify they migrated correctly. Pay attention to reusable Action Profiles. In case issues are found, Job Profile can be updated manually. For additional information on links created for that Job Profile - execute}^{script #15}^{(or follow the}^link^{), notify support.} Recommended script will provide list of Action profiles to help users manually recreate any affected Job profiles.	All	Overview : Action profiles connected to multiple job profiles are 'unlinked' from job profiles after migration to Poppy. It isn't happening for every library after migration or for every re-used profile. However, it is happening often enough that libraries should be aware and check. General confusion on how or if the migration to Poppy is causing this issue. Root cause is not migration, but the migration process does cause the profiles to unlink. Script #15 (noted in the topic column) provides a list of profiles that need to be fixed. It does not fix the links. That must be done manually by the library. The action profiles actually disappear from the job profile, not just 'unlink'. They must be re-added. Comments : Lots of work for libraries to recreate job profiles manually. Should be a CSP candidate. High priority for correction. Could be a blocker for some libraries to migrate to Poppy. The "unlinking from one unlinks them all" issue has popped up multiple times. Sidebar discussion in chat on how job profiles are deleted spurred #42 in the Data Import Issue Tracker. Until MODDICONV-361 is fixed, any time a re-used action profile is unlinked in a job profile it will be unlinked in all other job profiles. Fixing it after migration doesn't stop it from happening again should a re-used action profile be unlinked. The development team will be adding new test cases to their workflow to test this type of scenario (re-used profiles) going forward.	https://folio-org.atlassian.net/browse/MODDICONV-361 is scheduled for Quesnelia release to fix root cause of behavior.	Ryan will : Confirm this only impacts action profiles and not match profiles Get an idea for how commonly this occurs Test to ensure MODDICONV-361 only impacts re-used action profiles when 'unlinked' and not when added to a job profile. Talk to the dev team about bumping up the priority and included in CSP#1
Partial Matching:	Subject raised by @Yael Hod	Partial matching, e.g. begins with, ends with, is required but it does not function as it should regardless of how it is configured. System behaves as though it only looks for exact matches. Examples of use include prefixes/suffixes to an 035 added by a vendor or library to designate the source of the record. University of Chicago has had same issue. Corrie submitted MODDICORE-386 on their behalf. Question as to whether this is a bug or how the system is intended to function. Documentation is needed. #12 on the Data Import Issue Tracker.	https://folio-org.atlassian.net/browse/MODDICORE-386	Ryan will : Review Jira with Folijet leads to understand current design and identify requirement gaps.
Documentation: The group has identified a need for new, enhanced, or reorganized documentation around Data Import. In a previous session, we agreed that completing a functionality audit spreadsheet would be a good first step	All	Not discussed at the 1/31/ meeting Previous notes from 1/24 meeting: In lab session on 1/18/2024, we created a wiki page, Data Import Requesting a New Topic, with guidelines on how to contribute and a spreadsheet to track issues. This is based on the work done in the Acquisitions SIG. An archive area was also created where we could archive outdated pages such as the Archived Data Import Implementers and Feature Discussion Topics. The idea was to put down issues whether they were linked to a Jira issue or not. Some of the important information that we wanted to track was if there was a linked Jira and in particular when the issue was discussed in the working group and the decision(s) made in regard to that issue. The spreadsheet is still being developed. Before we add more issues, the group in lab wanted to know: Do we adopt this page and spreadsheet? If yes, do we have volunteers to populate it? To make sure this page is maintained, the group suggested that the working group look at it once a month to see what is outstanding or new. Is this a practice we want to adopt? Discussion: A link to the new Data Import topic tracker is at the top of the page. Format was worked on at last week's data import session. Q: is this only to track Jira tickets? Or will there be other topics added to the agenda. R: In Acq /RM individuals add stories to the topic tracker and the Jira may only be added later to the spreadsheet. (many think this is a good idea.) Can reference the Acq/Resource Management implementers topic tracker. Perhaps add widgets that bring in Jiras automatically based on the tag. Q: How to add "Click here and expand" text. R: Put the cursor where you want the text block to begin and use Insert Macro function. Type "Expand" to locate the Expand Macro. Agreed to use the de-duplication discussion to work on building a useful functionality framework.	N/A	Get volunteers to create a spreadsheet and start brainstorming - DONE
De-duplication: Continue conversation from previous session to clarify what we expect from de-duplication of field values when a record is loaded into FOLIO via Data Import.	All	Not discussed at the 1/31 meeting. Previous notes from 1/24 meeting: @Jennifer Eustis and @Aaron Neslin found comments in the data-import-processing-core code that provides details about expected behavior for de-duplication. These comments align with the behavior we are seeing except for when there is duplicate data in the incoming record. Data is being removed from the incoming record on update as well. Consensus seems to be that FOLIO should not be de-duplicating within the incoming record unless it is explicitly defined in an import profile. Q: Is de-duplication something that should be able to be deactivated on a field by field basis? R: Sounds like a reasonable approach. There is also some concern that this would complicate an already complicated situation. Possible solution - a tool to deduplicate in another tool rather than within data import instead. Suggestion to start with the functionality audit. RT can connect with the developers as a part of this audit. Q: Are we starting with how we as users expect functionality work or with how the developers expect it to work. R: Really should have both for each feature. Start from perceived / desired functionality of the users and add to it with designed functionality. Suggestion to provide examples to the developers so that it is clear what we are expecting. Pilot functionality audit with de-duplication and start with our understanding and then get input from the developers.	MODDATAIMP-879: Data Import removes duplicate 856s in SRS	Clarify current behavior of field value de-duplication. Define desired behavior of field value de-duplication (if different). @Christie Thomas will create some dummy data to illustrate deduping 856s.

Upcoming meetings/agenda topics:

Chat: