Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Requirements details Here                                                                    Additional discussion topics in Subgroup parking lot


Attendees: Jennifer Eustis Corrie Hutchinson (Unlicensed) Taylor Smith Lynne Fors Raegan Wiechert Robert Pleshar Peter Martinez Tess Amram, Lisa (TX A&M), wiljanen

Notetaker: Jennifer Eustis, Corrie Hutchinson, Christie Thomas

Links:

...

TopicWhoMeeting NotesRelated JiraDecisions and Actions

Announcements

Ryan
  • CSP #2 (which includes MODDICONV-365) is still in progress
.
Jira Legacy
serverSystem JIRA
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODDICONV-365


Review of Decisions & Action Ryan

The group reviewed the current items with checkboxes in the 'Decisions and Actions' column.  

  • None were marked completed.  All are in still in process.   
  • Ryan added his name to the De-duplication topics first checkbox to clarify current behavior. 


NEW
Feature/Bug Review: 

UXPROD-4704: Stop processing the job after it was canceled by user (FKA MODSOURMAN-970)

Ryan/All

MODSOURMAN-970 was transitioned to UXPROD-4704 after a review by development.  The fixes needed to address the issue involve multiple modules, hence why it is now a new feature. 

Target release is Ramsons and will be included on the priority review spreadsheet Ryan hopes to send out by the end of this week.  

Question : how does this impact or is impacted by the new data slicing functionality?

  • Answer : unknown; Ryan to investigate
Jira Legacy
serverSystem JIRA
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyUXPROD-4704
  •  RYAN : investigate any impacts on or by data slicing

NEW
Q Release Feature Review:

UXPROD-4471: Remove step of initial saving of incoming records to SRS

Ryan/All

UXPROD-4471: Remove step of initial saving of incoming records to SRS

  • Mostly backend work; see ticket and summary below for more details
  • Expected to be part of the Quesnelia release
  • Includes a new 'Incoming record' tab when viewing the DI log for an individual record/entry in the DI main log.  Currently available on folio-snapshot.

    Simplifying the basic DI flow was done not only to get performance improvements and make it easier to troubleshoot, but also resolve or unblock fixing of many bugs. Here's a few problematic areas that should see positive affect from the work completed in UXPROD-4471:
  1. MARC-MARC matching problems -
    • When we saved incoming records as a first step, we would mess up the matching that would happen as part of the Job Profile. For some flows we’d not be able to search for existing record, because we already saved the incoming version
  2. Inconsistent statuses for SRS MARC on DI log page.
    • Often status for SRS MARC would be CREATED even though the action on the linked entity would be UPDATED. We dealt with many bugs related to the status during Poppy, but all those fixes were around displaying the right status, despite the way we tracked it in our system. We now removed many of those workarounds, tested and closed some bugs that are no longer reproducing.
  3. Missing titles in DI log.
    • We now save the incoming records prior to any processing with job profile, therefore if record could be parsed, the title should always be available on the DI log screen
  4. Duplicating 035 identifiers/moving Instance HRID to 035 -
    • These issues are easier to fix now, because the logic moved to mod-inventory, where it is possible to check/deduplicate values.
  5. We added possibility to clean up the Modify MARC Bib behavior -
    • By moving logic to mod-inventory, and not saving the incoming version in SRS, it is possible to perform that action on incoming record, and it will affect existing/mapped entities only when profile is structured in that particular way.
  6. We are no longer piling up records in SRS that are not linked to Instances, Holdings or Authority.
    • Those “incorrect” records missed external identifier, but still were taken into consideration during MARC-MARC matching.
Jira Legacy
serverSystem JIRA
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyUXPROD-4471

Bug Review: 

MODDATAIMP-897 - Adding MARC modifications to single record overlay doesn't respect field protections

  • Discuss expectations of using Modify actions
Ryan/Jennifer/All

Last week, the DI lab started a spreadsheet to track this functionality. The group is still working on this and expects to at the 2/22 meeting.

Previous notes from 2/14:

Expand

Discussion notes:

RT: How are the protections working and how are they expected to work? 

Example from Jennifer Eustis:  Use Case: Export, Transform, Load. Import profile includes a marc modification to delete fields, such Matches on 999 ff $i.  Ideal to have a marc modification to remove unwanted marc fields: 029, 983, etc. Then a match on instance and update of instance. Marc modification at the end of the record results in marc modification. See screenshot of profile.


Comments that marc modifications was implemented with an expectation that marc modifications should be at the beginning of the job and should act on the incoming record. 

That is true, but past conversations in data import subgroup drew out two use cases: 1 to modify an incoming record before any actions are taken and 2) to modify the final srs record after all of the actions are taken. (Delete 9xx data after it is used to update the holdings and item, for example.)

General experience right now is that marc modifications are working as expected with creates, but is not working or working but with corruption (such as the deletion of protected fields) on updates.

RT: Is part of the problem how we are approaching updates vs modifications? Updates are designed to work with FOLIO records and modifications are designed to work on incoming records.  Should updates have the same potential actions as marc modifications applying the logic to the updated record?

Right now dependencies between srs and instance and the explicit nature of the updates on instance vs marc is problematic. It is difficult to understand what is happening with updates. Process is to put them anywhere to see where they work. Whether we are updating srs, instance or both we should be able to do the same thing. 

RT: You will see different behavior from marc modifications depending on its placement in the profile. Need to a deeper dive into how the behavior changes dependent on placement. 

This would be a good candidate for the functionality / documentation audit.  If development dives into this and the DI lab group dives into this, we could then come together to identify the best way forward


Jira Legacy
serverSystem JIRA
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODDATAIMP-897

This would be a good candidate for the functionality / documentation audit. 

If development dives into this and the DI lab group dives into this, we could then come together to identify the best way forward

  •  Development Review
  •  DI Lab Group Review

Missing Action Profiles in Job Profile after Poppy migration: As called out in Poppy Release Notes, there is a known issue that's been observed in which some links to reusable Action Profiles might be missing from Job Profiles after Poppy migration. 


Release notes recommend the following:

  • After migration, review existing Job Profiles to verify they migrated correctly. Pay attention to reusable Action Profiles. In case issues are found, Job Profile can be updated manually. For additional information on links created for that Job Profile - execute script #15 (or follow the link), notify support.

Recommended script will provide list of Action profiles to help users manually recreate any affected Job profiles.


All

MODDINCONV-365 part of CSP#2.  

Previous notes from 2/7:

Expand

There are 2 issues: experience of unlinking post migration and then experience of unlinking during migration. MODICONV-361 is a P1 with the hope to be released in a CSP #1. The MODICONV-365 is being investigated.

It looks like FOLIO system job profiles are being affected in terms of actions being unlinked. 5C saw that the default ISRI overlay wasn't working correctly. When we checked the default system job profile there were no actions profiles.

Ryan confirmed this issue only affects Action profiles. It is difficult to know how common this is. For 361, the behavior seems consistent. But for 365, this seems to be less common and different tenants have the issue occur on different jobs.

This is the 3rd or 4th time that the issue in MODDICONV-361 has appeared during a flower release. The unlinking/linking issues date back several releases.

To gather more information, it is worth keeping the corrupted jobs and create replacements.

A job with no action profiles or an empty job can be run. There are no error messages when such a job is run. This is something we shouldn't be able to do. Perhaps a warning or an error message is needed.


Previous notes from 1/31:

Expand

Overview : Action profiles connected to multiple job profiles are 'unlinked' from job profiles after migration to Poppy.

  • It isn't happening for every library after migration or for every re-used profile.
  • However, it is happening often enough that libraries should be aware and check.  
  • General confusion on how or if the migration to Poppy is causing this issue.  Root cause is not migration, but the migration process does cause the profiles to unlink.
  • Script #15 (noted in the topic column) provides a list of profiles that need to be fixed.  It does not fix the links.  That must be done manually by the library.  
  • The action profiles actually disappear from the job profile, not just 'unlink'.  They must be re-added.

Comments :

  • Lots of work for libraries to recreate job profiles manually.
  • Should be a CSP candidate. High priority for correction.
  • Could be a blocker for some libraries to migrate to Poppy.
  • The "unlinking from one unlinks them all" issue has popped up multiple times.

Sidebar discussion in chat on how job profiles are deleted spurred #42 in the Data Import Issue Tracker.  

Until MODDICONV-361 is fixed, any time a re-used action profile is unlinked in a job profile it will be unlinked in all other job profiles.  Fixing it after migration doesn't stop it from happening again should a re-used action profile be unlinked.  

The development team will be adding new test cases to their workflow to test this type of scenario (re-used profiles) going forward.


Jira Legacy
serverSystem JIRA
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODDICONV-361
Issue specific to unlinking of Action profiles when used by multiple Job profiles after Poppy migration. Ticket now closed and included within Poppy CSP #1.

Jira Legacy
serverSystem JIRA
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODDICONV-365

Issue specific to unlinking of Action profiles during migration as reported by Cornell. Plan to address have been identified and ticket is currently In Code Review.


Partial Matching:

Subject raised by Yael Hod 

Not discussed at the 2/21 meeting.  

Previous notes from 1/31:

Expand

Partial matching, e.g. begins with, ends with, is required but it does not function as it should regardless of how it is configured.

  • System behaves as though it only looks for exact matches.
  • Examples of use include prefixes/suffixes to an 035 added by a vendor or library to designate the source of the record.
  • University of Chicago has had same issue.  Corrie submitted MODDICORE-386 on their behalf.
  • Question as to whether this is a bug or how the system is intended to function.  Documentation is needed.
  • #12 on the Data Import Issue Tracker.  



Jira Legacy
serverSystem JIRA
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODDICORE-386

Ryan will :

  •  Review Jira with Folijet leads to understand current design and identify requirement gaps.

Documentation: The group has identified a need for new, enhanced, or reorganized documentation around Data Import.

  • In a previous session, we agreed that completing a functionality audit spreadsheet would be a good first step
All

Not discussed at the 12/31/ 21 meeting.  


Previous notes from 1/24 meeting:
Expand

In lab session on 1/18/2024, we created a wiki page, Data Import Implementers Topic Tracker, with guidelines on how to contribute and a spreadsheet to track issues. This is based on the work done in the Acquisitions SIG. An archive area was also created where we could archive outdated pages such as the Archived Data Import Implementers and Feature Discussion Topics.

The idea was to put down issues whether they were linked to a Jira issue or not. Some of the important information that we wanted to track was if there was a linked Jira and in particular when the issue was discussed in the working group and the decision(s) made in regard to that issue.

The spreadsheet is still being developed. Before we add more issues, the group in lab wanted to know:

  • Do we adopt this page and spreadsheet? If yes, do we have volunteers to populate it?
  • To make sure this page is maintained, the group suggested that the working group look at it once a month to see what is outstanding or new. Is this a practice we want to adopt?

Discussion: 

A link to the new Data Import topic tracker is at the top of the page. Format was worked on at last week's data import session. 

Q: is this only to track Jira tickets? Or will there be other topics added to the agenda. R: In Acq /RM individuals add stories to the topic tracker and the Jira may only be added later to the spreadsheet. (many think this is a good idea.)

Can reference the Acq/Resource Management implementers topic tracker.

Perhaps add widgets that bring in Jiras automatically based on the tag. 

Q: How to add "Click here and expand" text. R: Put the cursor where you want the text block to begin and use Insert Macro function. Type "Expand" to locate the Expand Macro.

Agreed to use the de-duplication discussion to work on building a useful functionality framework. 


N/A
  •  Get volunteers to create a spreadsheet and start brainstorming - DONE
De-duplication: Continue conversation from previous session to clarify what we expect from de-duplication of field values when a record is loaded into FOLIO via Data Import.All

Not discussed at the 12/31 21 meeting.  

Previous notes from 1/24 meeting:

Expand


Jennifer Eustis and Aaron Neslin found comments in the data-import-processing-core code that provides details about expected behavior for de-duplication.

These comments align with the behavior we are seeing except for when there is duplicate data in the incoming record. Data is being removed from the incoming record on update as well. 

Consensus seems to be that FOLIO should not be de-duplicating within the incoming record unless it is explicitly defined in an import profile.

Q: Is de-duplication something that should be able to be deactivated on a field by field basis? R: Sounds like a reasonable approach. There is also some concern that this would complicate an already complicated situation. 

Possible solution - a tool to deduplicate in another tool rather than within data import instead.

Suggestion to start with the functionality audit. RT can connect with the developers as a part of this audit. 

Q: Are we starting with how we as users expect functionality work or with how the developers expect it to work. R: Really should have both for each feature. Start from perceived / desired functionality of the users and add to it with designed functionality.  Suggestion to provide examples to the developers so that it is clear what we are expecting.

Pilot functionality audit with de-duplication and start with our understanding and then get input from the developers.



MODDATAIMP-879: Data Import removes duplicate 856s in SRS
  •  RYAN: Clarify current behavior of field value de-duplication.
  •  Define desired behavior of field value de-duplication (if different).
  •  Christie Thomas will create some dummy data to illustrate deduping 856s.

...