...
Requirements details Here Additional discussion topics in Subgroup parking lot
Attendees: Ellis Butler Peter Martinez Ryan Taylor Christie Thomas Tess Amram Lynne Fors Corrie Hutchinson (Unlicensed) Kim Wiljanen Jennifer Eustis Yael Hod Sara Colglazier Robert Pleshar
Notetaker: Christie Thomas
Links:
- Data Import Topic Tracker
- Poppy import planning dashboard
- Poppy timeline
- Quesnelia import planning dashboard
- Quesnelia timeline
- Folijet Current Development Board
- Folijet (Data import) Bug Dashboard
...
| Topic | Who | Meeting Notes | Related Jira | Decisions and Actions | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Announcements:
| Ryan/all | Poppy CSP 2 Release notes: https://folio-org.atlassian.net/wiki/spaces/REL/pages/39944194/Critical+Service+Patch+2
Poppy CSP#2 Test execution results: https://foliotest.testrail.io/index.php?/plans/view/2382 | Ramsons Priorities Exercise | Ryan |
| Demo: UXPROD-4303: Set instance/bib record for deletion | Ryan |
| Review issue reported by Christie: Quantity Electronic seems to always be zero (In Poppy with CSP 1) | Christie Thomas/all |
| Expand | ||||||||
|---|---|---|---|---|---|---|---|---|
When creating electronic orders are being created through data import, the electronic resource quantity is not being mapped from marc or from a default value in the mapping profile and the funds are not being encumbered. Test in Poppy CSP1 in local UChicago environment and in Poppy bugfest. See
For Stanford, Acquisition method is not mapping for Purchase. They are also not getting quantity or encumbrances. Also, order type is not mapping for Purchase when it is provided as a default in the import profile for orders. |
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Feature/Bug Review:
UXPROD-4704: Stop processing the job after it was canceled by user (FKA MODSOURMAN-970)Previous notes from 2/21:
| Expand |
|---|
MODSOURMAN-970 was transitioned to UXPROD-4704 after a review by development. The fixes needed to address the issue involve multiple modules, hence why it is now a new feature. Target release is Ramsons and will be included on the priority review spreadsheet Ryan hopes to send out by the end of this week. Question : how does this impact or is impacted by the new data slicing functionality?
|
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Answer 3/6:
it will not have any effect to data-slicing, slicing happens prior to the processing in mod-data-import. This feature will be focused on stopping the processing of records that starts in mod-source-record-manager and further.
MODDATAIMP-897 - Adding MARC modifications to single record overlay doesn't respect field protections
- Discuss expectations of using Modify actions
A number of issues were seen with doing updates. It seems there might be regressions along with functionality that doesn't work:
On Create:
- Adding a string to the beginning or end doesn't work
- Adding a field with multiple subfields works EXCEPT for the indicators which aren't mapped
On Update:
- Adding a string to the beginning or end doesn't work
- Adding a field with multiple subfields works EXCEPT for the indicators which aren't mapped
- Modification at the end of the file didn't remove the field from the incoming record and that field mapped to be removed was in the existing srs record at the completion of the job
- Modifications at the end of a job don't seem to work
Logs:
- On Update, encountered a number of errors such as
- incoming file may contain duplicates when the file only had 1 record
- 2 rows in the log when there was only 1 record in the file and where the summary had 1 update and 1 no action error
If all this work is being done, should we create jiras for all the ways in which marc modifications don't work? Should we create jiras for modifications that worked in orchid and that no longer work in poppy? It makes sense to create jiras for things that were working that aren't working in Poppy. Let's hold off on what marc modifications should work until we get the developers' findings.
Previous notes from 2/22 and 2/14:Discussion notes:
Last week, the DI lab started a spreadsheet to track this functionality. The group is still working on this and expects to at the 2/22 meeting.
RT: How are the protections working and how are they expected to work?
Example from Jennifer Eustis: Use Case: Export, Transform, Load. Import profile includes a marc modification to delete fields, such Matches on 999 ff $i. Ideal to have a marc modification to remove unwanted marc fields: 029, 983, etc. Then a match on instance and update of instance. Marc modification at the end of the record results in marc modification. See screenshot of profile.
Comments that marc modifications was implemented with an expectation that marc modifications should be at the beginning of the job and should act on the incoming record.
That is true, but past conversations in data import subgroup drew out two use cases: 1 to modify an incoming record before any actions are taken and 2) to modify the final srs record after all of the actions are taken. (Delete 9xx data after it is used to update the holdings and item, for example.)
General experience right now is that marc modifications are working as expected with creates, but is not working or working but with corruption (such as the deletion of protected fields) on updates.
RT: Is part of the problem how we are approaching updates vs modifications? Updates are designed to work with FOLIO records and modifications are designed to work on incoming records. Should updates have the same potential actions as marc modifications applying the logic to the updated record?
Right now dependencies between srs and instance and the explicit nature of the updates on instance vs marc is problematic. It is difficult to understand what is happening with updates. Process is to put them anywhere to see where they work. Whether we are updating srs, instance or both we should be able to do the same thing.
RT: You will see different behavior from marc modifications depending on its placement in the profile. Need to a deeper dive into how the behavior changes dependent on placement.
This would be a good candidate for the functionality / documentation audit. If development dives into this and the DI lab group dives into this, we could then come together to identify the best way forward
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
This would be a good candidate for the functionality / documentation audit.
If development dives into this and the DI lab group dives into this, we could then come together to identify the best way forward
- Development Review
- DI Lab Group Review
Missing Action Profiles in Job Profile after Poppy migration: As called out in Poppy Release Notes, there is a known issue that's been observed in which some links to reusable Action Profiles might be missing from Job Profiles after Poppy migration.
Release notes recommend the following:
- After migration, review existing Job Profiles to verify they migrated correctly. Pay attention to reusable Action Profiles. In case issues are found, Job Profile can be updated manually. For additional information on links created for that Job Profile - execute script #15 (or follow the link), notify support.
MODDINCONV-365 part of CSP#2.
Previous notes from 2/7:
| Expand |
|---|
There are 2 issues: experience of unlinking post migration and then experience of unlinking during migration. MODICONV-361 is a P1 with the hope to be released in a CSP #1. The MODICONV-365 is being investigated. It looks like FOLIO system job profiles are being affected in terms of actions being unlinked. 5C saw that the default ISRI overlay wasn't working correctly. When we checked the default system job profile there were no actions profiles. Ryan confirmed this issue only affects Action profiles. It is difficult to know how common this is. For 361, the behavior seems consistent. But for 365, this seems to be less common and different tenants have the issue occur on different jobs. This is the 3rd or 4th time that the issue in MODDICONV-361 has appeared during a flower release. The unlinking/linking issues date back several releases. To gather more information, it is worth keeping the corrupted jobs and create replacements. A job with no action profiles or an empty job can be run. There are no error messages when such a job is run. This is something we shouldn't be able to do. Perhaps a warning or an error message is needed. |
| Expand |
|---|
Overview : Action profiles connected to multiple job profiles are 'unlinked' from job profiles after migration to Poppy.
Comments :
Sidebar discussion in chat on how job profiles are deleted spurred #42 in the Data Import Issue Tracker. Until MODDICONV-361 is fixed, any time a re-used action profile is unlinked in a job profile it will be unlinked in all other job profiles. Fixing it after migration doesn't stop it from happening again should a re-used action profile be unlinked. The development team will be adding new test cases to their workflow to test this type of scenario (re-used profiles) going forward. |
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Partial Matching:
Not discussed at the 2/21 meeting.
Previous notes from 1/31:
| Expand |
|---|
Partial matching, e.g. begins with, ends with, is required but it does not function as it should regardless of how it is configured.
|
Jira Legacy server System JIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODDICORE-386
Ryan will :
- Review Jira with Folijet leads to understand current design and identify requirement gaps.
Documentation: The group has identified a need for new, enhanced, or reorganized documentation around Data Import.
- In a previous session, we agreed that completing a functionality audit spreadsheet would be a good first step
Not discussed at the 2/21 meeting.
Previous notes from 1/24 meeting:| Expand |
|---|
In lab session on 1/18/2024, we created a wiki page, Data Import Topic Tracker, with guidelines on how to contribute and a spreadsheet to track issues. This is based on the work done in the Acquisitions SIG. An archive area was also created where we could archive outdated pages such as the Archived Data Import Implementers and Feature Discussion Topics. The idea was to put down issues whether they were linked to a Jira issue or not. Some of the important information that we wanted to track was if there was a linked Jira and in particular when the issue was discussed in the working group and the decision(s) made in regard to that issue. The spreadsheet is still being developed. Before we add more issues, the group in lab wanted to know:
Discussion: A link to the new Data Import topic tracker is at the top of the page. Format was worked on at last week's data import session. Q: is this only to track Jira tickets? Or will there be other topics added to the agenda. R: In Acq /RM individuals add stories to the topic tracker and the Jira may only be added later to the spreadsheet. (many think this is a good idea.) Can reference the Acq/Resource Management implementers topic tracker. Perhaps add widgets that bring in Jiras automatically based on the tag. Q: How to add "Click here and expand" text. R: Put the cursor where you want the text block to begin and use Insert Macro function. Type "Expand" to locate the Expand Macro. Agreed to use the de-duplication discussion to work on building a useful functionality framework. |
- Get volunteers to create a spreadsheet and start brainstorming - DONE
Ryan has discussed this with the team. He will get this in writing and will share this when done. Christie did some work as well in Poppy bugfest.
Not discussed at the 2/21 meeting.
Previous notes from 1/24 meeting:
Jennifer Eustis and Aaron Neslin found comments in the data-import-processing-core code that provides details about expected behavior for de-duplication.
These comments align with the behavior we are seeing except for when there is duplicate data in the incoming record. Data is being removed from the incoming record on update as well.
Consensus seems to be that FOLIO should not be de-duplicating within the incoming record unless it is explicitly defined in an import profile.
Q: Is de-duplication something that should be able to be deactivated on a field by field basis? R: Sounds like a reasonable approach. There is also some concern that this would complicate an already complicated situation.
Possible solution - a tool to deduplicate in another tool rather than within data import instead.
Suggestion to start with the functionality audit. RT can connect with the developers as a part of this audit.
Q: Are we starting with how we as users expect functionality work or with how the developers expect it to work. R: Really should have both for each feature. Start from perceived / desired functionality of the users and add to it with designed functionality. Suggestion to provide examples to the developers so that it is clear what we are expecting.
Pilot functionality audit with de-duplication and start with our understanding and then get input from the developers.
- RYAN: Clarify current behavior of field value de-duplication.
- Define desired behavior of field value de-duplication (if different).
- Christie Thomas will create some dummy data to illustrate deduping 856s.
Upcoming meetings/agenda topics:
...
CSP 2 is out and CSP 3 is scheduled for March 29. Enhancement request: Use case for adding today's date to an Admin note: I want to be able to track changes made when updating e-resource records by using an admin note with the ###TODAY### format used in the Instance Cataloged date field. would want to be able to code it into an Import job. For example, Adding a new Admin note with "Updated ###TODAY###" and then today's date would be automatically populated. Or replacing a previous Admin Note with today's updated date. | ||||||||||||||||||||||||
| Ramsons Priorities Exercise | Ryan |
Q: Are we limited to just the features in the spreadsheet or can we add features in the spreadsheet. A: Intention was just the features, but open to others. Send Ryan a message to add something. Or can add a line and highlight it in yellow. Q: How do we consider things that don't have adequate representation in Jira, such as MARC modifications. A: We need to discuss in this group on how to get those things into Jira as features or bugs. Team is working on some Marc Modifications updates. Hopefully we can look at this next week. Will help us what is still missing. Q: Should we be using the voting functionality that is built into Jira. A: Ryan has not used it, but Jennifer Eustis has started voting for issues. As has Lynne Fors. And, Acq Sig used the voting functionality. Likes the idea of using the votes, but maybe reorganizing the approach. Pause on the activity and revisit plan of using voting. Feel free to review the issues in the spreadsheet. Ryan will come up with a new plan. | ||||||||||||||||||||||
| Demo: UXPROD-4303: Set instance/bib record for deletion | Ryan | Set bib record for deletion is in place in snapshot. New set record for deletion action. Need a specific permission to have access to this. 9Separate from inventory All permissions. Click the action to suppress the record from discovery, staff suppress the record, and set the LDR 05 to d. New SRS property "deleted" set to true. Staff suppress will now be defaulted to No, so will not show up in search. Q: What happens to attached orders? A: Nothing right now, but Ryan will check on impact on associated records. Q: Can you do this batch with a delete file in Data Import? A: No. Just an individual, manual action to take. Step 1 of longer term plans for full deletion options. Q: Are other steps going to be continued before this becomes part of a release? A: Will be included as is in Quesnalia. Holdings and items should not be affected as part of this release. Concern: Hacving holdings and items available with the instance being deleted. Cannot imagine a situation in which you want to delete an instance and still have the holdings and item information available with a search. Also, if an item is checked out, it should block you from deleting a bib. Or on hold. Or in course reserves. Not interactive link with inventory or circulation control. Follow up Q: Would there be a script available to get this added to Instance records a library has already marked for delete? Q: What is the push this out in its partially developed state? A: Someone would use it to delete duplicate instances in the system. These things usually do not have holdings or items. Q: What would happen if a deleted record was matched and updated as a part of a data import? A: Believe that these records are not discoverable via data import matching and updates, but will check on that. Q: How do holdings and items show up in search if the instance is not available in the search? A: notetaker is unclear of the answer to this question Q: Can you reverse this process or undelete things? A: Manually edit the instance to undo the suppression, but cannot edit the marc once it has been set to deletion. Triggered by the status to deletion. Question about whether we need to revisit this. This functionality should be robustly described in the documentation so users fully understand the implications of deleting something. Suggestion: The warning toast should be delete rather than just suppressed from discovery and staff suppressed because there are implications like not being able to edit the srs marc Suggestion to postpone until Ramsons. |
| |||||||||||||||||||||
| Review issue reported by Christie: Quantity Electronic seems to always be zero (In Poppy with CSP 1) | Christie Thomas/all | Open Questions:
|
| |||||||||||||||||||||
| MARC Modification Testing | Jennifer Eustis | Link to spreadsheet:
| ||||||||||||||||||||||
| Notes from previous meetings... | ||||||||||||||||||||||||
Feature/Bug Review: UXPROD-4704: Stop processing the job after it was canceled by user (FKA MODSOURMAN-970) | Ryan/All | Previous notes from 2/21:
|
|
| ||||||||||||||||||||
| Bug Review: MODDATAIMP-897 - Adding MARC modifications to single record overlay doesn't respect field protections
| Ryan/Jennifer/All | A number of issues were seen with doing updates. It seems there might be regressions along with functionality that doesn't work: On Create:
On Update:
Logs:
If all this work is being done, should we create jiras for all the ways in which marc modifications don't work? Should we create jiras for modifications that worked in orchid and that no longer work in poppy? It makes sense to create jiras for things that were working that aren't working in Poppy. Let's hold off on what marc modifications should work until we get the developers' findings. Previous notes from 2/22 and 2/14:
|
| This would be a good candidate for the functionality / documentation audit. If development dives into this and the DI lab group dives into this, we could then come together to identify the best way forward
| ||||||||||||||||||||
Missing Action Profiles in Job Profile after Poppy migration: As called out in Poppy Release Notes, there is a known issue that's been observed in which some links to reusable Action Profiles might be missing from Job Profiles after Poppy migration. Release notes recommend the following:
Recommended script will provide list of Action profiles to help users manually recreate any affected Job profiles. | All | MODDINCONV-365 part of CSP#2. Previous notes from 2/7:
|
| |||||||||||||||||||||
Partial Matching: | Subject raised by Yael Hod | Not discussed at the 2/21 meeting. Previous notes from 1/31:
|
| Ryan will :
| ||||||||||||||||||||
Documentation: The group has identified a need for new, enhanced, or reorganized documentation around Data Import.
| All | Not discussed at the 2/21 meeting. Previous notes from 1/24 meeting:
| N/A |
| ||||||||||||||||||||
| De-duplication: Continue conversation from previous session to clarify what we expect from de-duplication of field values when a record is loaded into FOLIO via Data Import. | All | Ryan has discussed this with the team. He will get this in writing and will share this when done. Christie did some work as well in Poppy bugfest. Not discussed at the 2/21 meeting. Previous notes from 1/24 meeting:
| MODDATAIMP-879: Data Import removes duplicate 856s in SRS |
|
Upcoming meetings/agenda topics:
Chat:
13:09:45 From Lynne Fors to Everyone:
Use case for adding today's date to an Admin note: I want to be able to track changes made when updating e-resource records by using an admin note with the ###TODAY### format used in the Instance Cataloged date field. would want to be able to code it into an Import job. For example, Adding a new Admin note with "Updated ###TODAY###" and then today's date would be automatically populated. Or replacing a previous Admin Note with today's updated date.
13:15:22 From Lynne Fors to Everyone:
| Lref gdrive file | ||
|---|---|---|
|
13:20:52 From Yael to Everyone:
am i the only one not hearing anything?
13:21:11 From Lynne Fors to Everyone:
Replying to "am i the only one no..."
My audio is fine.
13:21:35 From Yael to Everyone:
Replying to "am i the only one no..."
ugh, mine disapeared
13:21:47 From Jennifer Eustis to Everyone:
Replying to "am i the only one no..."
Mine is ok too
13:23:08 From Jennifer Eustis to Everyone:
I've started voted for issues
13:23:15 From Yael to Everyone:
Replying to "am i the only one no..."
ok, back now
13:23:30 From Lynne Fors to Everyone:
I have started to vote too
13:23:31 From scolglaz to Everyone:
We do it in ACQ SIG
13:24:22 From scolglaz to Everyone:
We are talking about the thumbs up thing, right?
13:24:28 From Jennifer Eustis to Everyone:
Reacted to "We are talking about..." with 💯
13:28:03 From Lynne Fors to Everyone:
Reacted to "We are talking about..." with 💯
13:33:02 From Kimberly Wiljanen to Everyone:
What happens to any orders that might be attached?
13:33:27 From Lynne Fors to Everyone:
Can you do this in batch with a delete file in Data Import?
13:34:59 From Yael to Everyone:
When is this planned for?
13:35:05 From Lynne Fors to Everyone:
Would there be a script available to get this added to Instance records a library has already marked for delete?
13:36:59 From Jennifer Eustis to Everyone:
Or on hold
13:37:12 From Lynne Fors to Everyone:
This would be a problem with Courses as it makes a copy of the Instance but doesn't interact with Inventory
13:37:30 From Lynne Fors to Everyone:
Replying to "Or on hold"
Or on course reserve
13:45:50 From Christie Thomas (she/her) to Everyone:
I captured this question in the notes, but I am not really sure about the answer, so there is no answer to this question provided. Feel free to add to my notes!
13:46:07 From Lynne Fors to Everyone:
Is Staff Suppress visibility a prermission that needs to be added?
13:46:58 From Lynne Fors to Everyone:
Can you unmark for delete and reverse the process?
13:48:45 From Lynne Fors to Everyone:
We definitely should be able to reverse the process.
13:49:05 From Christie Thomas (she/her) to Everyone:
I missed your first question, Jennifer so it is not in the notes. About the toast / warning.
13:50:21 From Jennifer Eustis to Everyone:
The warning toast should be delete rather than just suppressed from discovery and staff suppressed because there are implications like not being able to edit the srs marc
13:50:28 From Robert Pleshar (he/him) Library to Everyone:
Reacted to "The warning toast sh..." with 👍
13:50:44 From Jennifer Eustis to Everyone:
+1 Christie
13:50:49 From Lynne Fors to Everyone:
Reacted to "The warning toast sh..." with 👍
13:50:53 From Lynne Fors to Everyone:
Reacted to "+1 Christie" with ➕
13:51:12 From Jennifer Eustis to Everyone:
And I would prefer to delay this to Ramsons so that we can consider all the implications and what we want for the 1st iteration of this functionality
13:53:53 From Yael to Everyone:
Reacted to "And I would prefer t..." with 👍
13:54:04 From Yael to Everyone:
Reacted to "+1 Christie" with ➕
13:54:49 From Peter Martinez to Everyone:
Reacted to "And I would prefer t..." with 👍
13:55:00 From Lynne Fors to Everyone:
Reacted to "And I would prefer t..." with 👍
13:59:35 From Yael to Everyone:
I'm sorry. I have another meeting to get to.