Batch Importer (Bib/Acq) (UXPROD-47)

[MODSOURMAN-848] Data Import log displays disconcertingly ambiguous information for successful matches on anything but 999i/instance UUID Created: 30/May/22  Updated: 07/Feb/24

Status: In QA
Project: mod-source-record-manager
Components: None
Affects versions: None
Fix versions: None
Parent: Batch Importer (Bib/Acq)

Type: Bug Priority: P2
Reporter: Lisa Sjögren Assignee: Ryan Taylor
Resolution: Unresolved Votes: 0
Labels: folijet-olamide, known-issue-morningglory, known-issue-nolana, known-issue-orchid, known-issue-poppy, log-status, needs-testrail, support
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified
Environment:

https://bugfest-kiwi.folio.ebsco.com


Attachments: File Lisa please review.mp4     PNG File Screenshot 2024-02-06 at 3.13.23 PM.png     PNG File image-2022-05-30-15-19-42-106.png     PNG File image-2022-05-30-15-20-05-111.png     PNG File image-2022-05-30-15-30-18-227.png    
Issue links:
Blocks
is blocked by MODDATAIMP-743 2 SPIKE: Design approach for DI to re... Open
Defines
defines UXPROD-2742 MARC-MARC matching enhancements: Narr... In Progress
RCA Improvement
to be improved by MODDATAIMP-743 2 SPIKE: Design approach for DI to re... Open
Relates
relates to MODSOURMAN-891 SRS MARC Created when No Create Insta... Closed
relates to MODSOURCE-530 Duplicate records in incoming file ca... Closed
Sprint: Folijet Sprint 184
Story Points: 0
Development Team: Folijet
Release: Quesnelia (R1 2024)
Affected Institution:
Spokane Public Libraries, Trinity College, Stanford University
Epic Link: Batch Importer (Bib/Acq)
RCA Group: Incomplete/missing requirements

 Description   

Overview: If your match profile matches incoming MARC 020 on instance ISBN, and the incoming MARC record matches an instance, the reported status in the log of the SRS record will be Updated if the MARC record also contains a 999-ff-i matching the instance's UUID, but Created if the incoming MARC record does not contain a 999-ff-i. In both cases the status reported for the instance in the log is Updated.

To a librarian wishing to overlay existing FOLIO records with matching incoming MARC record, it is a source of great confusion that, when there is a match and an underlying SRS record, the status of the SRS record is sometimes reported as Updated and sometimes as Created – even if you are matching on the same field, overlaying the same record. This may cause enough uncertainty about whether Data Import works as expected, overlaying the right records when it should, to make the librarian reluctant about using Data Import.

It seems that either

  • there is something wrong in how FOLIO creates/updates existing SRS records when matching on anything but the UUID in 999i
    or
  • the information about SRS update/creation displayed in the log is incorrect or designed in a way that is more confusing than useful to end users (librarians)

It would be great, if as a first step, someone with insight into how Data Import works could review the current behaviour to assess whether librarians can "safely" use Data Import with match profiles even though the log shows this ambiguous result.

Steps to Reproduce:

To test, you need:

  • a FOLIO instance with an underlying SRS record
  • three .mrc files: one with the UUID of the instance in 999i, one without a 999i field but with the ISBN of the instance in 020a, and one with a 999i field that contains a value which is not an instance UUID.

Use the following job profile:
https://bugfest-kiwi.folio.ebsco.com/settings/data-import/job-profiles/view/5d105836-e127-48f4-a942-dec42ef265e6

Try the following three:

  1. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with a FOLIO identifier.
  2. Using a job profile that matches on identifier ISBN, import a MARC record that does not contain a 999i field with a FOLIO identifier.
  3. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with the value "catsarecute"

Expected Results:

  1. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with a FOLIO identifier.
    The incoming 020 of the incoming MARC record successfully matches on the ISBN of the instance. In the log, instance and SRS have status Updated.
  1. Using a job profile that matches on identifier ISBN, import a MARC record that does not contain a 999i field with a FOLIO identifier.
    *The incoming 020 of the incoming MARC record successfully matches on the ISBN of the instance. In the log, instance and SRS have status Updated.
  1. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with the value "catsarecute"
    The incoming 020 of the incoming MARC record successfully matches on the ISBN of the instance. In the log, instance and SRS have status Updated.

Actual Results:

  1. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with a FOLIO identifier.
    The incoming MARC record successfully matches the instance. In the log, instance and SRS have status Updated.
  1. Using a job profile that matches on identifier ISBN, import a MARC record that does not contain a 999i field with a FOLIO identifier.
    The incoming 020 of the incoming MARC record successfully matches on the ISBN of the instance. In the log, the instance has status Updated and the SRS record has status Created.
  1. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with the value "catsarecute"
    The import job hangs.

Looking at the three scenarios above, and the different behaviours, it appears that whatever you actually select as your matchpoint – FOLIO will also match on any existing 999i/instance UUID.

  • If the matchpoint matches and there is also a 999i that matches the UUID of the same instance, the SRS record is reported Updated.
  • If the matchpoint matches and there is also no 999i, the SRS record is reported Created.
  • If the matchpoint is a match and but the 999i does not match the UUID of the same instance, the job hangs.

URL: https://bugfest-kiwi.folio.ebsco.com/
Interested parties: Ben Taylor Robert Roose



 Comments   
Comment by Anya [ 31/May/22 ]

Support : Ann-Marie Breaux 

Comment by Ann-Marie Breaux (Inactive) [ 01/Jun/22 ]

Hi Anya and Lisa Sjögren I want to discuss with the DI Subgroup, so I've changed it to draft and assigned it to myself. Also changed from SUP to UIDATIMP

Comment by Charlotte Whitt [ 20/Jun/22 ]

Support SIG: Ann-Marie Breaux - We notice that this ticket is in Draft and has not FOLJIET Sprint cycle assigned. Will this work be ready for the Morning Glory release?

Comment by Ann-Marie Breaux (Inactive) [ 20/Jun/22 ]

Hi Charlotte Whitt It has the Morning Glory release value and is linked to a Morning Glory feature. Any bugs with dev team = Folijet Support do not get sprints, because they are worked on via a Kanban board instead. I still need to work through this one some more before changing it from Draft to Open.

Comment by Debra Howell [ 27/Jun/22 ]

Ann-Marie Breaux from SUPPORT: the deadline for code freeze for Morning Glory has passed. Would you please update the release date for this issue?

Comment by Debra Howell [ 25/Jul/22 ]

From SUPPORT SIG: Ann-Marie Breaux - what is the status of this bug please?

Comment by Ivan Kryzhanovskyi [ 04/Aug/22 ]

Moved to Nolana, to reduce MG bugfix team workload.

Comment by Charlotte Whitt [ 06/Sep/22 ]

Support SIG: Ann-Marie Breaux is this ticket still to be considered in Draft status?

Comment by Charlotte Whitt [ 19/Sep/22 ]

Support SIG: One more gentle reminder Ann-Marie Breaux re. our question if this ticket is still to be considered in Draft status?

Comment by Charlotte Whitt [ 26/Sep/22 ]

Support SIG: Hi Ann-Marie Breaux, This work is a P2, will you confirm that this work will be included in Nolana? We notice this ticket is still in Draft status?

Comment by Debra Howell [ 24/Oct/22 ]

Ann-Marie Breaux Should the Release be changed to Nolana Bug Fix since the code freeze for Nolana has already happened? Will this be fixed in Nolana?

Comment by Ann-Marie Breaux (Inactive) [ 09/Nov/22 ]

Hi Debra Howell  Yes, I've changed it. The situation with statuses has changed some since Lisa wrote up this bug, and I want to get additional feedback from her, so I'm not sure if we will fix this in Nolana or Orchid. If not in Nolana, we will mark it as a known issue.

Lisa Sjögren  Please review the attached recording Lisa please review.mp4

Based on the 3 scenarios you outline, I want to be sure that we agree on what the status info should be in the import log:

  • Scenario 1: Match on ISBN, Incoming record has no 999 ff $i
    • SRS status = Created, but since the MARC already exists in FOLIO SRS, it should be Updated. Is that correct? That makes sense to me, and I think, to users in general
    • Instance status = Updated - no change neede
  • Scenario 2: Match on ISBN, Incoming record has 999 ff $i with Instance UUID 
    • SRS status = Updated - no change needed
    • Instance status = Updated - no change needed
  • Scenario 3: Match on ISBN, Incoming record has 999 ff $i with non-UUID data in it
    • SRS status = none
    • Instance status = none
    • FOLIO reserves the MARC 999 ff field for FOLIO-only use. If FOLIO encounters anything other than SRS MARC and Inventory Instance UUIDs in a 999 ff, it cannot parse the information correctly.
      • If the library needs to park "catsarecute" someplace in the record, they can put it in any repeatable MARC field other than 999 ff. They could use 930 or 590. They could even use 999 ee if they want, just not 999 ff.  
    • Right now, import shows that the original file contained 3 records, but the 999 ff with bad data confounds it so much that the record completely disappears from the log, and no attempt is made to match or take any actions. What would be your preferred outcome in the status that the user sees?
      • Nothing in the log. The only way the user knows something went wrong is 3 (in the original file) - 2 (in the final log) = 1 (was a problem)
      • A line for record 3, with an error status of some sort, maybe indicating unexpected data encountered in a FOLIO restricted field? It would likely be an error for SRS and nothing or Discarded for the the Instance
      • Something else?

Once I have this feedback from you, I'll discuss more with the developers, and we'll figure out what's possible, and when

Thank you!

Comment by Kateryna Senchenko [ 09/Nov/22 ]

Hi Ann-Marie Breaux, Lisa Sjögren,

Right now we rely on 999 ff UUID values to determine whether the record already exists in SRS. Hence, if this value is missing the record is treated as a new one and is saved with newly assigned 999 ff values, which is then reflected in the status. I'm not sure we can change or bypass this mechanism in scope of a bugfix, we'll need to discuss it with Olamide Kolawole and get an architectural perspective. 

However, we could deal with Scenario 3 and make a Nolana bugfix for that - add additional validation on the values passed in 999 ff fields and complete the job with an error. Please let us know what you think. Thank you!

Comment by Charlotte Whitt [ 14/Nov/22 ]

Support SIG: I this work still going to be ready for Nolana Bugfix release. Also the ticket is still in Draft?

Comment by Ann-Marie Breaux (Inactive) [ 21/Nov/22 ]

Hi Charlotte Whitt and Lisa Sjögren We're moving this to Orchid, and maybe even Poppy - we know we still need to clean up some of the Import log statuses. In this issue
Scenario 1: is OK
Scenario 2: need to reconcile the SME expectation of "Updated" versus the system currently showing "Created" for SMS
Scenario 3: need to ensure the job doesn't hang, and that we validate the 999 ff, so that the user gets an error message if they try to use that field for anything other than UUIDs

Comment by Anya [ 13/Feb/23 ]

Support: - Ann-Marie Breaux  is this still in Draft?

Comment by Ann-Marie Breaux (Inactive) [ 06/Mar/23 ]

Spike in Poppy to design a solution that will resolve this issue; once that is designed, this bug will be fixed in Quesnelia

Comment by Debra Howell [ 07/Aug/23 ]

From SUPPORT SIG: Kateryna Senchenko and Ann-Marie Breaux please list what Jira this ticket is "BLOCKED" on so that we can track. If it is not blocked by another Jira issue, please change the status.

Comment by Anya [ 16/Oct/23 ]

From SUPPORT SIG: Kateryna Senchenko and Ann-Marie Breaux please list what Jira this ticket is "BLOCKED" on so that we can track. If it is not blocked by another Jira issue, please change the status.

Comment by Charlotte Whitt [ 20/Nov/23 ]

Support SIG: Hi Ann-Marie Breaux - the ticket which is blocking the work on this bug is MODDATAIMP-743 Open , and it is in Open state but not scheduled yet. Do you have plans for when FOLJIET will be working on these issues?

Comment by Anya [ 11/Dec/23 ]

Support: The blocked by has been added. 

Comment by Ann-Marie Breaux (Inactive) [ 12/Dec/23 ]

Ryan Taylor Please review after the MARC matching feature is complete, to see if this is still an issue. Between the matching updates and the log status updates, we may have cleared up some (most?) of the confusion or ambiguity in the log

Comment by Anya [ 29/Jan/24 ]

SUpport: Ryan Taylor - Checking in on this . 

Comment by Kateryna Senchenko [ 06/Feb/24 ]

Tested on folio-snapshot:

1 scenario - works as expected

2 scenario - works as expected

3 scenario - import completes with errors showing the validations issue (999 ff i cannot contain a string that is not a UUID). Ryan Taylor, please confirm that is an expected behavior and let’s correct the description of this issue.

Moving this ticket in QA to create/update test case and verify all scenarios are working as expected.

Generated at Thu Feb 08 22:22:30 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.