Batch Importer (Bib/Acq) (UXPROD-47)

[MODSOURMAN-714] Legacy 999 (non-ff) fields cause data import failure Created: 05/Jan/22  Updated: 04/Jan/23  Resolved: 13/Mar/22

Status: Closed
Project: mod-source-record-manager
Components: None
Affects versions: None
Fix versions: 3.3.0
Parent: Batch Importer (Bib/Acq)

Type: Bug Priority: P2
Reporter: Ian Hardy Assignee: Aivar Iusupov
Resolution: Done Votes: 0
Labels: data-import, epam-folijet, folijet-support, has-testrail, sprint-135, support
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File 61PRINT160129.mrc     File AMB_999_with_subs_is.mrc     File MODSOURCE-435 Part 1.mp4     File MODSOURCE-435 Part 2.mp4     File record-multiple-999s.mrc     File record-one-999.mrc     File single_local999_test3.mrc    
Issue links:
Cloners
is cloned by MODSOURMAN-723 Legacy 999 (non-ff) fields cause data... Closed
Defines
defines UXPROD-3463 NFR: Data Import R1 2022 Lotus Suppor... Closed
Relates
relates to MODSOURMAN-204 If 999 field without ff indicators al... Closed
relates to MODDATAIMP-340 999 mismatch causes silent data impor... Closed
Sprint:
Story Points: 3
Development Team: Folijet Support
Release: Lotus R1 2022
Potential Workaround: remove or renumber all other 999 fields
Affected Institution:
Lehigh
Epic Link: Batch Importer (Bib/Acq)
RCA Group: Legitimate regression

 Description   

Overview
Using data import to overlay a record with legacy 999 other than the one FOLIO creates with the inventory and srs UUIDs fields fails silently.

Steps to Reproduce:

  1. Log into FOLIO as diku_admin (affects kiwi-bugfest, and probably snapshot-load)
  2. Import the attached file single_local999_test3.mrc of MARC records, using the job profile: Default - Create instance and SRS MARC Bib
  3. Once imported, check to be sure that the local 999 field (with blank indicators) was retained, and a new 999 field (with ff indicators and the Instance and SRS MARC Bib UUIDs) was created.
  4. Go to Inventory and search for that Instance by HRID (do NOT use the hotlink in the Data Import log)
  5. Go to Actions/View source and confirm
    • The existing 999 with blank indicators was retained
    • A new 999 with ff indicators was created and has the Instance UUID in $i and SRS MARC Bib UUID in $s
    • NOTE:
      • This single_local999_test3.mrc file is not currently importing to Kiwi Bugfest (gets stuck in "Running")
      • On Kiwi Bugfest, if you search Instance by HRID insta7155403, you will see the record when it was previously imported, but the View source option is greyed out.
      • Ann-Marie Breaux will check on the hosted ref envs once they come back to life).
      • On Kiwi demo env (https://folio-kiwi.dev.folio.org/), import for this single_local999_test3.mrc file also gets stuck in "Running" - so it appears that even importing to create when there is a 999 field is not working. It previously worked; see MODSOURMAN-204 Closed . When I import the file attached to MODSOURMAN-204 Closed using the Default Create job profile (61PRINT160129 also attached here), it imports cleanly in Kiwi BF and in Kiwi demo env.
  6. Close the source record and the instance detail view. That should leave the 1 search result in your Inventory search results screen
  7. Go to Actions/Save instance UUIDs. That will trigger download of a csv file with that one UUID in it
  8. Go to the Data Export app and upload the csv file
  9. Select the Default job profile to export the associated MARC record.
  10. Doublecheck the exported file to make sure that the non-fff 99 field and the 999 ff field are both in the exported MARC record
  11. Go to Settings/Data Import and create the following profiles
    • Match profile: MARC 001 matching to Instance HRID
    • Field mapping profile for Instance: suppress from Inventory and add a statistical code
    • Action profile: Update Instance (and link above field mapping profile)
    • Job profile: Update instance
      • Match profile
        • If yes, then Update instance
  12. Go to the Data Import app
  13. Upload the exported MARC file
  14. Select the newly-created job profile and import the file
  15. Check Inventory to see if the Instance was updated properly (Suppressed from Inventory, and statistical code added)
  16. Check View source to ensure the SRS MARC record has the non-ff 999 field and the 999 ff UUID field
  17. NOTE: When I did this in Kiwi Bugfest, it worked properly. See https://bugfest-kiwi.folio.ebsco.com/inventory/view/d0cf4f85-f451-44b9-bcd1-891b69b27f3f and job 7806, with log https://bugfest-kiwi.folio.ebsco.com/data-import/job-summary/633ce3f2-e89e-465a-b80c-e3d83c2e5b8f?sort=recordNumber

Expected Behavior
Record is imported cleanly

Additional Information
The following error will come up in mod-source-record-storage:

21:03:50.486 [vert.x-worker-thread-6] ERROR rdChunksKafkaHandler [873231eqId] Can't process the kafka record: 
java.lang.IllegalArgumentException: Invalid UUID string: Y 

Removing all Local 999 fields other than the one with $i and $s for UUIDs will work. Tested on mod-source-record-storage 5.1.9 and 5.0.5

Attached a file to represent the record with one 999 (works) and legacy 999s (does not work)
Notes from Ann-Marie Breaux

  • This is a regression. this previously worked.
  • I wonder if it's related to the data in the Lehigh MARC record. It would be helpful to understand why the Lehigh file is bombing (event for create at this point), but my file is working (for create and update)

 



 Comments   
Comment by Lisa McColl [ 05/Jan/22 ]

Testing on Lehigh University's production server (Juniper) and Kiwi-bugfest:

When a MARC record contains 999 fields other than a 999 that FOLIO exports (i.e.:  a 999 field with indicators "ff" and a subfield i and subfield s example:

=999 ff$i29573076-a7ee-462a-8f9b-2659ab7df23c$s7ca42730-9ba6-4bc8-98d3-f068728504c9)

 

then import fails. I tested this using the "Default - Create instance and SRS MARC Bib" import profile. Job never runs if record contains 999 fields that have indicators and subfields other than the ones FOLIO exports. As soon as these "legacy" 999s were removed, then the MARC record imported successfully.

 

Example of 999 fields that occur in Lehigh University's records

=999
$aSC LVF M698 pt.1$wDEWEYSAN$c1$i39151009428455$d4/22/1996$lL-SPCOLL-A$mLEHIGH$rN$sY$tROOM-USE$u5/25/1995
=999
$aSC LVF M698 pt.2$wDEWEYSAN$c1$i39151009428448$lL-SPCOLL-A$mLEHIGH$rN$sY$tROOM-USE$u9/30/2011

 

Expected behavior: FOLIO will accept records with 999 fields if the indicators are not "ff" and with various subfields

Possible workaround if this is not possible: Move local 999 fields to another 99X field.

 

Comment by Ann-Marie Breaux (Inactive) [ 05/Jan/22 ]

Hi Lisa McColl and Ian Hardy Thank you for writing up this bug. It's definitely a regression because at one point, we were ignoring 999s, so long as their indicators were not ff. ( MODSOURMAN-204 Closed ) I'll add some notes and then we'll work on sorting this out.

Comment by Lisa McColl [ 05/Jan/22 ]

Confirming : the indicators of the 999s were not ff. I posted the 999 example in the comment above, in case it helps. They are also in the MARC files attached.

Comment by Ann-Marie Breaux (Inactive) [ 26/Jan/22 ]

Ann-Marie Breaux add repro steps to the description and then give to devs

Comment by Charlotte Whitt [ 21/Feb/22 ]

From SUPPORT SIG: Ann-Marie Breaux We notice that this is still in OPEN when it's targeted for Lotus. Do you have an update?

cc: Lisa McColl

Comment by Ann-Marie Breaux (Inactive) [ 21/Feb/22 ]

Hi Charlotte Whitt Yes, we'll be working on it in Sprint 135, possibly 134 if we finish releases early enough.

Comment by Ann-Marie Breaux (Inactive) [ 23/Feb/22 ]

Hi Serhii_Nosko Thank you for picking up this ticket. I'll add repro steps, but I can't validate it in current FOLIO since the hosted ref envs are down

Comment by Ann-Marie Breaux (Inactive) [ 23/Feb/22 ]

Hi Serhii_Nosko I added repro steps - confirmed in Kiwi BF and the Kiwi demo env that the Lehigh file does not work for Create OR Update. I used a separate file that we used when we fixed this bug the first time ( MODSOURMAN-204 Closed ) and it worked fine on Kiwi BF for Create and update. Not sure why they would behave differently.

Comment by Ann-Marie Breaux (Inactive) [ 23/Feb/22 ]

Hi Serhii_Nosko OMG - I think I figured it out! See the attached videos MODSOURCE-435 Part 1.mp4 and MODSOURCE-435 Part 2.mp4, and attached file AMB_999_with_subs_is.mrc

I think the reason I was getting different results from Lisa McColl is that her single_local999_test3.mrc file has a subfield s in the 999 field, and FOLIO is trying to interpret that as a UUID. In my original imported file, the 999 field did not have a $s. I edited 1 record from my file to include $s in the 999 (and for good measure, $i also), then tried to import it again, and it failed, with the same error message as Lisa's.

All is attached, and I hope the video is clear. I'll add some details to the expected outcomes. Let me know if you want to look at anything together

Comment by Aivar Iusupov [ 09/Mar/22 ]

Hi Ann-Marie Breaux! Tested it on folio-snapshot hosted env and everything works as it should.

Could we close it as done?

Comment by Ann-Marie Breaux (Inactive) [ 13/Mar/22 ]

Hi Aivar Iusupov Tested on folio-snapshot-2 and all looks good. Created the TestRail. Moved to Awaiting release, so that it will get into Lotus Bugfest

Comment by Ann-Marie Breaux (Inactive) [ 15/Mar/22 ]

Cloned and created a Morning Glory issue as well, linked to MG feature, and closed the new issue. MODSOURMAN-723 Closed

Comment by Ann-Marie Breaux (Inactive) [ 06/Apr/22 ]

Hi Aivar Iusupov and Serhii_Nosko One of the SMEs, Leeda Adkins - tested this on Lotus BF on 21 March and passed it, so closing this issue. Thank you!

Comment by Lisa McColl [ 04/Jan/23 ]

Ann-Marie Breaux - I'm seeing this bug again in our production version of Morning Glory. Can this bug be reopened or does something else need to be done? Thank you.

Generated at Thu Feb 08 23:27:40 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.