[SUP-100] QuickMARC fail to load Successfully imported MARC records - Record cannot be found or loaded. Invalid 008 Created: 31/Mar/23  Updated: 22/Jun/23  Resolved: 05/Apr/23

Status: Closed
Project: Support
Components: None
Affects versions: None
Fix versions: None

Type: Bug Priority: P2
Reporter: Theodor Tolstoy (One-Group.se) Assignee: Theodor Tolstoy (One-Group.se)
Resolution: Won't Do Votes: 1
Labels: RRT, Support, back-end, support
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: XML File icpsr-abbrev.xml     XML File icpsr-specific-record.xml     PNG File image-2023-03-31-16-15-06-958.png     PNG File image-2023-04-04-10-34-12-774.png    
Issue links:
Relates
relates to MODQM-326 BE - Make 008 field required for MARC... Closed
relates to UIQM-459 Make 008 field required for MARC bibl... Closed
relates to MODQM-329 MARC bib/holdings/authority 008 Valid... Closed
Sprint:
Development Team: Spitfire
Affected Institution:
TAMU, Villanova University
RCA Group: TBD
Affected releases:
Orchid (R1 2023), Nolana (R3 2022)

 Description   

Overview:
Steps to Reproduce:

  1. Log in to a Nolana or Orchid instance of FOLIO
  2. Import any of the two attached MARC records into FOLIO using the Default - Create instance and SRS MARC Bib profile or similar
  3. Locate the record in inventory and click on "Edit MARC Bibliographic record"

Expected Results:
Quick MARC opens up and allows the user to edit the record
Actual Results:
Inventory comes back with a message saying Record cannot be found or loaded.


Additional Information:
The SRS record is there, verified by API
This behaviour has been verified in both Nolana (Bugfest) and Orchid (FSE-hosted env) as well as in Villanova's Nolana environment.
Interested parties:



 Comments   
Comment by Ann-Marie Breaux (Inactive) [ 03/Apr/23 ]

Hi Khalilah Gambrell There was no dev team on this bug. Not sure if it belongs to Spitfire or Folijet, but I marked as Spitfire since it mentions quickMARC. Please review and change to Folijet if necessary. Thank you!

Comment by Khalilah Gambrell [ 03/Apr/23 ]

Hey Theodor Tolstoy (One-Group.se) - I believe this was also reported to the RRT channel too. Have you spoken to schandel?

Comment by Theodor Tolstoy (One-Group.se) [ 03/Apr/23 ]

I have not spoken to anyone. What is the RRT channel?

Comment by Anya [ 03/Apr/23 ]

Support :  Khalilah Gambrell  should this be mod quick marc or ui quick marc. 

Comment by Pavlo Smahin [ 03/Apr/23 ]

Khalilah Gambrell, the problem is with the 008 fields. We expect to have a 40 characters as a value of the string. We have such expectations because we parse the incoming string to populate separate boxes on UI according to MARC format documentation. But in provided records it's lower length: "220802s1984    miu f a eng d" - just 28 characters. 
How should we handle such case?

Comment by Khalilah Gambrell [ 03/Apr/23 ]

Hey Pavlo Smahin - could we populate the missing positions with default values? Or allow the user to view the record edit and require user to update 008 to save record.

Comment by Pavlo Smahin [ 04/Apr/23 ]

Khalilah Gambrell, how can we understand which positions are missing? On the start, end or in the middle of the string?
UI is expecting to have parsed string to put values into boxes and I’m not sure if they can handle just some string.
cc: @Denys Bohdan can you help with understanding of this?

Comment by Ann-Marie Breaux (Inactive) [ 04/Apr/23 ]

HI Pavlo Smahin Would it make sense to just load the existing data into the first x positions of the 008 field, and then show an error for the 008 in the QM UI and disallow the save of the record until the first manual editor of the record corrected it?

Comment by Denys Bohdan [ 04/Apr/23 ]

Hey Pavlo Smahin Ann-Marie Breaux Khalilah Gambrell on UI we can't populate missing boxes because we get 500 error from BE when loading the record so there's no data to show at all

Comment by Khalilah Gambrell [ 04/Apr/23 ]

Hey Theodor Tolstoy (One-Group.se) - Can Villanova reload the records with valid 008s? OR reload these records without 008 on these records? How many records have this issue?

Comment by Khalilah Gambrell [ 04/Apr/23 ]

Hey Pavlo Smahin - can we improve the error messaging to help detect this issue in the future?

Comment by Khalilah Gambrell [ 04/Apr/23 ]

Ann-Marie Breaux, Pavlo Smahin, and Denys Bohdan

  • I think the short-term solution is to request the library to re-load with corrected 008s/or no 008s.
  • Long-term option: DI does not allow creation/update of a bib record with poorly formed 008

What do you all think?

Comment by Pavlo Smahin [ 04/Apr/23 ]

Khalilah Gambrell, yes we can improve error messaging. I think it has to be done with all types of records, not only bibs.
I agree with the short-term solution. About long-term: I think it's also okay, but if the Folijet team will say that this is something that is better to handle on our side, then we should think about how to handle it on quickMarc UI, if BE will provide just raw string.

Comment by Demian Katz [ 04/Apr/23 ]

Hello, everyone – just to provide an update from the Villanova side: the records which caused this problem were harvested from the public OAI-PMH endpoint for ICPSR. I've done some digging and confirmed that the 008 fields are too short in the original source data, and this is not a problem that was introduced by any of Villanova's tooling. We will contact ICPSR and see if they can fix that on their end. (It looks to me like whitespace in the 008 field is getting collapsed, so multiple consecutive blanks are getting represented as a single blank... there's really no obvious way to fix that after the fact, either on our end or inside FOLIO).

 

From the QuickMARC perspective, would one approach be to simply drop the 008 field if it's an invalid length, with a warning indicating that corrupted data will be lost if changes are saved?

Comment by Theodor Tolstoy (One-Group.se) [ 05/Apr/23 ]

Demian Katz are you trying to pursue the short-term solution Khalilah Gambrell  proposes for now?
If so, you need to make sure that matching is involved in order to prevent duplicates getting added, and hope that Data Import's Matching algorithm can handle bad 008:s.

Khalilah Gambrell agree that the long-term solution is to prevent DI from adding in these records. BUT, how do we deal with the bad records already in FOLIO across libraries? I think allowing them to be loaded into QM needs to be added to the long-term solution as well, since the alternative (checking for bad records accross tenants and handling them) would likely prove complicated

Comment by Khalilah Gambrell [ 05/Apr/23 ]

Hey Demian Katz and Theodor Tolstoy (One-Group.se)
I think the long-term solution might be a combination of what you both stated. I am going to close this issue and create an issue for better handling an imported record with an invalid 008.

Comment by Demian Katz [ 05/Apr/23 ]

Theodor Tolstoy (One-Group.se), as noted above, our first step is to see whether we can get the record provider to correct the problem and give us a valid set of records. I would rather have correct records straight from the source than add an extra step to our ingest process to manipulate the provided records (since the best we could do is strip out the 008 fields, which will prevent VuFind from indexing formats correctly). If we're able to get the record set, I'm not sure what David Burke's plans would be for loading it – whether we would try to match and overlay, or simply delete the old bad records and load the new set fresh.

I also agree with you that QM should be given the ability to edit these types of records. It's obviously better to prevent them from being loaded in the first place, but given that it can happen, it is better to be able to deal with the legacy problems in a convenient way. In general, there is so much badly-formatted MARC out there that I think it's wise to make editors as tolerant as possible. Warnings are great, but avoidable fatal errors are not.

Comment by Khalilah Gambrell [ 05/Apr/23 ]

Will create another story for addressing imported records with invalid 008.

Comment by Demian Katz [ 05/Apr/23 ]

Thanks, Khalilah Gambrell!

Comment by Lisa Sjögren [ 21/Apr/23 ]

Will create another story for addressing imported records with invalid 008.

Khalilah Gambrell Do you have a link for this?

Comment by Brooks Travis [ 24/Apr/23 ]

Lisa Sjögren I think this is it: https://folio-org.atlassian.net/browse/MODQM-329

Comment by Khalilah Gambrell [ 31/May/23 ]

Hey jroot and Demian Katz - this issue has been fixed with Poppy release. 

Comment by Demian Katz [ 31/May/23 ]

Thanks, Khalilah Gambrell!

Comment by Ann-Marie Breaux (Inactive) [ 22/Jun/23 ]

Hi Khalilah Gambrell  This bug came up with another library today, which is still on Morning Glory. I played with this on the Morning Glory reference environment, and tried to import and update the SRS or Instance, based on an 010 match. Neither job succeeded. 

The only other thing I can think of would be to import the corrected Bib as a new instance, then move the holdings and items over to the new instance, and suppress the existing instance. Can you think of any other workaround?

Comment by Khalilah Gambrell [ 22/Jun/23 ]

Hey Ann-Marie Breaux  - The only other option was the one Demian Katz mentioned which was to delete the records from the database and then re-load.

 

Generated at Thu Feb 08 22:23:13 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.