[SUP-100] QuickMARC fail to load Successfully imported MARC records - Record cannot be found or loaded. Invalid 008 Created: 31/Mar/23 Updated: 22/Jun/23 Resolved: 05/Apr/23 |
|
| Status: | Closed |
| Project: | Support |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Bug | Priority: | P2 |
| Reporter: | Theodor Tolstoy (One-Group.se) | Assignee: | Theodor Tolstoy (One-Group.se) |
| Resolution: | Won't Do | Votes: | 1 |
| Labels: | RRT, Support, back-end, support | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||
| Issue links: |
|
||||||||||||||||
| Sprint: | |||||||||||||||||
| Development Team: | Spitfire | ||||||||||||||||
| Affected Institution: |
TAMU, Villanova University
|
||||||||||||||||
| RCA Group: | TBD | ||||||||||||||||
| Affected releases: |
Orchid (R1 2023), Nolana (R3 2022)
|
||||||||||||||||
| Description |
|
Overview:
Expected Results:
|
| Comments |
| Comment by Ann-Marie Breaux (Inactive) [ 03/Apr/23 ] |
|
Hi Khalilah Gambrell There was no dev team on this bug. Not sure if it belongs to Spitfire or Folijet, but I marked as Spitfire since it mentions quickMARC. Please review and change to Folijet if necessary. Thank you! |
| Comment by Khalilah Gambrell [ 03/Apr/23 ] |
|
Hey Theodor Tolstoy (One-Group.se) - I believe this was also reported to the RRT channel too. Have you spoken to schandel? |
| Comment by Theodor Tolstoy (One-Group.se) [ 03/Apr/23 ] |
|
I have not spoken to anyone. What is the RRT channel? |
| Comment by Anya [ 03/Apr/23 ] |
|
Support : Khalilah Gambrell should this be mod quick marc or ui quick marc. |
| Comment by Pavlo Smahin [ 03/Apr/23 ] |
|
Khalilah Gambrell, the problem is with the 008 fields. We expect to have a 40 characters as a value of the string. We have such expectations because we parse the incoming string to populate separate boxes on UI according to MARC format documentation. But in provided records it's lower length: "220802s1984 miu f a eng d" - just 28 characters. |
| Comment by Khalilah Gambrell [ 03/Apr/23 ] |
|
Hey Pavlo Smahin - could we populate the missing positions with default values? Or allow the user to view the record edit and require user to update 008 to save record. |
| Comment by Pavlo Smahin [ 04/Apr/23 ] |
|
Khalilah Gambrell, how can we understand which positions are missing? On the start, end or in the middle of the string? |
| Comment by Ann-Marie Breaux (Inactive) [ 04/Apr/23 ] |
|
HI Pavlo Smahin Would it make sense to just load the existing data into the first x positions of the 008 field, and then show an error for the 008 in the QM UI and disallow the save of the record until the first manual editor of the record corrected it? |
| Comment by Denys Bohdan [ 04/Apr/23 ] |
|
Hey Pavlo Smahin Ann-Marie Breaux Khalilah Gambrell on UI we can't populate missing boxes because we get 500 error from BE when loading the record so there's no data to show at all |
| Comment by Khalilah Gambrell [ 04/Apr/23 ] |
|
Hey Theodor Tolstoy (One-Group.se) - Can Villanova reload the records with valid 008s? OR reload these records without 008 on these records? How many records have this issue? |
| Comment by Khalilah Gambrell [ 04/Apr/23 ] |
|
Hey Pavlo Smahin - can we improve the error messaging to help detect this issue in the future? |
| Comment by Khalilah Gambrell [ 04/Apr/23 ] |
|
Ann-Marie Breaux, Pavlo Smahin, and Denys Bohdan
What do you all think? |
| Comment by Pavlo Smahin [ 04/Apr/23 ] |
|
Khalilah Gambrell, yes we can improve error messaging. I think it has to be done with all types of records, not only bibs. |
| Comment by Demian Katz [ 04/Apr/23 ] |
|
Hello, everyone – just to provide an update from the Villanova side: the records which caused this problem were harvested from the public OAI-PMH endpoint for ICPSR. I've done some digging and confirmed that the 008 fields are too short in the original source data, and this is not a problem that was introduced by any of Villanova's tooling. We will contact ICPSR and see if they can fix that on their end. (It looks to me like whitespace in the 008 field is getting collapsed, so multiple consecutive blanks are getting represented as a single blank... there's really no obvious way to fix that after the fact, either on our end or inside FOLIO).
From the QuickMARC perspective, would one approach be to simply drop the 008 field if it's an invalid length, with a warning indicating that corrupted data will be lost if changes are saved? |
| Comment by Theodor Tolstoy (One-Group.se) [ 05/Apr/23 ] |
|
Demian Katz are you trying to pursue the short-term solution Khalilah Gambrell proposes for now? Khalilah Gambrell agree that the long-term solution is to prevent DI from adding in these records. BUT, how do we deal with the bad records already in FOLIO across libraries? I think allowing them to be loaded into QM needs to be added to the long-term solution as well, since the alternative (checking for bad records accross tenants and handling them) would likely prove complicated |
| Comment by Khalilah Gambrell [ 05/Apr/23 ] |
|
Hey Demian Katz and Theodor Tolstoy (One-Group.se) |
| Comment by Demian Katz [ 05/Apr/23 ] |
|
Theodor Tolstoy (One-Group.se), as noted above, our first step is to see whether we can get the record provider to correct the problem and give us a valid set of records. I would rather have correct records straight from the source than add an extra step to our ingest process to manipulate the provided records (since the best we could do is strip out the 008 fields, which will prevent VuFind from indexing formats correctly). If we're able to get the record set, I'm not sure what David Burke's plans would be for loading it – whether we would try to match and overlay, or simply delete the old bad records and load the new set fresh. I also agree with you that QM should be given the ability to edit these types of records. It's obviously better to prevent them from being loaded in the first place, but given that it can happen, it is better to be able to deal with the legacy problems in a convenient way. In general, there is so much badly-formatted MARC out there that I think it's wise to make editors as tolerant as possible. Warnings are great, but avoidable fatal errors are not. |
| Comment by Khalilah Gambrell [ 05/Apr/23 ] |
|
Will create another story for addressing imported records with invalid 008. |
| Comment by Demian Katz [ 05/Apr/23 ] |
|
Thanks, Khalilah Gambrell! |
| Comment by Lisa Sjögren [ 21/Apr/23 ] |
Khalilah Gambrell Do you have a link for this? |
| Comment by Brooks Travis [ 24/Apr/23 ] |
|
Lisa Sjögren I think this is it: https://folio-org.atlassian.net/browse/MODQM-329 |
| Comment by Khalilah Gambrell [ 31/May/23 ] |
|
Hey jroot and Demian Katz - this issue has been fixed with Poppy release. |
| Comment by Demian Katz [ 31/May/23 ] |
|
Thanks, Khalilah Gambrell! |
| Comment by Ann-Marie Breaux (Inactive) [ 22/Jun/23 ] |
|
Hi Khalilah Gambrell This bug came up with another library today, which is still on Morning Glory. I played with this on the Morning Glory reference environment, and tried to import and update the SRS or Instance, based on an 010 match. Neither job succeeded. The only other thing I can think of would be to import the corrected Bib as a new instance, then move the holdings and items over to the new instance, and suppress the existing instance. Can you think of any other workaround? |
| Comment by Khalilah Gambrell [ 22/Jun/23 ] |
|
Hey Ann-Marie Breaux - The only other option was the one Demian Katz mentioned which was to delete the records from the database and then re-load.
|