Page status: Status colour Yellow title In progress
Table of Contents
What is MARC record validation
FOLIO already supports a basic level of MARC record validation when a user creates/edits/derives a MARC bib/authority/holdings record via quickMARC. This page represents additional development to support a.) robust record validation based on Library of Congress (LOC) MARC documentation and b.) library's ability to customize default validation settings. MARC record validation IS NOT contextual help OR making certain MARC fields/subfields drop-downs/auto-complete/suggest.
Why do MARC record validation
Cataloging team The library's cataloging staff wants to make sure that their staff is following MARC standards and the library's cataloging rules to ensure consistency and quality in the bib/authority/holdings records that are used in many library workflows and is also made available to library users via discovery service/opac layers. Without validation, the library's catalog maybe difficult to search/browse and with missing/duplicate records.
However this does not mean that the library wants to prevent the record from being created/updated when a validation rule error is identified. With some MARC validation rule errors, the library wants the cataloger to know there are validation rule errors and either fix the errors or proceed with saving the record with these errors.
Scenario
A tenant/library may have cataloging rules for MARC bib field 906 (as an example). The library's cataloging practices may state that MARC bib field 906 is required and must always have a $b. In this case, the library's cataloging staff cannot create/derive/edit a MARC bib record via quickMARC that does not have a MARC bib field 906 $b.
Assumptions
- Initial development: Validation will be implemented for create/derive/edit MARC bib and authority records via quickMARC UI.
- In future releases, this implementation should span extend to supportsupport
- Data import (support MARC bib and MARC authority records only)
- Data export (support MARC bib and MARC authority records only)
- MARC holdings support via quickMARC/Data import/Data export
- Bulk-edit support
- Configuration
- Each tenant will have a default MARC bib record validation that can be customized per workflow. Initial Initial development: Create Create/Derive/Edit MARC bib via quickMARC
- Each tenant will have a default MARC authority record validation that can be customized per workflow. Initial Initial development: Create/Edit MARC authority via quickMARC
- Initial implementation - No UI will be provided. Future releases will support a UI to customize validation rules.
- The library/tenant will not be able to customize/override global MARC bib record validation rules for the following fields: Leader, 001, 005, 999 ff $s / $i, 006, 007, 008. Future implementation - we may allow for customization/override of select Leader/006/007/008 positions.
- Localization and Unicode support
- Enhanced Consortia Support
- Release upgrade
- When initial development is complete and released, there should be minimal impact to existing libraries. It should not require multiple hours to update existing records.
- It should not require significant work for hosting providers.
- Ideally existing libraries will not be asked to update all bib and authority records once this capability is released.
- Migrations from one ILS to FOLIO
- Customizations should be setup prior to loading records
- If loading records outside of data import, then we need to assume that MARC bib/authority validation is bypassed and if so, we need to ensure that the user can view and edit the record when the record has errors.
- Changing rules expectations
- Ideally a global change add/update/delete to MARC bib/authority validation rules can be done easily and not require an update to all bib and authority records.
- A tenant should be able to customize MARC bib/authority validation rules at anytime AND does not require an update to all bib and authority records.
- If the tenant adds/updates/deletes a rule and it now conflicts with existing records then we need to ensure that the user can view and edit the record when the record has errors.
- Data import
- Initial development: No impact (includes single record import)
- If the imported record conflicts with validation rules then we need to ensure that the user can view and edit the record when the record has errors.
- Reporting
- TBD
- Performance
- No change to performance when Create/Derive/Edit via quickMARC
Technical Design High-Level Requirements - Phase 1
- The tech design should not be MARC centric meaning that it should be flexible enough to assign a set of validation rules per record type + format + per workflow.
- The tech design should also allow for any flow that creates/updates/export MARC to opt-in to using these validation rules (this includes data import and data export)
- Technical design will focus on MARC bibliographic and MARC authority records. MARC holdings will be handled at a later release.
...
- See MARC validation - Logic for rules to be implemented for Phase 1.
Proposed approach - (draft)
| Release | Deliverables |
|---|---|
| Quesnelia |
|
| |
| Ramsons |
|
|
|
- Phase 1 - Complete implementation
- Logging
- Reporting
- MARC punctuation handling
- Possibly expose authority control mapping rules
- Remove subfield 9 logic on linkable fields
Phase 2 - ISBN/ISSN validation (can the existing tool be used?)
Phase 2 - Reporting (Lists app?)
Phase 3a - Data import support
Phase 3b - Data import support
Feature Matrix - Phase 1
...
Global MARC bib rules validation that includes
- MARC tags (repeatable/non-repeatable)
- MARC tags (required/not required)
- MARC indicators (required/not required)
- MARC subfields (repeatable/non-repeatable)
- MARC subfields (required/not required)
- MARC subfield values?
- MARC fixed fields
- Help documentation link
...
- Potential UI work to show help documentation
...
Global MARC authority rules validation that includes
- MARC tags (repeatable/non-repeatable)
- MARC tags (required/not required)
- MARC indicators (required/not required)
- MARC subfields (repeatable/non-repeatable)
- MARC subfields (required/not required)
- MARC subfield values?
- MARC fixed fields
- Help documentation link
...
- No front-end dev
- Potential UI work to show help documentation
...
Implement required MARC bib rules – structural based on LOC documentation
...
Select fields only. Will include FE/BE development.
...
Implement required MARC authority rules – structural based on LOC documentation
...
Allow a tenant to clone/customize/override MARC bib global rules validation to create/edit/derive via quickMARC
...
- Potential UI work to show help documentation
...
Allow a tenant to clone/customize/override MARC authority global rules validation to create/edit via quickMARC
...
- Potential UI work to show help documentation
...
- No front-end dev
...
Track/log/display
- Errors that prevent record from being saved
- Errors that do not prevent record from being saved
...
- Possible front-end work
...
Authority control validation support
Supporting Materials
...
| ||||||||||||||||
| Sunflower (TBD) |
| |||||||||||||||
| Trillium (TBD) |
| |||||||||||||||
| Umbrellaleaf (TBD) |
| |||||||||||||||
| Vetch (TBD) |
|
Supporting Materials
- MARC validation - Logic
Status colour Yellow title In progress - quickMARC - MARC21 Validation
)Status colour Yellow title In progress - quickMARC – Error handling
Status title Under Review - MARC bib: logic and rules in FOLIO
Status title Under Review - MARC Authority - System updates and quickMARC validation rules
Status title Under Review - MARC Holdings - System updates and quickMARC validation rules
Status title Under Review - Some Jira issues related to MARC validation
Status title Under Review
Potential
...
third party solution
- MarcEdit > MARCValidator > Ability to edit rulescmharlowShareFest15MetadataQAexamples/MARCEdit/MARCvalidation.md
Potential open-source development implementation options
- National Library of Finland > https://github.com/NatLibFi/marc-record-validate/blob/master/README.md
- Catmandu > https://metacpan.org/pod/Catmandu::Validator::MARC
- ??? > https://jorol.github.io/processing-marc/#/validation
- ??? > https://pkiraly.github.io/2018/01/28/marc21-in-json/
- Anyway to use MARC4j?
Market Intelligence
Voyager documentation: https://library.princeton.edu/departments/tsd/katmandu/voyager/validm.html
...
WMS documentation: https://help.oclc.org/Metadata_Services/WorldShare_Record_Manager/Bibliographic_records/MARC_21_view/Validate_MARC_21_bibliographic_records
Aleph:
Polaris: