MARC Validation Requirements Overview

Page status: IN PROGRESS

What is MARC record validation

FOLIO already supports a basic level of MARC record validation when a user creates/edits/derives a MARC bib/authority/holdings record via quickMARC. This page represents additional development to support a.) robust record validation based on Library of Congress (LOC) MARC documentation and b.) library's ability to customize default validation settings. MARC record validation IS NOT contextual help OR making certain MARC fields/subfields drop-downs/auto-complete/suggest.

Why do MARC record validation

The library's cataloging staff wants to make sure that their staff is following MARC standards and the library's cataloging rules to ensure consistency and quality in the bib/authority/holdings records that are used in many library workflows and is also made available to library users via discovery service/opac layers. Without validation, the library's catalog maybe difficult to search/browse and with missing/duplicate records.

However this does not mean that the library wants to prevent the record from being created/updated when a validation rule error is identified. With some MARC validation rule errors, the library wants the cataloger to know there are validation rule errors and either fix the errors or proceed with saving the record with these errors.

Scenario

A tenant/library may have cataloging rules for MARC bib field 906 (as an example). The library's cataloging practices may state that MARC bib field 906 is required and must always have a $b. In this case, the library's cataloging staff cannot create/derive/edit a MARC bib record via quickMARC that does not have a MARC bib field 906 $b.

Assumptions

Initial development: Validation will be implemented for create/derive/edit MARC bib and authority records via quickMARC UI.
In future releases, this implementation should extend to support
- Data import (support MARC bib and MARC authority records only)
- Data export (support MARC bib and MARC authority records only)
- MARC holdings support via quickMARC/Data import/Data export
- Bulk-edit support
Configuration
- Each tenant will have a default MARC bib record validation that can be customized per workflow. Initial development: Create/Derive/Edit MARC bib via quickMARC
- Each tenant will have a default MARC authority record validation that can be customized per workflow. Initial development: Create/Edit MARC authority via quickMARC
- Initial implementation - No UI will be provided. Future releases will support a UI to customize validation rules.
- The library/tenant will not be able to customize/override global MARC bib record validation rules for the following fields: Leader, 001, 005, 999 ff $s / $i, 006, 007, 008. Future implementation - we may allow for customization/override of select Leader/006/007/008 positions.
Localization and Unicode support
Enhanced Consortia Support
Release upgrade
- When initial development is complete and released, there should be minimal impact to existing libraries. It should not require multiple hours to update existing records.
- It should not require significant work for hosting providers.
- Ideally existing libraries will not be asked to update all bib and authority records once this capability is released.
Migrations from one ILS to FOLIO
- Customizations should be setup prior to loading records
- If loading records outside of data import, then we need to assume that MARC bib/authority validation is bypassed and if so, we need to ensure that the user can view and edit the record when the record has errors.
Changing rules expectations
- Ideally a global add/update/delete to MARC bib/authority validation rules can be done easily and not require an update to all bib and authority records.
- A tenant should be able to customize MARC bib/authority validation rules at anytime AND does not require an update to all bib and authority records.
- If the tenant adds/updates/deletes a rule and it now conflicts with existing records then we need to ensure that the user can view and edit the record when the record has errors.
Data import
- Initial development: No impact (includes single record import)
- If the imported record conflicts with validation rules then we need to ensure that the user can view and edit the record when the record has errors.
Reporting
- TBD
Performance
- No change to performance when Create/Derive/Edit via quickMARC

Technical Design High-Level Requirements - Phase 1

The tech design should not be MARC centric meaning that it should be flexible enough to assign a set of validation rules per record type + format + per workflow.
The tech design should also allow for any flow that creates/updates/export MARC to opt-in to using these validation rules (this includes data import and data export)
Technical design will focus on MARC bibliographic and MARC authority records. MARC holdings will be handled at a later release.
See MARC validation - Logic for rules to be implemented for Phase 1.

Proposed approach - (draft)

Release	Deliverables
Quesnelia	Phase 1 - Define technical approach when creating/editing/deriving MARC bib/auth via quickMARC that also accounts for ECS. Implement several related MARC validation issues
Ramsons	Implement MARC record validation when creating/editing/deriving MARC bib/auth via quickMARC Includes ECS support Supports validation for MARC bib fields outlined https://www.loc.gov/marc/bibliographic/ecbdlist.html (including local fields) AND MARC bib fields not outlined https://www.loc.gov/marc/bibliographic/ecbdlist.html MARC authority fields outlined https://www.loc.gov/marc/authority/ecadhome.html (including local fields) AND MARC authority fields not outlined https://www.loc.gov/marc/authority/ecadhome.html Access to Field help documentation LCCN structure validation and duplication checking. UXPROD-4537 - Getting issue details... STATUS UXPROD-4060 - Getting issue details... STATUS
Sunflower (TBD)	UI configuration? Authority control - Subject validation? MARC punctuation handling (determine a front-end approach)? Handle deprecated fields - https://www.loc.gov/marc/bibliographic/bdapndxh.html?
Trillium (TBD)	ISBN/ISSN validation Expose authority control mapping rules?
Umbrellaleaf (TBD)	Data import support (including single record import)
Vetch (TBD)	Data import support (including single record import)

Supporting Materials

MARC validation - Logic IN PROGRESS
quickMARC - MARC21 Validation IN PROGRESS
quickMARC – Error handling UNDER REVIEW
MARC bib: logic and rules in FOLIO UNDER REVIEW
MARC Authority - System updates and quickMARC validation rules UNDER REVIEW
MARC Holdings - System updates and quickMARC validation rulesUNDER REVIEW
Some Jira issues related to MARC validation UNDER REVIEW

Potential third party solution

MarcEdit > MARCValidator > Ability to edit rules
- https://github.com/reeset/marcrulesfiles/blob/master/marcrules.txt

Potential open-source development implementation options

National Library of Finland > https://github.com/NatLibFi/marc-record-validate/blob/master/README.md
Catmandu > https://metacpan.org/pod/Catmandu::Validator::MARC
??? > https://jorol.github.io/processing-marc/#/validation
??? > https://pkiraly.github.io/2018/01/28/marc21-in-json/
Anyway to use MARC4j?

Market Intelligence

Voyager documentation: https://library.princeton.edu/departments/tsd/katmandu/voyager/validm.html

ALMA: https://knowledge.exlibrisgroup.com/Alma/Product_Documentation/010Alma_Online_Help_(English)/Metadata_Management/210Metadata_Management_Configuration/Configuring_Cataloging

WMS documentation: https://help.oclc.org/Metadata_Services/WorldShare_Record_Manager/Bibliographic_records/MARC_21_view/Validate_MARC_21_bibliographic_records

Aleph:

Sierra: https://documentation.iii.com/sierrahelp/Default.htm#sgcat/sgcat_validate_MARC.html?Highlight=MARC%20record%20validation

Polaris: