MARC Validation Requirements Overview

Page status: IN PROGRESS

What is MARC record validation 

FOLIO already supports a basic level of MARC record validation when a user creates/edits/derives a MARC bib/authority/holdings record via quickMARC. This page represents additional development to support a.) robust record validation based on Library of Congress (LOC) MARC documentation and b.) library's ability to customize default validation settings. MARC record validation IS NOT contextual help OR making certain MARC fields/subfields drop-downs/auto-complete/suggest. 

Why do MARC record validation

The library's cataloging staff wants to make sure that their staff is following MARC standards and the library's cataloging rules to ensure consistency and quality in the bib/authority/holdings records that are used in many library workflows and is also made available to library users via discovery service/opac layers. Without validation, the library's catalog maybe difficult to search/browse and with missing/duplicate records. 

Scenario 

A tenant/library may have cataloging rules for MARC bib field 906 (as an example). The library's cataloging practices may state that MARC bib field 906 is required and must always have a $b. In this case, the library's cataloging staff cannot create/derive/edit a MARC bib record via quickMARC that does not have a MARC bib field 906 $b. 

Assumptions 

• Initial development: Validation will be implemented for create/derive/edit MARC bib and authority records via quickMARC UI.
• In future releases, this implementation should extend to support 
  
  • Data import (support MARC bib and MARC authority records only)
  • Data export (support MARC bib and MARC authority records only)
  • MARC holdings support via quickMARC/Data import/Data export 
• Configuration 
  • Each tenant will have a default MARC bib record validation that can be customized per workflow. Initial development: Create/Derive/Edit MARC bib via quickMARC
  • Each tenant will have a default MARC authority record validation that can be customized per workflow. Initial development: Create/Edit MARC authority via quickMARC
  • Initial implementation - No UI will be provided. Future releases will support a UI to customize validation rules. 
  • The library/tenant will not be able to customize/override global MARC bib record validation rules for the following fields: Leader, 001, 005, 999 ff $s / $i, 006, 007, 008. Future implementation - we may allow for customization/override of select Leader/006/007/008 positions. 
• Localization and Unicode support 
• Enhanced Consortia Support  
• Release upgrade
  • When initial development is complete and released, there should be minimal impact to existing libraries. It should not require multiple hours to update existing records. 
  • It should not require significant work for hosting providers. 
  • Ideally existing libraries will not be asked to update all bib and authority records once this capability is released. 
• Migrations from one ILS to FOLIO
  • Customizations should be setup prior to loading records
  • If loading records outside of data import, then we need to assume that MARC bib/authority validation is bypassed and if so, we need to ensure that the user can view and edit the record when the record has errors. 
• Changing rules expectations 
  • Ideally a global add/update/delete to MARC bib/authority validation rules can be done easily and not require an update to all bib and authority records. 
  • A tenant should be able to customize MARC bib/authority validation rules at anytime AND does not require an update to all bib and authority records. 
  • If the tenant adds/updates/deletes a rule and it now conflicts with existing records then we need to ensure that the user can view and edit the record when the record has errors. 
• Data import 
  • Initial development: No impact (includes single record import) 
  • If the imported record conflicts with validation rules then we need to ensure that the user can view and edit the record when the record has errors. 
• Reporting
  
  • TBD
•  Performance
  • No change to performance when Create/Derive/Edit via quickMARC

Technical Design High-Level Requirements - Phase 1

• The tech design should not be MARC centric meaning that it should be flexible enough to assign a set of validation rules per record type + format + per workflow. 
• The tech design should also allow for any flow that creates/updates/export MARC to opt-in to using these validation rules (this includes data import and data export) 
• Technical design will focus on MARC bibliographic and MARC authority records. MARC holdings will be handled at a later release. 

Phased approach (covers Phase 1- 3) 

Feature Matrix - Phase 1

Supporting Materials 

• quickMARC - MARC21 Validation (IN PROGRESS)
• quickMARC – Error handling UNDER REVIEW
• MARC bib: logic and rules in FOLIO UNDER REVIEW
• MARC Authority - System updates and quickMARC validation rules UNDER REVIEW
• MARC Holdings - System updates and quickMARC validation rulesUNDER REVIEW
• Some Jira issues related to MARC validation UNDER REVIEW

Potential third party solution 

• MarcEdit > MARCValidator > Ability to edit rules
  • https://github.com/reeset/marcrulesfiles/blob/master/marcrules.txt 

Potential open-source development implementation options

• National Library of Finland > https://github.com/NatLibFi/marc-record-validate/blob/master/README.md
• Catmandu > https://metacpan.org/pod/Catmandu::Validator::MARC
• ??? > https://jorol.github.io/processing-marc/#/validation
• ??? > https://pkiraly.github.io/2018/01/28/marc21-in-json/
• Anyway to use MARC4j?
  

Market Intelligence

Voyager documentation: https://library.princeton.edu/departments/tsd/katmandu/voyager/validm.html

ALMA:  https://knowledge.exlibrisgroup.com/Alma/Product_Documentation/010Alma_Online_Help_(English)/Metadata_Management/210Metadata_Management_Configuration/Configuring_Cataloging

WMS documentation: https://help.oclc.org/Metadata_Services/WorldShare_Record_Manager/Bibliographic_records/MARC_21_view/Validate_MARC_21_bibliographic_records

Aleph: 

Sierra: https://documentation.iii.com/sierrahelp/Default.htm#sgcat/sgcat_validate_MARC.html?Highlight=MARC%20record%20validation

Polaris: