MARC Validation Requirements Overview

Page status: IN PROGRESS

What is MARC record validation

FOLIO already supports a basic level of MARC record validation when a user creates/edits/derives a MARC bib/authority/holdings record via quickMARC. This page represents additional development to support a.) robust record validation based on Library of Congress (LOC) MARC documentation and b.) library's ability to customize default validation settings. MARC record validation IS NOT contextual help OR making certain MARC fields/subfields drop-downs/auto-complete/suggest.

Why do MARC record validation

The library's cataloging staff wants to make sure that their staff is following MARC standards and the library's cataloging rules to ensure consistency and quality in the bib/authority/holdings records that are used in many library workflows and is also made available to library users via discovery service/opac layers. Without validation, the library's catalog maybe difficult to search/browse and with missing/duplicate records.

However this does not mean that the library wants to prevent the record from being created/updated when a validation rule error is identified. With some MARC validation rule errors, the library wants the cataloger to know there are validation rule errors and either fix the errors or proceed with saving the record with these errors.

Scenario

A tenant/library may have cataloging rules for MARC bib field 906 (as an example). The library's cataloging practices may state that MARC bib field 906 is required and must always have a $b. In this case, the library's cataloging staff cannot create/derive/edit a MARC bib record via quickMARC that does not have a MARC bib field 906 $b.

Assumptions

Initial development: Validation will be implemented for create/derive/edit MARC bib and authority records via quickMARC UI.
In future releases, this implementation should extend to support
- Data import (support MARC bib and MARC authority records only)
- Data export (support MARC bib and MARC authority records only)
- MARC holdings support via quickMARC/Data import/Data export
- Bulk-edit support
Configuration
- Each tenant will have a default MARC bib record validation that can be customized per workflow. Initial development: Create/Derive/Edit MARC bib via quickMARC
- Each tenant will have a default MARC authority record validation that can be customized per workflow. Initial development: Create/Edit MARC authority via quickMARC
- Initial implementation - No UI will be provided. Future releases will support a UI to customize validation rules.
- The library/tenant will not be able to customize/override global MARC bib record validation rules for the following fields: Leader, 001, 005, 999 ff $s / $i, 006, 007, 008. Future implementation - we may allow for customization/override of select Leader/006/007/008 positions.
Localization and Unicode support
Enhanced Consortia Support
Release upgrade
- When initial development is complete and released, there should be minimal impact to existing libraries. It should not require multiple hours to update existing records.
- It should not require significant work for hosting providers.
- Ideally existing libraries will not be asked to update all bib and authority records once this capability is released.
Migrations from one ILS to FOLIO
- Customizations should be setup prior to loading records
- If loading records outside of data import, then we need to assume that MARC bib/authority validation is bypassed and if so, we need to ensure that the user can view and edit the record when the record has errors.
Changing rules expectations
- Ideally a global add/update/delete to MARC bib/authority validation rules can be done easily and not require an update to all bib and authority records.
- A tenant should be able to customize MARC bib/authority validation rules at anytime AND does not require an update to all bib and authority records.
- If the tenant adds/updates/deletes a rule and it now conflicts with existing records then we need to ensure that the user can view and edit the record when the record has errors.
Data import
- Initial development: No impact (includes single record import)
- If the imported record conflicts with validation rules then we need to ensure that the user can view and edit the record when the record has errors.
Reporting
- TBD
Performance
- No change to performance when Create/Derive/Edit via quickMARC

Technical Design High-Level Requirements - Phase 1

The tech design should not be MARC centric meaning that it should be flexible enough to assign a set of validation rules per record type + format + per workflow.
The tech design should also allow for any flow that creates/updates/export MARC to opt-in to using these validation rules (this includes data import and data export)
Technical design will focus on MARC bibliographic and MARC authority records. MARC holdings will be handled at a later release.
See MARC validation - Logic for rules to be implemented for Phase 1.

Phased approach (covers Phase 1- 3)

Release	Deliverables	Jira Requirements
Quesnelia	Phase 1 - Define technical approach when creating/editing/deriving MARC bib/auth via quickMARC that also accounts for ECS. Implement several related MARC validation issues	MARC validation - Logic LDR contextual help
Ramsons	Phase 1 - Implement MARC record validation when creating/editing/deriving MARC bib/auth via quickMARC Includes ECS support LCCN structure validation Support validation for MARC bib fields outlined https://www.loc.gov/marc/bibliographic/ MARC bib fields not outlined https://www.loc.gov/marc/bibliographic/ OR is a 9XX field https://www.loc.gov/marc/bibliographic/bdapndxh.html MARC authority fields outlined https://www.loc.gov/marc/authority/ecadhome.html MARC authority fields not outlined https://www.loc.gov/marc/authority/ecadhome.html Log errors that warn but do not prevent user from saving MARC bib/authority records. Authority control - Subject validation (stretch) Access to help documentation - UI only (stretch)
Sunflower	Phase 1 - Complete implementation View Errors that do not prevent save Additional Logging MARC punctuation handling Expose authority control mapping rules? Remove subfield 9 logic on linkable fields?
Trillium	Phase 2 - ISBN/ISSN validation (can the existing tool be used?) Phase 2 - Reporting (Lists app?)
Umbrellaleaf	Phase 3a - Data import support (including single record import)
Vetch	Phase 3b - Data import support (including single record import)

Feature Matrix - Phase 1

Feature	Description / Jira issue(s)	Notes
Tenant level - Configure MARC bib rules validation that supports logic outlined - MARC validation - Logic Define Global validation rules Define which Global validation rules cannot be overridden by a tenant Specify which validation rule(s) can be overridden by a tenant Specify which validation rule(s) prevents user from saving record OR just warns a user and allows save to continue Set Help documentation link	MODQM-390 - Getting issue details... STATUS	quickMARC front-end work Display error and warn messaging Option to validate Link(s) to help documentation
Tenant level - Configure MARC authority rules validation that supports logic outlined - MARC validation - Logic Define Global validation rules Define which Global validation rules cannot be overridden by a tenant Specify which validation rule(s) can be overridden by a tenant Specify which validation rule(s) prevents user from saving record OR just warns a user and allows save to continue Set Help documentation link	MODQM-391 - Getting issue details... STATUS MODQM-393 - Getting issue details... STATUS	quickMARC front-end work Display error and warn messaging Option to validate Link(s) to help documentation
Allow a tenant to create/modify validation rules for local bib fields or fields not outlined in LOC MARC documentation Repeatability Valid values (indicators, subfields) Required/optional
Allow a tenant to create/modify validation rules for local authority fields or fields not outlined in LOC MARC documentation Repeatability Valid values (indicators, subfields) Required/optional
Allow a user to view and edit a MARC bib/MARC authority that bypasses rules validation and has errors. Examples: a record imported via data import or direct to the db.
Allow a tenant to delete validation rules for local bib fields or fields not outlined in LOC MARC documentation
Allow a tenant to delete validation rules for local authority fields or fields not outlined in LOC MARC documentation
Authority control subject validation		Frontend dev quickMARC display error messaging
Ability to export validation rules in a json/text file

Supporting Materials

MARC validation - Logic IN PROGRESS
quickMARC - MARC21 Validation IN PROGRESS
quickMARC – Error handling UNDER REVIEW
MARC bib: logic and rules in FOLIO UNDER REVIEW
MARC Authority - System updates and quickMARC validation rules UNDER REVIEW
MARC Holdings - System updates and quickMARC validation rulesUNDER REVIEW
Some Jira issues related to MARC validation UNDER REVIEW

Potential third party solution

MarcEdit > MARCValidator > Ability to edit rules
- https://github.com/reeset/marcrulesfiles/blob/master/marcrules.txt

Potential open-source development implementation options

National Library of Finland > https://github.com/NatLibFi/marc-record-validate/blob/master/README.md
Catmandu > https://metacpan.org/pod/Catmandu::Validator::MARC
??? > https://jorol.github.io/processing-marc/#/validation
??? > https://pkiraly.github.io/2018/01/28/marc21-in-json/
Anyway to use MARC4j?

Market Intelligence

Voyager documentation: https://library.princeton.edu/departments/tsd/katmandu/voyager/validm.html

ALMA: https://knowledge.exlibrisgroup.com/Alma/Product_Documentation/010Alma_Online_Help_(English)/Metadata_Management/210Metadata_Management_Configuration/Configuring_Cataloging

WMS documentation: https://help.oclc.org/Metadata_Services/WorldShare_Record_Manager/Bibliographic_records/MARC_21_view/Validate_MARC_21_bibliographic_records

Aleph:

Sierra: https://documentation.iii.com/sierrahelp/Default.htm#sgcat/sgcat_validate_MARC.html?Highlight=MARC%20record%20validation

Polaris: