[Tech Debt]: MARC BIB records are converted to graph unnecessarily & then discarded

Description

Currently, mod-linked-data processes all newly created MARC bibliographic records in FOLIO from the Kafka topic srs.source_records in order to update the HRID of instance resources in the graph. It converts these MARC records into Resource objects using the marc2LD library. After conversion, it checks if the corresponding instance already exists in the Linked Data graph by matching the inventory instance ID. If no match is found, the newly created Resource is discarded. If a match is found, the existing resource's administrative metadata is updated with the new resource's metadata.

However, the majority of instances do not exist in the Linked Data graph, leading to two main issues:

  1. Unnecessary Processing: Many MARC records are converted into Resource objects only to be discarded, consuming unnecessary resources (CPU, Memory).

  2. Log Clutter: The log files get filled with unwanted logs from the MARC to Resource conversion process, making debugging and monitoring more difficult.

Proposed Solution:
Refactor the process to first check whether the source of the instance is Linked Data. This can be done by checking for the presence of 999 $l. If 999 $l is not present in the incoming MARC BIB record, then discard the event.

Environment

None

Potential Workaround

None

Checklist

hide

Activity

Show:
Done

Details

Assignee

Reporter

Priority

Story Points

Sprint

Development Team

Citation

Fix versions

Release

Sunflower (R1 2025)

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs
Created January 30, 2025 at 8:28 PM
Updated March 13, 2025 at 8:45 PM
Resolved February 4, 2025 at 2:55 PM
TestRail: Cases
TestRail: Runs

Flag notifications