Automate linking

Automate linking in quickMARC app

Introduction

This document outlines the design of a backend feature, which will allow users to automatically validate/update/create links for MARC bib fields to an authority record when editing a MARC bib record.

More details on a feature: [UXPROD-3874] and spike story: [MODELINKS-79]

Requirements

Functional Requirements

The API must allow finding all applicable MARC authorities to control the MARC bib record based on the $0 subfield of the bib's fields.
The API must provide only suggestions for links, and must not save any new data, saving will be performed by a user.
The API must allow saving the MARC bib record even if linking a MARC bib field to a MARC authority record was unsuccessful.
The API must allow sending a MARC record for links assignment with already existing links.
The API must allow sending a MARC record for links assignment with not saved links and changes to the bib record.
The API must provide a notification of what fields were successfully linked and what fields that are applicable for linking failed to link.
The API must provide different types of failures for failed fields.

Non-Functional Requirements

Automate linking should take no longer than ~2 seconds

Architecture

An API endpoint will be implemented in mod-quick-marc to provide UI with suggested links for the record. mod-quick-marc will work as a proxy module, that will call the mod-entities-links newly created endpoint. All main linking logic will be implemented in mod-entities-links. mod-search will be used as a search service for finding applicable MARC authorities. mod-source-record-storage will be used for fetching data needed to construct controllable fields in the MARC bib record. All interaction between modules is via HTTP.

Component Diagram Source

@startuml
skinparam componentStyle rectangle

[User Interface]


package "Backend" {
  [Okapi] --> [mod-entities-links]
  [mod-entities-links] ..> [Okapi]
  [Okapi] --> [mod-search]
  [mod-search] ..> [Okapi]
  [Okapi] --> [mod-quick-marc]
  [mod-quick-marc] ..> [Okapi]
  [Okapi] --> [mod-source-record-storage]
  [mod-source-record-storage] ..> [Okapi]
}

[User Interface] --> [Okapi]
[Okapi] --> [User Interface]

database "PostgreSql" {
  [entities-links]
  [source-record-storage]
}

database "OpenSearch/ElasticSearch" {
  [authorities]
}

[mod-entities-links] <--> [entities-links]
[mod-source-record-storage] <--> [source-record-storage]
[mod-search] <--> [authorities]

@enduml

Data Flow and Processing

PlantUML diagram code

@startuml
title QuickMarc Autolinking 

participant UI as ui
participant "quick-marc" as qm
participant "entities-links" as el
participant "search" as ms
participant "source-record-storage" as rs

autonumber
ui -> qm ++ : request record\nwith links
qm -> qm: convert
qm -> el ++ : request assign links
el -> el : get linking rules
el -> el : extract $0s
el -> ms ++ : search authorities (without count)
ms --> el -- : authorities
el -> rs ++ : get records by external ids
rs --> el -- : source records
el -> el : set links data
el --> qm -- : record with links
qm -> qm -- : convert
qm --> ui : record with links
@enduml

The UI sends a request to the backend API.
mod-quick-marc receives the request and converts the record into an SRS-like format.
mod-quick-marc sends a request to the mod-entities-links API.
mod-entities-links receives the request and fetches linking rules from the database using cache.
From MARC bib fields that are applicable for linking according to linking rules, $0 subfield values are extracted.
$0 values are used for search authorities in mod-search. The current mod-search endpoint also calculates a number of already existing links in the instance index, this should be omitted to speed up the process.
mod-entities-links receives a collection of authority records and prepares a request to the mod-source-record-storage.
mod-entities-links sends a request to the mod-source-record-storage bulk endpoint.
mod-entities-links receives a collection of authority source records.
mod-entities-links analyze results, prepare data for links according to linking rules, and set constructed links into the record.
mod-quick-marc receives the record with links.
mod-quick-marc converts the record into the appropriate format.
UI receives the record with suggested links.

Automate linking