SPIKE: Update MARC authority records via Data import

MODDATAIMP-585 - Getting issue details... STATUS

Feature: UXPROD-2990 - Getting issue details... STATUS

Data Import Subgroup Requirements


Now the data-import is able to import Marc authority records, mod-inventory can generate Folio records, mod-search provides API for the search. In this story, we need to understand how to implement the Update for Marc authorities in data-import, what changes are needed to support the Search of the updated records via mod-search.

The Update is an action that is specified by the Action profile, so anytime the user needs to choose the Action profile if there is a need to update records. The Matching profile defines the types of incoming records and existing records, that need to be updated. Also, a Match profile contains the Match criteria, which specifies parameters for records comparison. 

Reference page: Update SRS MARC Authority records via Data Import


MARC Authority update workflow in data-import

The user uploads the MARC file, chooses an Action profile & Matching profile on the UI application, starts the import process

Step№ModuleTaskDI Event
1mod-data-importSends a batch of records to Kafka queueDI_RAW_RECORDS_CHUNK_READ

Receives records, parses to the JSON, sends it to Kafka queue


Receives the event from SRM, Stores Authority records into the database, and returns the result back via Kafka queue.
The SRS will increment 'generation' field matching records by 999 ff $s.

The SRS now can match records only by 001 and 999 ff $s, this logic is hard to expand. Matching 010 $a or other tags requires implementing a spike MODDATAIMP-583 - Getting issue details... STATUS

4mod-source-record-managerReceives the event from SRS, creates JSON payload (containing parsed MARC, profile, mapping parameters) for further processing, and sends it to the queueDI_SRS_MARC_AUTHORITY_RECORD_CREATED

Receives the event from SRM, loads the Folio record that matches Marc tags, creates JSON payload with a matched Folio record, sends it to the queue.

We have to implement AuthorityLoader on this step. The loader retrieves the Folio record from mod-inventory-storage that matches Marc tags 001, 010 $a, 999 ff $s , or use Folio record id


Receives the event from itself. Updates the existing Folio record.

We have to implement UpdateAuthorityEventHandler on this step. The handler updates the existing Folio record by the matched record taken from the event payload



7mod-inventory-storageReceives HTTP PUT request with the record that needs to be updated, updates the record in the database, puts record to Kafka topic
8mod-searchReceives a record from Kafka topic on an update, deletes the existing record matching by id, inserts the new record

Sequence diagram sources
PlantUML diagram

Field protection is not needed at the beginning

       User stories

ModuleDescriptionJiraHigh-level rough est
mod-source-record-storageSpike to extend matching

MODDATAIMP-583 - Getting issue details... STATUS

mod-data-import-converter-storageStory to create default Action profile & Matching profilesTo be defined laterTo be defined later
data-import-processing-coreStory to add new event types for the Update

MODDICORE-235 - Getting issue details... STATUS

mod-inventoryStory to implement the AuthorityLoader

MODINV-606 - Getting issue details... STATUS

mod-inventory Story to implement the UpdateAuthorityEventHandler

MODINV-607 - Getting issue details... STATUS

mod-source-record-managerLog the non-matched records
update DataImportJournalConsumersVerticle and JournalParam

MODSOURMAN-653 - Getting issue details... STATUS

mod-source-record-storageSet matched_id, external_id, and parsedRecord data

MODSOURCE-429 - Getting issue details... STATUS


mod-source-record-storage: Update matching criteria for authorities to support 010 field
ui-data-import: Update Matching profiles to enable Authorities 
ui-plugin-find-import-profile: https://github.com/folio-org/ui-plugin-find-import-profile does it require some work ???