/
3. Modify MARC_BIB

3. Modify MARC_BIB

Dec 17, 2025

Description

Modify action was designed to be applied on the incoming MARC Bib record before mapping any other entities, to ensure such behavior it should be placed at the beginning of the Job Profile. However, Modify action can also be applied at the end of the Job Profile, following actions on MARC Bib/Instance, in that case Modify will affect existing record that was acted upon in the context of the Job. Following are the supported scenarios using Modify action.

1. Modify Incoming MARC Bib + Create Instance

NB In this case Create Instance action can be followed by Create Holdings and Create Item actions. They've been omitted for brevity. 

Flow (Updated with changes made in Quesnelia R1 2024)

  1. [1-4] UploadDefinition is created for importing files in mod-data-import

  2. [2-3] JobExecution is created in mod-srm

  3. [5-7] MARC files are uploaded from WEB client to mod-data-import

  4. [6] Uploaded file is split into chunks in mod-data-import

  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files

  6. [10-11] Profile ID is set for JobExecution

  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ

  8. [13-15] mod-srm reads batches from the queue, parses MARC records and saves the Incoming records in the DB. JobExecution is set to IN_PROGRESS status on first chunk received.

  9. [15-16] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_INCOMING_MARC_BIB_RECORD_PARSED

  10. [17] mod-inventory processes Modify action on incoming MARC Bib and substitutes MARC_BIBLIOGRAPHIC value in the event payload (no existing entities are affected in this case)

  11. [20] mod-inventory issues DI_SRS_MARC_BIB_RECORD_MODIFIED event

  12. [20-24] mod-inventory reads the message, MARC Bib value from '001' + '003' is moved to '035', after which the Instance is mapped and created  via HTTP in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. 

  13. [25-27] Instance HRID is set to '001' MARC_BIB field, Instance ID is set to '999 ff i' field and is saved in mod-srs via HTTP

  14. [30] mod-inventory issues DI_INVENTORY_INSTANCE_CREATED event 

  15. [31] mod-inventory reads message and exports to DI_COMPLETED

  16. [32] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

 

Flow prior to Quesnelia changes

  1. JobDefinition (uuid, profile - defines job type: insert/update) for import created in mod-srm

  2. MARC file + JobDefinition ID are uploaded from WEB client to mod-data-import (stored in memory, can be persisted. possible oom)

  3. MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ

  4. mod-srm reads batches from the queue, validates and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobStarts on first chunk received.

  5. mod-srs stores records into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED

  6. mod-srm reads the profile and creates JSON payload (containing parsed MARC, profile, mapping parameters) for processing. exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED

  7. mod-srs reads message from DI_SRS_MARC_BIB_RECORD_CREATED, modifies the record and publishes DI_SRS_MARC_BIB_RECORD_MODIFIED.

  8. mod-inventory reads message from DI_SRS_MARC_BIB_RECORD_MODIFIED and creates an Instance (according to profile), publishes DI_INVENTORY_INSTANCE_CREATED. If no more actions in profile: export to DI_COMPLETE, otherwise processing continues according to the JobProfile.

Diagram (Updated with changes made in Quesnelia R1 2024)

Diagram prior to Quesnelia changes

2. Create Instance + Modify MARC Bib 

NB: In this case Create Instance action can also be followed by Create Holdings and Create Item actions. They've been omitted for brevity. 

Flow

  1. [1-4] UploadDefinition is created for importing files in mod-data-import

  2. [2-3] JobExecution is created in mod-srm

  3. [5-7] MARC files are uploaded from WEB client to mod-data-import

  4. [6] Uploaded file is split into chunks in mod-data-import

  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files

  6. [10-11] Profile ID is set for JobExecution

  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ

  8. [13-15] mod-srm reads batches from the queue, parses MARC records and saves the Incoming records in the DB. JobExecution is set to IN_PROGRESS status on first chunk received.

  9. [15-16] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_INCOMING_MARC_BIB_RECORD_PARSED

  10. [17-20] mod-inventory reads the message, MARC Bib value from '001' + '003' is moved to '035', after which the Instance is mapped and created  via HTTP in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. 

  11. [21-23] Instance HRID is set to '001' MARC_BIB field, Instance ID is set to '999 ff i' field and is saved in mod-srs via HTTP

  12. [26] mod-inventory issues DI_INVENTORY_INSTANCE_CREATED event 

  13. [27] mod-inventory modifies MARC_BIBLIOGRAPHIC, re-maps INSTANCE that are present in the payload (created as a result of the first action)

  14. [28-33] mod-inventory updates Instance in mod-inventory-storage and SRS MARC in mod-srm

  15. [36] mod-inventory issues DI_SRS_MARC_BIB_RECORD_MODIFIED event

  16. [37] mod-inventory reads message and exports to DI_COMPLETED

  17. [38] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

Diagram

 

3. Modify MARC Bib + Match and Update Instance

NB: In this case any Match can be used (including MARC-to-MARC Match), and any allowed actions placed under the Match.

Flow

  1. [1-4] UploadDefinition is created for importing files in mod-data-import

  2. [2-3] JobExecution is created in mod-srm

  3. [5-7] MARC files are uploaded from WEB client to mod-data-import

  4. [6] Uploaded file is split into chunks in mod-data-import

  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files

  6. [10-11] Profile ID is set for JobExecution

  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ

  8. [13-15] mod-srm reads batches from the queue, parses MARC records and saves the Incoming records in the DB. JobExecution is set to IN_PROGRESS status on first chunk received.

  9. [15-16] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_INCOMING_MARC_BIB_RECORD_PARSED

  10. [17] mod-inventory processes Modify action on incoming MARC Bib and substitutes MARC_BIBLIOGRAPHIC value in the event payload (no existing entities are affected in this case)

  11. [20] mod-inventory issues DI_SRS_MARC_BIB_RECORD_MODIFIED event

  12. [21-22] mod-inventory reads the message, processes Match profile - reads Match criteria and constructs a CQL search query for mod-inventory-storage, sends an HTTP request to get the Instance

  13. [23] mod-inventory processes search result and issues an appropriate event (either DI_INVENTORY_INSTANCE_NOT_MATCHED if no Instance exists satisfying search criteria, or DI_INVENTORY_INSTANCE_MATCHED if such Instance is found)

  14. [24-26] In case Instance is not found Job completes for this particular Job Profile (no other actions are placed under a Non-Match branch of the Job Profile)

  15. [27-33] In case Instance is found - mod-inventory processes Update Instance action. Instance for an update is mapped from the modified version of incoming MARC Bib.

  16. [37] mod-inventory reads message and exports to DI_COMPLETED

  17. [38] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

Diagram

 

4. Match and Update Instance + Modify MARC Bib

NB: In this particular case Modify can also be placed either inside the Match block following other actions, or outside the Match block. Modify will be processed in the same way. 

NB: Any Match profile and any allowed Actions can be used before the Modify action, however Modify at the end will be processed only if there was an explicit or implicit action on the Instance.

Flow

  1. [1-4] UploadDefinition is created for importing files in mod-data-import

  2. [2-3] JobExecution is created in mod-srm

  3. [5-7] MARC files are uploaded from WEB client to mod-data-import

  4. [6] Uploaded file is split into chunks in mod-data-import

  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files

  6. [10-11] Profile ID is set for JobExecution

  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ

  8. [13-15] mod-srm reads batches from the queue, parses MARC records and saves the Incoming records in the DB. JobExecution is set to IN_PROGRESS status on first chunk received.

  9. [15-16] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_INCOMING_MARC_BIB_RECORD_PARSED

  10. [17-18] mod-inventory reads the message, processes Match profile - reads Match criteria and constructs a CQL search query for mod-inventory-storage, sends an HTTP request to get the Instance

  11. [19] mod-inventory processes search result and issues an appropriate event (either DI_INVENTORY_INSTANCE_NOT_MATCHED if no Instance exists satisfying search criteria, or DI_INVENTORY_INSTANCE_MATCHED if such Instance is found)

  12. [20-22] In case Instance is not found Job completes for this particular Job Profile (no other actions are placed under a Non-Match branch of the Job Profile)

  13. [23-29] In case Instance is found - mod-inventory processes Update Instance action. Instance for an update is mapped from the incoming MARC Bib and Instance is updated in mod-inventory-storage, SRS MARC is updated in mod-srs

  14. [33] mod-inventory processes Modify action, modifies MARC_BIBLIOGRAPHIC, re-maps INSTANCE that are present in the payload (found and placed in the context as a result of the Match)

  15. [34-39] mod-inventory updates Instance in mod-inventory-storage and SRS MARC in mod-srs

  16. [42] mod-inventory issues DI_SRS_MARC_BIB_RECORD_MODIFIED event

  17. [43] mod-inventory reads message and exports to DI_COMPLETED

  18. [44] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

Diagram

 

5. Match and update MARC Bib, holdings + modify MARC Bib (Available since Trillium)

Flow

  1. [1-4] UploadDefinition is created for importing files in mod-data-import

  2. [2-3] JobExecution is created in mod-source-record-manager

  3. [5] MARC files are uploaded from WEB client to mod-data-import

  4. [6] Uploaded file is saved chunk by chunk in mod-data-import

  5. [8-9] User selects the Job Profile and initiates the processing of the uploaded file

  6. [10-11] The job profile ID is set for JobExecution

  7. [12-13] MARC file is split into batches of MARC records and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ

  8. [14-15] mod-srm reads batches from the DI_RAW_RECORDS_CHUNK_READ queue, parses MARC records, and saves the incoming records to the DB to provide them for data import logs. JobExecution is set to IN_PROGRESS status on a first chunk receiving.

  9. [16-17] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Sends it to an appropriate Kafka queue (one message per MARC entry) - DI_MARC_FOR_UPDATE_RECEIVED

  10. [18-19] mod-inventory reads the message and processes MARC-BIB match profile - tries to find a MARC-BIB in mod-source-record-storage using match criteria from the match profile. Processes MARC-BIB search result and issues an appropriate event (DI_SRS_MARC_BIB_RECORD_NOT_MATCHED if no MARC-BIB record exists satisfying search criteria, or DI_SRS_MARC_BIB_RECORD_MATCHED if such MARC-BIB record is found)

  11. [20] If MARC-BIB is not found - mod-inventory sends DI_COMPLETED since no other profiles are placed for "non-matches" branch of the MARC-BIB Match Profile. The processing flow for the particular incoming record is completed. The sent event contains result event indication (DI_SRS_MARC_BIB_RECORD_NOT_MATCHED) denoting MARC-BIB matching result

    1. [21] mod-srm receives DI_COMPLETED and updates the import job progress

  12. [22] If MARC-BIB record is found - mod-inventory sends DI_SRS_MARC_BIB_RECORD_MATCHED 

  13. [23-24] mod-inventory reads the message and processes holdings match profile - searches holdings record in mod-inventory-storage using match criteria from the match profile. Processes holdings search result and issues an appropriate event (DI_INVENTORY_HOLDING_NOT_MATCHED if no holdings record is found, or DI_INVENTORY_HOLDING_MATCHED if holding found)

  14. [25] If holding is not found - mod-inventory sends DI_COMPLETED since no other profiles are placed for "non-matches" branch of the Holdings Match Profile. The processing flow for the particular incoming record is completed. The sent event contains result event indication (DI_INVENTORY_HOLDING_NOT_MATCHED) denoting holdings matching result

    1. [26] mod-srm receives DI_COMPLETED and updates the import job progress

  15. [27] If holding is found - mod-inventory sends DI_INVENTORY_HOLDING_MATCHED 

  16. [28-29] mod-source-record-storage reads the message and performs 1st phase of the update MARC-BIB action profile processes. Updates MARC-BIB record found on the 10th step according to mapping profile linked to the update action profile. Saves the updated record as a new ACTUAL version to the DB and sends DI_SRS_MARC_BIB_RECORD_MODIFIED_READY_FOR_POST_PROCESSING

  17. [30-33] mod-inventory reads the message and performs 2nd phase of the update MARC-BIB action profile processes. The module re-maps the related instance according to the default mapping rules and saves the updated instance in mod-inventory-storage. If the instance update succeeds, the module sends DI_SRS_MARC_BIB_RECORD_UPDATED

  18. [34-37] mod-inventory reads the message and processes update holdings action profile - maps holding according to mapping profile linked to the update action profile, saves updated holding in mod-inventory-storage and sends DI_INVENTORY_HOLDING_UPDATED

  19. [38-47] mod-inventory reads the message and processes modify MARC-BIB action profile - modifies the MARC-BIB record according to a mapping profile linked to the modify MARC-BIB action profile, saves the modified/changed MARC-BIB record in mod-source-record-storage. Further, the module re-maps the related instance according to the default mapping rules, saves the updated instance in mod-inventory-storage and sends DI_SRS_MARC_BIB_RECORD_MODIFIED

  20. [47] In case of a successful a modify MARC-BIB action profile processing the mod-inventory sends DI_COMPLETED event

  21. [47-48] mod-srm receives DI_COMPLETED event and marks JobExecution status as "COMMITTED" if all incoming record have been processed without errors, otherwise sets JobExecution status to "ERROR"

Diagram