3. Modify MARC_BIB

Owned by Kateryna Senchenko
Last updated: Mar 31, 2024

Description

Modify action was designed to be applied on the incoming MARC Bib record before mapping any other entities, to ensure such behavior it should be placed at the beginning of the Job Profile. However, Modify action can also be applied at the end of the Job Profile, following actions on MARC Bib/Instance, in that case Modify will affect existing record that was acted upon in the context of the Job. Following are the supported scenarios using Modify action.

1. Modify Incoming MARC Bib + Create Instance

NB In this case Create Instance action can be followed by Create Holdings and Create Item actions. They've been omitted for brevity. 

Flow (Updated with changes made in Quesnelia R1 2024)

  1. [1-4] UploadDefinition is created for importing files in mod-data-import
  2. [2-3] JobExecution is created in mod-srm
  3. [5-7] MARC files are uploaded from WEB client to mod-data-import
  4. [6] Uploaded file is split into chunks in mod-data-import
  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files
  6. [10-11] Profile ID is set for JobExecution
  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  8. [13-15] mod-srm reads batches from the queue, parses MARC records and saves the Incoming records in the DB. JobExecution is set to IN_PROGRESS status on first chunk received.
  9. [15-16] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_INCOMING_MARC_BIB_RECORD_PARSED
  10. [17] mod-inventory processes Modify action on incoming MARC Bib and substitutes MARC_BIBLIOGRAPHIC value in the event payload (no existing entities are affected in this case)
  11. [20] mod-inventory issues DI_SRS_MARC_BIB_RECORD_MODIFIED event
  12. [20-24] mod-inventory reads the message, MARC Bib value from '001' + '003' is moved to '035', after which the Instance is mapped and created  via HTTP in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. 
  13. [25-27] Instance HRID is set to '001' MARC_BIB field, Instance ID is set to '999 ff i' field and is saved in mod-srs via HTTP
  14. [30] mod-inventory issues DI_INVENTORY_INSTANCE_CREATED event 
  15. [31] mod-inventory reads message and exports to DI_COMPLETED
  16. [32] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED


 Click here to expand...

Flow prior to Quesnelia changes

  1. JobDefinition (uuid, profile - defines job type: insert/update) for import created in mod-srm
  2. MARC file + JobDefinition ID are uploaded from WEB client to mod-data-import (stored in memory, can be persisted. possible oom)
  3. MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  4. mod-srm reads batches from the queue, validates and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobStarts on first chunk received.
  5. mod-srs stores records into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED
  6. mod-srm reads the profile and creates JSON payload (containing parsed MARC, profile, mapping parameters) for processing. exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED
  7. mod-srs reads message from DI_SRS_MARC_BIB_RECORD_CREATED, modifies the record and publishes DI_SRS_MARC_BIB_RECORD_MODIFIED.
  8. mod-inventory reads message from DI_SRS_MARC_BIB_RECORD_MODIFIED and creates an Instance (according to profile), publishes DI_INVENTORY_INSTANCE_CREATED. If no more actions in profile: export to DI_COMPLETE, otherwise processing continues according to the JobProfile.

Diagram (Updated with changes made in Quesnelia R1 2024)

Source

 Click here to expand...

Diagram prior to Quesnelia changes

Source

2. Create Instance + Modify MARC Bib 

NB: In this case Create Instance action can also be followed by Create Holdings and Create Item actions. They've been omitted for brevity. 

Flow

  1. [1-4] UploadDefinition is created for importing files in mod-data-import
  2. [2-3] JobExecution is created in mod-srm
  3. [5-7] MARC files are uploaded from WEB client to mod-data-import
  4. [6] Uploaded file is split into chunks in mod-data-import
  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files
  6. [10-11] Profile ID is set for JobExecution
  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  8. [13-15] mod-srm reads batches from the queue, parses MARC records and saves the Incoming records in the DB. JobExecution is set to IN_PROGRESS status on first chunk received.
  9. [15-16] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_INCOMING_MARC_BIB_RECORD_PARSED
  10. [17-20] mod-inventory reads the message, MARC Bib value from '001' + '003' is moved to '035', after which the Instance is mapped and created  via HTTP in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. 
  11. [21-23] Instance HRID is set to '001' MARC_BIB field, Instance ID is set to '999 ff i' field and is saved in mod-srs via HTTP
  12. [26] mod-inventory issues DI_INVENTORY_INSTANCE_CREATED event 
  13. [27] mod-inventory modifies MARC_BIBLIOGRAPHIC, re-maps INSTANCE that are present in the payload (created as a result of the first action)
  14. [28-33] mod-inventory updates Instance in mod-inventory-storage and SRS MARC in mod-srm
  15. [36] mod-inventory issues DI_SRS_MARC_BIB_RECORD_MODIFIED event
  16. [37] mod-inventory reads message and exports to DI_COMPLETED
  17. [38] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

Diagram

Source


3. Modify MARC Bib + Match and Update Instance


NB: In this case any Match can be used (including MARC-to-MARC Match), and any allowed actions placed under the Match.

Flow

  1. [1-4] UploadDefinition is created for importing files in mod-data-import
  2. [2-3] JobExecution is created in mod-srm
  3. [5-7] MARC files are uploaded from WEB client to mod-data-import
  4. [6] Uploaded file is split into chunks in mod-data-import
  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files
  6. [10-11] Profile ID is set for JobExecution
  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  8. [13-15] mod-srm reads batches from the queue, parses MARC records and saves the Incoming records in the DB. JobExecution is set to IN_PROGRESS status on first chunk received.
  9. [15-16] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_INCOMING_MARC_BIB_RECORD_PARSED
  10. [17] mod-inventory processes Modify action on incoming MARC Bib and substitutes MARC_BIBLIOGRAPHIC value in the event payload (no existing entities are affected in this case)
  11. [20] mod-inventory issues DI_SRS_MARC_BIB_RECORD_MODIFIED event
  12. [21-22] mod-inventory reads the message, processes Match profile - reads Match criteria and constructs a CQL search query for mod-inventory-storage, sends an HTTP request to get the Instance
  13. [23] mod-inventory processes search result and issues an appropriate event (either DI_INVENTORY_INSTANCE_NOT_MATCHED if no Instance exists satisfying search criteria, or DI_INVENTORY_INSTANCE_MATCHED if such Instance is found)
  14. [24-26] In case Instance is not found Job completes for this particular Job Profile (no other actions are placed under a Non-Match branch of the Job Profile)
  15. [27-33] In case Instance is found - mod-inventory processes Update Instance action. Instance for an update is mapped from the modified version of incoming MARC Bib.
  16. [37] mod-inventory reads message and exports to DI_COMPLETED
  17. [38] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

Diagram

Source


4. Match and Update Instance + Modify MARC Bib

NB: In this particular case Modify can also be placed either inside the Match block following other actions, or outside the Match block. Modify will be processed in the same way. 
NB: Any Match profile and any allowed Actions can be used before the Modify action, however Modify at the end will be processed only if there was an explicit or implicit action on the Instance.

Flow

  1. [1-4] UploadDefinition is created for importing files in mod-data-import
  2. [2-3] JobExecution is created in mod-srm
  3. [5-7] MARC files are uploaded from WEB client to mod-data-import
  4. [6] Uploaded file is split into chunks in mod-data-import
  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files
  6. [10-11] Profile ID is set for JobExecution
  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  8. [13-15] mod-srm reads batches from the queue, parses MARC records and saves the Incoming records in the DB. JobExecution is set to IN_PROGRESS status on first chunk received.
  9. [15-16] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_INCOMING_MARC_BIB_RECORD_PARSED
  10. [17-18] mod-inventory reads the message, processes Match profile - reads Match criteria and constructs a CQL search query for mod-inventory-storage, sends an HTTP request to get the Instance
  11. [19] mod-inventory processes search result and issues an appropriate event (either DI_INVENTORY_INSTANCE_NOT_MATCHED if no Instance exists satisfying search criteria, or DI_INVENTORY_INSTANCE_MATCHED if such Instance is found)
  12. [20-22] In case Instance is not found Job completes for this particular Job Profile (no other actions are placed under a Non-Match branch of the Job Profile)
  13. [23-29] In case Instance is found - mod-inventory processes Update Instance action. Instance for an update is mapped from the incoming MARC Bib and Instance is updated in mod-inventory-storage, SRS MARC is updated in mod-srs
  14. [33] mod-inventory processes Modify action, modifies MARC_BIBLIOGRAPHIC, re-maps INSTANCE that are present in the payload (found and placed in the context as a result of the Match)
  15. [34-39] mod-inventory updates Instance in mod-inventory-storage and SRS MARC in mod-srs
  16. [42] mod-inventory issues DI_SRS_MARC_BIB_RECORD_MODIFIED event
  17. [43] mod-inventory reads message and exports to DI_COMPLETED
  18. [44] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

Diagram

Source