2. Update Marc Bibs, update Instance, Holdings, Items

Description

Update entities, match by Instance/Holdings/Item

Flow (according to changes made in Quesnelia R1 2024)

  1. [1-4] UploadDefinition is created for importing files in mod-data-import
  2. [2-3] JobExecution is created in mod-srm
  3. [5] MARC files are uploaded from WEB client to mod-data-import
  4. [6] Uploaded file is saved chunk by chunk in mod-data-import
  5. [8-9] User selects the Job Profile and initiates the processing of the uploaded file
  6. [10-11] The job profile ID is set for JobExecution
  7. [12-13] MARC file is split into batches of MARC records and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  8. [14-15] mod-srm reads batches from the DI_RAW_RECORDS_CHUNK_READ queue, parses MARC records, and saves the incoming records to the DB to provide them for data import logs. JobExecution is set to IN_PROGRESS status on first chunk received.
  9. [16-17] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Sends it to an appropriate Kafka queue (one message per MARC entry) - DI_MARC_FOR_UPDATE_RECEIVED
  10. [18-19] mod-inventory reads the message and processes instance match profile - tries to find an instance in mod-inventory-storage using match criteria from the match profile. Processes instance search result and issues an appropriate event (DI_INVENTORY_INSTANCE_NOT_MATCHED if no Instance exists satisfying search criteria, or DI_INVENTORY_INSTANCE_MATCHED if such Instance is found)
  11. [20] If instance is not found - mod-inventory sends DI_COMPLETED since no other profiles are placed for "non-matches" branch of the Instance Match Profile. The processing flow for the particular incoming record is completed. The sent event contains result event indication (DI_INVENTORY_INSTANCE_NOT_MATCHED) denoting instance matching result
    1. [21] mod-srm receives DI_COMPLETED and updates the import job progress
  12. [22] If instance is found - mod-inventory sends DI_INVENTORY_INSTANCE_MATCHED
  13. [23-25] mod-inventory reads the message and processes update instance action profile - maps instance according to mapping profile linked to update action profile and saves updated instance in mod-inventory-storage
  14. [26-29] mod-inventory updates underlying MARC record in mod-srs via HTTP and sends DI_INVENTORY_INSTANCE_UPDATED
  15. [30-31] mod-inventory reads the message and processes holdings match profile - searches holdings record in mod-inventory-storage using match criteria from the match profile. Processes holdings search result and issues an appropriate event (DI_INVENTORY_HOLDING_NOT_MATCHED if no holdings record is found, or DI_INVENTORY_HOLDING_MATCHED if holding found)
  16. [32] If holding is not found - mod-inventory sends DI_COMPLETED since no other profiles are placed for "non-matches" branch of the Holdings Match Profile. The processing flow for the particular incoming record is completed. The sent event contains result event indication (DI_INVENTORY_HOLDING_NOT_MATCHED) denoting holdings matching result
    1. [33] mod-srm receives DI_COMPLETED and updates the import job progress
  17. [34] If holding is found - mod-inventory sends DI_INVENTORY_HOLDING_MATCHED 
  18. [35-38] mod-inventory reads the message and processes update holdings action profile - maps holding according to mapping profile linked to the update action profile, saves updated holding in mod-inventory-storage and sends DI_INVENTORY_HOLDING_UPDATED
  19. [38-40] mod-inventory reads the message and processes item match profile - searches item in mod-inventory-storage using match criteria from the match profile. Processes item search result and issues an appropriate event (DI_INVENTORY_ITEM_NOT_MATCHED if no item is found, or DI_INVENTORY_ITEM_MATCHED if item found)
  20. [41] If item is not found - mod-inventory sends DI_COMPLETED since no other profiles are placed for "non-matches" branch of the Item Match Profile. The processing flow for the particular incoming record is completed. The sent event contains result event indication (DI_INVENTORY_ITEM_NOT_MATCHED ) denoting item matching result
    1. [42] mod-srm receives DI_COMPLETED and updates the import job progress
  21. [43] If item is found - mod-inventory sends DI_INVENTORY_ITEM_MATCHED
  22. [43-46] mod-inventory reads the message and processes update item action profile - maps item according to mapping profile linked to the update action profile, saves updated item in mod-inventory-storage and sends DI_INVENTORY_ITEM_UPDATED
  23. [49] In case of a successful item update mod-inventory sends DI_COMPLETED event
  24. [50] mod-srm receives DI_COMPLETED event and marks JobExecution status as "COMMITTED" if all incoming record have been processed without errors, otherwise sets JobExecution status to "ERROR"

Diagram (updated with changes made in Quesnelia R1 2024)

Source


 Click here to expand flow prior to Quesnelia changes

Flow (prior to changes made in Quesnelia)

  1. JobDefinition (uuid, profile - defines job type: insert/update) for import created in mod-srm
  2. MARC file + JobDefinition ID are uploaded from WEB client to mod-data-import
  3. MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  4. mod-srm reads batches from the queue, validates and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobStarts on first chunk received.
  5. mod-srs stores records into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED
  6. mod-srm reads the profile and creates JSON payload (containing parsed MARC, profile, mapping parameters) for processing. exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED
  7. mod-inventory reads message from DI_SRS_MARC_BIB_RECORD_CREATED. Tries to find entity according to match criteria (in mod-inventory-storage via OKAPI HTTP)
  8. If found: exports result to DI_INVENTORY_INSTANCE_MATCHED / DI_INVENTORY_HOLDING_MATCHED / DI_INVENTORY_ITEM_MATCHED
    1. mod-inventory receives match result and updates Instance/Holding/Item (according to action in profile) in mod-inventory-storage according to profile. And publishes result to applied topic (one per entity type) DI_INVENTORY_INSTANCE_UPDATED / DI_INVENTORY_HOLDING_UPDATED / DI_INVENTORY_ITEM_UPDATED
    2. mod-inventory reads message from prev. step (1 of 3 topics) and seeks for more actions (return to a.)  If no more actions in profile: export to DI_COMPLETE
  9. If not found:  DI_INVENTORY_INSTANCE_NOT_MATCHED / DI_INVENTORY_HOLDING_NOT_MATCHED / DI_INVENTORY_ITEM_NOT_MATCHED events are published
    1. mod-srm reads DI_INVENTORY_INSTANCE_NOT_MATCHED / DI_INVENTORY_HOLDING_NOT_MATCHED / DI_INVENTORY_ITEM_NOT_MATCHED events and updates job status accordingly with "NOT_MATCHED" status for record


Diagram (prior to changes made in Quesnelia)

 

Source

Comments

For not empty "non-match" branch in profile a new flow description needs to be created