/
2. Update Marc Bibs, update Instance, Holdings, Items

2. Update Marc Bibs, update Instance, Holdings, Items

Apr 14, 2024

Description

Update entities, match by Instance/Holdings/Item

Flow (according to changes made in Quesnelia R1 2024)

  1. [1-4] UploadDefinition is created for importing files in mod-data-import

  2. [2-3] JobExecution is created in mod-srm

  3. [5] MARC files are uploaded from WEB client to mod-data-import

  4. [6] Uploaded file is saved chunk by chunk in mod-data-import

  5. [8-9] User selects the Job Profile and initiates the processing of the uploaded file

  6. [10-11] The job profile ID is set for JobExecution

  7. [12-13] MARC file is split into batches of MARC records and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ

  8. [14-15] mod-srm reads batches from the DI_RAW_RECORDS_CHUNK_READ queue, parses MARC records, and saves the incoming records to the DB to provide them for data import logs. JobExecution is set to IN_PROGRESS status on first chunk received.

  9. [16-17] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Sends it to an appropriate Kafka queue (one message per MARC entry) - DI_MARC_FOR_UPDATE_RECEIVED

  10. [18-19] mod-inventory reads the message and processes instance match profile - tries to find an instance in mod-inventory-storage using match criteria from the match profile. Processes instance search result and issues an appropriate event (DI_INVENTORY_INSTANCE_NOT_MATCHED if no Instance exists satisfying search criteria, or DI_INVENTORY_INSTANCE_MATCHED if such Instance is found)

  11. [20] If instance is not found - mod-inventory sends DI_COMPLETED since no other profiles are placed for "non-matches" branch of the Instance Match Profile. The processing flow for the particular incoming record is completed. The sent event contains result event indication (DI_INVENTORY_INSTANCE_NOT_MATCHED) denoting instance matching result

    1. [21] mod-srm receives DI_COMPLETED and updates the import job progress

  12. [22] If instance is found - mod-inventory sends DI_INVENTORY_INSTANCE_MATCHED

  13. [23-25] mod-inventory reads the message and processes update instance action profile - maps instance according to mapping profile linked to update action profile and saves updated instance in mod-inventory-storage

  14. [26-29] mod-inventory updates underlying MARC record in mod-srs via HTTP and sends DI_INVENTORY_INSTANCE_UPDATED

  15. [30-31] mod-inventory reads the message and processes holdings match profile - searches holdings record in mod-inventory-storage using match criteria from the match profile. Processes holdings search result and issues an appropriate event (DI_INVENTORY_HOLDING_NOT_MATCHED if no holdings record is found, or DI_INVENTORY_HOLDING_MATCHED if holding found)

  16. [32] If holding is not found - mod-inventory sends DI_COMPLETED since no other profiles are placed for "non-matches" branch of the Holdings Match Profile. The processing flow for the particular incoming record is completed. The sent event contains result event indication (DI_INVENTORY_HOLDING_NOT_MATCHED) denoting holdings matching result

    1. [33] mod-srm receives DI_COMPLETED and updates the import job progress

  17. [34] If holding is found - mod-inventory sends DI_INVENTORY_HOLDING_MATCHED 

  18. [35-38] mod-inventory reads the message and processes update holdings action profile - maps holding according to mapping profile linked to the update action profile, saves updated holding in mod-inventory-storage and sends DI_INVENTORY_HOLDING_UPDATED

  19. [38-40] mod-inventory reads the message and processes item match profile - searches item in mod-inventory-storage using match criteria from the match profile. Processes item search result and issues an appropriate event (DI_INVENTORY_ITEM_NOT_MATCHED if no item is found, or DI_INVENTORY_ITEM_MATCHED if item found)

  20. [41] If item is not found - mod-inventory sends DI_COMPLETED since no other profiles are placed for "non-matches" branch of the Item Match Profile. The processing flow for the particular incoming record is completed. The sent event contains result event indication (DI_INVENTORY_ITEM_NOT_MATCHED ) denoting item matching result

    1. [42] mod-srm receives DI_COMPLETED and updates the import job progress

  21. [43] If item is found - mod-inventory sends DI_INVENTORY_ITEM_MATCHED

  22. [43-46] mod-inventory reads the message and processes update item action profile - maps item according to mapping profile linked to the update action profile, saves updated item in mod-inventory-storage and sends DI_INVENTORY_ITEM_UPDATED

  23. [49] In case of a successful item update mod-inventory sends DI_COMPLETED event

  24. [50] mod-srm receives DI_COMPLETED event and marks JobExecution status as "COMMITTED" if all incoming record have been processed without errors, otherwise sets JobExecution status to "ERROR"

Diagram (updated with changes made in Quesnelia R1 2024)

 

Flow (prior to changes made in Quesnelia)

  1. JobDefinition (uuid, profile - defines job type: insert/update) for import created in mod-srm

  2. MARC file + JobDefinition ID are uploaded from WEB client to mod-data-import

  3. MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ

  4. mod-srm reads batches from the queue, validates and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobStarts on first chunk received.

  5. mod-srs stores records into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED

  6. mod-srm reads the profile and creates JSON payload (containing parsed MARC, profile, mapping parameters) for processing. exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED

  7. mod-inventory reads message from DI_SRS_MARC_BIB_RECORD_CREATED. Tries to find entity according to match criteria (in mod-inventory-storage via OKAPI HTTP)

  8. If found: exports result to DI_INVENTORY_INSTANCE_MATCHED / DI_INVENTORY_HOLDING_MATCHED / DI_INVENTORY_ITEM_MATCHED

    1. mod-inventory receives match result and updates Instance/Holding/Item (according to action in profile) in mod-inventory-storage according to profile. And publishes result to applied topic (one per entity type) DI_INVENTORY_INSTANCE_UPDATED / DI_INVENTORY_HOLDING_UPDATED / DI_INVENTORY_ITEM_UPDATED

    2. mod-inventory reads message from prev. step (1 of 3 topics) and seeks for more actions (return to a.)  If no more actions in profile: export to DI_COMPLETE

  9. If not found:  DI_INVENTORY_INSTANCE_NOT_MATCHED / DI_INVENTORY_HOLDING_NOT_MATCHED / DI_INVENTORY_ITEM_NOT_MATCHED events are published

    1. mod-srm reads DI_INVENTORY_INSTANCE_NOT_MATCHED / DI_INVENTORY_HOLDING_NOT_MATCHED / DI_INVENTORY_ITEM_NOT_MATCHED events and updates job status accordingly with "NOT_MATCHED" status for record

 

Diagram (prior to changes made in Quesnelia)

 

Comments

For not empty "non-match" branch in profile a new flow description needs to be created