/
1. Create MARC Bib, create Instance, Holdings, Item

1. Create MARC Bib, create Instance, Holdings, Item

Feb 13, 2024

Description

Save MARC_BIB + create Instance, Holdings, Item

NB: Save MARC_BIB action is implicit

Flow (Updated with changes made in Quesnelia R1 2024)

  1. [1-4] UploadDefinition is created for importing files in mod-data-import

  2. [2-3] JobExecution is created in mod-srm

  3. [5-7] MARC files are uploaded from WEB client to mod-data-import

  4. [6] Uploaded file is split into chunks in mod-data-import

  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files

  6. [10-11] Profile ID is set for JobExecution

  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ

  8. [13-15] mod-srm reads batches from the queue, parses MARC records and saves the Incoming records in the DB. JobExecution is set to IN_PROGRESS status on first chunk received.

  9. [15-16] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_INCOMING_MARC_BIB_RECORD_PARSED

  10. [17-20] mod-inventory reads the message, MARC Bib value from '001' + '003' is moved to '035', after which the Instance is mapped and created  via HTTP in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. 

  11. [21-23] Instance HRID is set to '001' MARC_BIB field, Instance ID is set to '999 ff i' field and is saved in mod-srs via HTTP

  12. [26] mod-inventory sends DI_INVENTORY_INSTANCE_CREATED containing Instance and MARC Bib in the payload

  13. [26-29] mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, creates Holdings. Stores (via HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED

  14. [32-35] mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, creates Items. Stores (via HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED

  15. [39] mod-inventory reads message and exports to DI_COMPLETED

  16. [40] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED



Flow (prior to Q updates)

  1. [1-4] UploadDefinition is created for importing files in mod-data-import

  2. [2-3] JobExecution is created in mod-srm

  3. [5-7] MARC files are uploaded from WEB client to mod-data-import

  4. [6] Uploaded file is stored in memory of mod-data-import

  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files

  6. [10-11] Profile ID is set for JobExecution

  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ

  8. [13-15] mod-srm reads batches from the queue, parses MARC records and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobExecution is set to IN_PROGRESS status on first chunk received.

  9. [16-17] mod-srs stores records into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED

  10. [18-19] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED

  11. [20-22] mod-inventory reads the message, creates Instance. Stores (via OKAPI HTTP) in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. Exports message DI_INVENTORY_INSTANCE_CREATED_READY_FOR_POST_PROCESSING

  12. [25-30] The post-processing stage: mod-srs reads message and updates according entry (Instance HRID is set to '001' MARC_BIB field, the value from '001' + '003' is moved to '035', Instance ID is set to '999 ff i' field). Creates new messages with updated payload DI_SRS_MARC_BIB_INSTANCE_HRID_SET. mod-inventory reads message from DI_SRS_MARC_BIB_INSTANCE_HRID_SET and updates Instance with updated identifiers fields - no new messages are sent at this point.

  13. [31] At the same time as in step [27] mod-srs sends DI_INVENTORY_INSTANCE_CREATED event

  14. [32-34] mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, creates Holdings. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED

  15. [37-40] mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, creates Items. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED

  16. [43-44] mod-inventory reads message and exports to DI_COMPLETED

  17. [44-45] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

Diagram 

(Updated with changes made in Quesnelia R1 2024)

Source



 Diagram (prior to Q updates)

Source





Comments

There's a potential process break on step 7 (we need to send multiple Kafka messages at once). To be addressed in stabilization plan