Skip to end of banner
Go to start of banner

1. Create MARC Bib, create Instance, Holdings, Item

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 37 Current »

Description

Save MARC_BIB + create Instance, Holdings, Item


NB: Save MARC_BIB action is implicit

Flow (Updated with changes made in Quesnelia R1 2024)

  1. [1-4] UploadDefinition is created for importing files in mod-data-import
  2. [2-3] JobExecution is created in mod-srm
  3. [5-7] MARC files are uploaded from WEB client to mod-data-import
  4. [6] Uploaded file is split into chunks in mod-data-import
  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files
  6. [10-11] Profile ID is set for JobExecution
  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  8. [13-15] mod-srm reads batches from the queue, parses MARC records and saves the Incoming records in the DB. JobExecution is set to IN_PROGRESS status on first chunk received.
  9. [15-16] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_INCOMING_MARC_BIB_RECORD_PARSED
  10. [17-20] mod-inventory reads the message, MARC Bib value from '001' + '003' is moved to '035', after which the Instance is mapped and created  via HTTP in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. 
  11. [21-23] Instance HRID is set to '001' MARC_BIB field, Instance ID is set to '999 ff i' field and is saved in mod-srs via HTTP
  12. [26] mod-inventory sends DI_INVENTORY_INSTANCE_CREATED containing Instance and MARC Bib in the payload
  13. [26-29] mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, creates Holdings. Stores (via HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED
  14. [32-35] mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, creates Items. Stores (via HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED
  15. [39] mod-inventory reads message and exports to DI_COMPLETED
  16. [40] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED


 Click here to expand...

Flow (prior to Q updates)

  1. [1-4] UploadDefinition is created for importing files in mod-data-import
  2. [2-3] JobExecution is created in mod-srm
  3. [5-7] MARC files are uploaded from WEB client to mod-data-import
  4. [6] Uploaded file is stored in memory of mod-data-import
  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files
  6. [10-11] Profile ID is set for JobExecution
  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  8. [13-15] mod-srm reads batches from the queue, parses MARC records and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobExecution is set to IN_PROGRESS status on first chunk received.
  9. [16-17] mod-srs stores records into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED
  10. [18-19] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED
  11. [20-22] mod-inventory reads the message, creates Instance. Stores (via OKAPI HTTP) in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. Exports message DI_INVENTORY_INSTANCE_CREATED_READY_FOR_POST_PROCESSING
  12. [25-30] The post-processing stage: mod-srs reads message and updates according entry (Instance HRID is set to '001' MARC_BIB field, the value from '001' + '003' is moved to '035', Instance ID is set to '999 ff i' field). Creates new messages with updated payload DI_SRS_MARC_BIB_INSTANCE_HRID_SET. mod-inventory reads message from DI_SRS_MARC_BIB_INSTANCE_HRID_SET and updates Instance with updated identifiers fields - no new messages are sent at this point.
  13. [31] At the same time as in step [27] mod-srs sends DI_INVENTORY_INSTANCE_CREATED event
  14. [32-34] mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, creates Holdings. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED
  15. [37-40] mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, creates Items. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED
  16. [43-44] mod-inventory reads message and exports to DI_COMPLETED
  17. [44-45] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

Diagram 

(Updated with changes made in Quesnelia R1 2024)

Source

 Click here to expand...

 Diagram (prior to Q updates)

Source



Comments

There's a potential process break on step 7 (we need to send multiple Kafka messages at once). To be addressed in stabilization plan


  • No labels