Versions Compared

Old Version 37

changes.mady.by.user Kateryna Senchenko

Saved on

compared with

New Version Current

changes.mady.by.user Kateryna Senchenko

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Description

Save MARC_BIB + create Instance, Holdings, Item


NB: Save MARC_BIB action is implicit

Flow (Updated with changes made in Quesnelia R1 2024)

  1. [1-4] UploadDefinition is created for importing files in mod-data-import
  2. [2-3] JobExecution is created in mod-srm
  3. [5-7] MARC files are uploaded from WEB client to mod-data-import
  4. [6] Uploaded file is split into chunks in mod-data-import
  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files
  6. [10-11] Profile ID is set for JobExecution
  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  8. [13-15] mod-srm reads batches from the queue, parses MARC records and saves the Incoming records in the DB. JobExecution is set to IN_PROGRESS status on first chunk received.
  9. [15-16] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_INCOMING_MARC_BIB_RECORD_PARSED
  10. [17-20] mod-inventory reads the message, MARC Bib value from '001' + '003' is moved to '035', after which the Instance is mapped and created  via HTTP in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. 
  11. [21-23] Instance HRID is set to '001' MARC_BIB field, Instance ID is set to '999 ff i' field and is saved in mod-srs via HTTP
  12. [26] mod-inventory sends DI_INVENTORY_INSTANCE_CREATED containing Instance and MARC Bib in the payload
  13. [26-29] mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, creates Holdings. Stores (via HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED
  14. [32-35] mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, creates Items. Stores (via HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED
  15. [39] mod-inventory reads message and exports to DI_COMPLETED
  16. [40] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED


Expand

Flow (prior to Q updates)

  1. [1-4] UploadDefinition is created for importing files in mod-data-import
  2. [2-3] JobExecution is created in mod-srm
  3. [5-7] MARC files are uploaded from WEB client to mod-data-import
  4. [6] Uploaded file is stored in memory of mod-data-import
  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files
  6. [10-11] Profile ID is set for JobExecution
  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  8. [13-15] mod-srm reads batches from the queue, parses MARC records and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobExecution is set to IN_PROGRESS status on first chunk received.
  9. [16-17] mod-srs stores records into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED
  10. [18-19] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED
  11. [20-22] mod-inventory reads the message, creates Instance. Stores (via OKAPI HTTP) in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. Exports message DI_INVENTORY_INSTANCE_CREATED_READY_FOR_POST_PROCESSING
  12. [25-30] The post-processing stage: mod-srs reads message and updates according entry (Instance HRID is set to '001' MARC_BIB field, the value from '001' + '003' is moved to '035', Instance ID is set to '999 ff i' field). Creates new messages with updated payload DI_SRS_MARC_BIB_INSTANCE_HRID_SET. mod-inventory reads message from DI_SRS_MARC_BIB_INSTANCE_HRID_SET and updates Instance with updated identifiers fields - no new messages are sent at this point.
  13. [31] At the same time as in step [27] mod-srs sends DI_INVENTORY_INSTANCE_CREATED event
  14. [32-34] mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, creates Holdings. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED
  15. [37-40] mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, creates Items. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED
  16. [43-44] mod-inventory reads message and exports to DI_COMPLETED
  17. [44-45] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

Diagram 

(Updated with changes made in Quesnelia R1 2024)

Source


Expand

 Diagram (prior to Q updates)

Source



Comments

Anchor
issue1
issue1
There's a potential process break on step 7 (we need to send multiple Kafka messages at once). To be addressed in stabilization plan