Flow (prior to Q updates)

1. [1-4] UploadDefinition is created for importing files in mod-data-import
2. [2-3] JobExecution is created in mod-srm
3. [5-7] MARC files are uploaded from WEB client to mod-data-import
4. [6] Uploaded file is stored in memory of mod-data-import
5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files
6. [10-11] Profile ID is set for JobExecution
7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
   
8. [13-15] mod-srm reads batches from the queue, parses MARC records and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobExecution is set to IN_PROGRESS status on first chunk received.
9. [16-17] mod-srs stores records into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED
10. [18-19] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED
11. [20-22] mod-inventory reads the message, creates Instance. Stores (via OKAPI HTTP) in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. Exports message DI_INVENTORY_INSTANCE_CREATED_READY_FOR_POST_PROCESSING
    
12. [25-30] The post-processing stage: mod-srs reads message and updates according entry (Instance HRID is set to '001' MARC_BIB field, the value from '001' + '003' is moved to '035', Instance ID is set to '999 ff i' field). Creates new messages with updated payload DI_SRS_MARC_BIB_INSTANCE_HRID_SET. mod-inventory reads message from DI_SRS_MARC_BIB_INSTANCE_HRID_SET and updates Instance with updated identifiers fields - no new messages are sent at this point.
13. [31] At the same time as in step [27] mod-srs sends DI_INVENTORY_INSTANCE_CREATED event
14. [32-34] mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, creates Holdings. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED
15. [37-40] mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, creates Items. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED
16. [43-44] mod-inventory reads message and exports to DI_COMPLETED
17. [44-45] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

 Diagram (prior to Q updates)

Source

Image Modified