Description

Save MARC_BIB + create Instance, Holdings, Item

NB: Save MARC_BIB action is implicit

Flow (Updated with changes made in Quesnelia R1 2024)

[1-4] UploadDefinition is created for importing files in mod-data-import
[2-3] JobExecution is created in mod-srm
[5-7] MARC files are uploaded from WEB client to mod-data-import
[6] Uploaded file is split into chunks in mod-data-import
[8-9] User selects the JobProfile and initiates the processing of the uploaded files
[10-11] Profile ID is set for JobExecution
[12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
[13-15] mod-srm reads batches from the queue, parses MARC records and saves the Incoming records in the DB. JobExecution is set to IN_PROGRESS status on first chunk received.
[15-16] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_INCOMING_MARC_BIB_RECORD_PARSED
[17-20] mod-inventory reads the message, MARC Bib value from '001' + '003' is moved to '035', after which the Instance is mapped and created via HTTP in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated.
[21-23] Instance HRID is set to '001' MARC_BIB field, Instance ID is set to '999 ff i' field and is saved in mod-srs via HTTP
[26] mod-inventory sends DI_INVENTORY_INSTANCE_CREATED containing Instance and MARC Bib in the payload
[26-29] mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, creates Holdings. Stores (via HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED
[32-35] mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, creates Items. Stores (via HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED
[39] mod-inventory reads message and exports to DI_COMPLETED
[40] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

Click here to expand...

Flow (prior to Q updates)

[1-4] UploadDefinition is created for importing files in mod-data-import
[2-3] JobExecution is created in mod-srm
[5-7] MARC files are uploaded from WEB client to mod-data-import
[6] Uploaded file is stored in memory of mod-data-import
[8-9] User selects the JobProfile and initiates the processing of the uploaded files
[10-11] Profile ID is set for JobExecution
[12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
[13-15] mod-srm reads batches from the queue, parses MARC records and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobExecution is set to IN_PROGRESS status on first chunk received.
[16-17] mod-srs stores records into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED
[18-19] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED
[20-22] mod-inventory reads the message, creates Instance. Stores (via OKAPI HTTP) in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. Exports message DI_INVENTORY_INSTANCE_CREATED_READY_FOR_POST_PROCESSING
[25-30] The post-processing stage: mod-srs reads message and updates according entry (Instance HRID is set to '001' MARC_BIB field, the value from '001' + '003' is moved to '035', Instance ID is set to '999 ff i' field). Creates new messages with updated payload DI_SRS_MARC_BIB_INSTANCE_HRID_SET. mod-inventory reads message from DI_SRS_MARC_BIB_INSTANCE_HRID_SET and updates Instance with updated identifiers fields - no new messages are sent at this point.
[31] At the same time as in step [27] mod-srs sends DI_INVENTORY_INSTANCE_CREATED event
[32-34] mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, creates Holdings. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED
[37-40] mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, creates Items. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED
[43-44] mod-inventory reads message and exports to DI_COMPLETED
[44-45] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

Diagram

(Updated with changes made in Quesnelia R1 2024)

Source

Click here to expand...

Diagram (prior to Q updates)

Source

Comments

There's a potential process break on step 7 (we need to send multiple Kafka messages at once). To be addressed in stabilization plan