...

1. JobDefinition (uuid, profile - defines job type: insert/update) for import created in mod-srm
2. MARC file + JobDefinition ID are uploaded from WEB client to mod-data-import
3. MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
   
4. mod-srm reads batches from the queue, validates and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobStarts on first chunk received.
5. mod-srs stores records into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED
6. mod-srm reads the profile and creates JSON payload (containing parsed MARC, profile, mapping parameters) for processing. exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED,
7. mod-inventory reads the message, creates Instance. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message DI_INVENTORY_INSTANCE_CREATED_READY_FOR_POST_PROCESSING
   
8. mod-srs reads message and updates according entry (Instance HRID is set to '001' MARC_BIB field, the value from '001' is moved to '035', Instance ID is set to '999 ff i' field). Creates new messages with updated payload DI_SRS_MARC_BIB_INSTANCE_HRID_SET and DI_INVENTORY_INSTANCE_CREATED or DI_COMPLETED (in case JobProfile contains action only for Instance create)
9. mod-inventory reads message from DI_SRS_MARC_BIB_INSTANCE_HRID_SET and updates Instance with updated identifiers fields - no new messages are sent at this point.
10. mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, creates Holdings. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED
11. mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, creates Items. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED
12. mod-inventory reads message and exports to DI_COMPLETED

Diagram

Source

Image RemovedImage Added

Comments

...