Skip to end of banner
Go to start of banner

1. Create

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 32 Next »

Description

Save MARC_BIB + create Instance, Holdings, Item


NB: Save MARC_BIB action is implicit

Flow

  1. JobDefinition (uuid, profile - defines job type: insert/update) for import created in mod-srm
  2. MARC file + JobDefinition ID are uploaded from WEB client to mod-data-import
  3. MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  4. mod-srm reads batches from the queue, validates and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobStarts on first chunk received.
  5. mod-srs stores records into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED
  6. mod-srm reads the profile and creates JSON payload (containing parsed MARC, profile, mapping parameters) for processing. exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED,
  7. mod-inventory reads the message, creates Instance. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message DI_INVENTORY_INSTANCE_CREATED_READY_FOR_POST_PROCESSING
  8. mod-srs reads message and updates according entry (Instance HRID is set to '001' MARC_BIB field, the value from '001' is moved to '035', Instance ID is set to '999 ff i' field). Creates new messages with updated payload DI_SRS_MARC_BIB_INSTANCE_HRID_SET and DI_INVENTORY_INSTANCE_CREATED or DI_COMPLETED (in case JobProfile contains action only for Instance create)
  9. mod-inventory reads message from DI_SRS_MARC_BIB_INSTANCE_HRID_SET and updates Instance with updated identifiers fields - no new messages are sent at this point.
  10. mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, creates Holdings. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED
  11. mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, creates Items. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED
  12. mod-inventory reads message and exports to DI_COMPLETED

Diagram

Source

Comments

There's a potential process break on step 7 (we need to send multiple Kafka messages at once). To be addressed in stabilization plan


  • No labels