Skip to end of banner
Go to start of banner

Data Import

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

Flow description

  1. JobDefinition (uuid, profile - defines job type: insert/update) for import created in mod-srm
  2. MARC file + JobDefinition ID are uploaded from WEB client to mod-data-import (stored in memory, can be persisted. possible oom)
  3. MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  4. mod-srm reads batches from the queue, validates and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobStarts on first chunk received. In case JobProfile contains an action for MARC_BIB update - an event is published to different a queue DI_MARC_BIB_FOR_UPDATE_RECEIVED and step 5 is skipped.
  5. mod-srs stores records (unless MARC_BIB update action is present in JobProfile) into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED
  6. mod-srm reads the profile and creates JSON payload (containing parsed MARC, profile, mapping parameters) for processing. exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED, DI_SRS_MARC_AUTHORITY_RECORD_CREATED, DI_SRS_MARC_HOLDING_RECORD_CREATED or DI_EDIFACT_RECORD_CREATED

FOR CREATE

  1. mod-inventory reads the message, creates Instance. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message DI_INVENTORY_INSTANCE_CREATED_READY_FOR_POST_PROCESSING
  2. mod-srs reads message and updates according entry (Instance HRID is set to '001' MARC_BIB field, the value from '001' is moved to '035', Instance ID is set to '999 ff i' field). Creates new messages with updated payload DI_SRS_MARC_BIB_INSTANCE_HRID_SET, and DI_INVENTORY_INSTANCE_CREATED or DI_COMPLETED (in case JobProfile contains action only for Instance create)
  3. mod-inventory reads message from DI_SRS_MARC_BIB_INSTANCE_HRID_SET and updates Instance with updated identifiers fields - no new messages are sent at this point.
  4. mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, creates Holdings. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED
  5. mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, creates Items. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED
  6. mod-inventory reads message and exports to DI_COMPLETED

FOR UPDATE

  1. mod-inventory / mod-srs reads the message from (topic: Kateryna Senchenko)

UPDATE:

  1. mod-inventory reads message from 'topic-1'. Tries to find entity according to match criteria (in mod-inventory-storage via OKAPI HTTP)
  2. If found: exports result to ('inventory instance matched': Kateryna Senchenko )
    1. mod-inventory receives match result and updates Instance/Holding/Item (according to action in profile) in mod-inventory-storage according to profile. And publishes result to applied topic (one per entity type) Kateryna Senchenko)
    2. mod-inventory reads message from prev. step (1 of 3 topics) and seeks for more actions (return to a.)  If no more actions in profile: export to 'DI_COMPLETED'
  3. If not found: follow 'NON-MATCHED' branch in profile (go to 'CREATE' or 'UPDATE' or export to 'DI_COMPLETED' for empty actions)


AFTER PROCESSING

  1. mod-srm reads 'DI_COMPLETED'/'DI_ERROR' updates job progress


  • No labels