Skip to end of banner
Go to start of banner

Data Import

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

Flow description

  1. JobDefinition (uuid, profile - defines job type: insert/update) for import created in mod-srm
  2. MARC file + JobDefinition ID are uploaded from WEB client to mod-data-import (stored in memory, can be persisted. possible oom)
  3. MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  4. mod-srm reads batches from the queue, validates and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobStarts on first chunk received. In case JobProfile contains an action for MARC_BIB update - an event is published to different a queue DI_MARC_BIB_FOR_UPDATE_RECEIVED and step 5 is skipped.
  5. mod-srs stores records (unless MARC_BIB update action is present in JobProfile) into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED
  6. mod-srm reads the profile and creates JSON payload (containing parsed MARC, profile, mapping parameters) for processing. exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED, DI_SRS_MARC_AUTHORITY_RECORD_CREATED, DI_SRS_MARC_HOLDING_RECORD_CREATED or DI_EDIFACT_RECORD_CREATED

FOR CREATE

  1. mod-inventory reads the message, creates Instance. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message ('INVENTORY_INSTANCE_CREATED')
  2. mod-srs reads message and updates according entry with instance id. Creates new message with updated payload ('DI_SRS_MARC_BIB_INSTANCE_HRID_SET')
  3. mod-inventory updates Instance with updated fields (Kateryna Senchenko export or same thread?)
  4. mod-inventory reads the message, creates Holdings. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message (Kateryna Senchenko)
  5. mod-inventory reads the message, creates Items. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message (Kateryna Senchenko)
  6. mod-inventory reads message and exports to 'DI_COMPLETED' (Kateryna Senchenko)

FOR UPDATE

  1. mod-inventory / mod-srs reads the message from (topic: Kateryna Senchenko)

UPDATE:

  1. mod-inventory reads message from 'topic-1'. Tries to find entity according to match criteria (in mod-inventory-storage via OKAPI HTTP)
  2. If found: exports result to ('inventory instance matched': Kateryna Senchenko )
    1. mod-inventory receives match result and updates Instance/Holding/Item (according to action in profile) in mod-inventory-storage according to profile. And publishes result to applied topic (one per entity type) Kateryna Senchenko)
    2. mod-inventory reads message from prev. step (1 of 3 topics) and seeks for more actions (return to a.)  If no more actions in profile: export to 'DI_COMPLETED'
  3. If not found: follow 'NON-MATCHED' branch in profile (go to 'CREATE' or 'UPDATE' or export to 'DI_COMPLETED' for empty actions)


AFTER PROCESSING

  1. mod-srm reads 'DI_COMPLETED'/'DI_ERROR' updates job progress


  • No labels