Skip to end of banner
Go to start of banner

Data Import

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Flow description

  1. JobDefinition (uuid, profile - defines job type: insert/update) for import created in mod-srm
  2. MARC file + JobDefinition ID are uploaded from WEB client to mod-data-import (stored in memory, can be persisted. possible oom)
  3. MARC records are packed into batches and put to Kafka queue
  4. mod-srm reads batches from the queue, validates and passes to mod-srs via Kafka queue. JobStarts on first chunk received
  5. mod-srs stores records (creates new generation for updates, has exclusion! Kateryna Senchenko ) into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') -
  6. mod-srm reads the profile and creates JSON payload (containing parsed MARC, profile, mapping parameters) for processing. exports it to Kafka queue (one message per MARC entry), (topic-1 Kateryna Senchenko)

FOR CREATE

  1. mod-inventory reads the message, creates Instance. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message ('INVENTORY_INSTANCE_CREATED')
  2. mod-srs reads message and updates according entry with instance id. Creates new message with updated payload ('DI_SRS_MARC_BIB_INSTANCE_HRID_SET')
  3. mod-inventory updates Instance with updated fields (Kateryna Senchenko export or same thread?)
  4. mod-inventory reads the message, creates Holdings. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message (Kateryna Senchenko)
  5. mod-inventory reads the message, creates Items. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message (Kateryna Senchenko)
  6. mod-inventory reads message and exports to 'DI_COMPLETED' (Kateryna Senchenko)

FOR UPDATE

  1. mod-inventory / mod-srs reads the message from (topic: Kateryna Senchenko)

UPDATE:

  1. mod-inventory reads message from 'topic-1'. Tries to find entity according to match criteria (in mod-inventory-storage via OKAPI HTTP)
  2. If found: exports result to ('inventory instance matched': Kateryna Senchenko )
    1. mod-inventory receives match result and updates Instance/Holding/Item (according to action in profile) in mod-inventory-storage according to profile. And publishes result to applied topic (one per entity type) Kateryna Senchenko)
    2. mod-inventory reads message from prev. step (1 of 3 topics) and seeks for more actions (return to a.)  If no more actions in profile: export to 'DI_COMPLETED'
  3. If not found: follow 'NON-MATCHED' branch in profile (go to 'CREATE' or 'UPDATE' or export to 'DI_COMPLETED' for empty actions)


AFTER PROCESSING

  1. mod-srm reads 'DI_COMPLETED'/'DI_ERROR' updates job progress


  • No labels