Authority remapping (MODMARCMIG-10)

[MODMARCMIG-6] Implement Async Records Mapping Mechanism for MARC Migration (Prepare Chunks) Created: 12/Dec/23  Updated: 08/Feb/24  Resolved: 08/Feb/24

Status: Closed
Project: mod-marc-migrations
Components: None
Affects versions: None
Fix versions: 1.0.0
Parent: Authority remapping

Type: Story Priority: P3
Reporter: Pavlo Smahin Assignee: Viacheslav Kolesnyk
Resolution: Done Votes: 0
Labels: back-end, epam-spitfire
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File MODMARCMIG-6_verified.png    
Issue links:
Gantt End to Start
has to be done before MODMARCMIG-7 Implement Async Records Mapping Mecha... In Progress
has to be done after MODMARCMIG-5 POST endpoint: Register new marc-migr... Closed
Relates
relates to UXPROD-4082 Long term solution for applying mappi... In Progress
Sprint: Spitfire Sprint 184
Story Points: 8
Development Team: Spitfire
Release: Quesnelia (R1 2024)
Epic Link: Authority remapping

 Description   

Overview:

Following the successful registration of MARC migration operations through the POST endpoint, this story focuses on implementing an asynchronous mechanism for mapping records. The objective is to prepare chunks of records with the first and last record IDs, facilitating efficient and incremental mapping processes.

Requirements/Scope:

  1. Develop an asynchronous mechanism for mapping records corresponding to the registered MARC migration operation.
  2. Prepare record chunks by identifying the first and last record IDs for mapping purposes.

Approach:

  1. Trigger Async Processing Post Operation Registration
    1. Initiate the asynchronous processing immediately after a new MARC migration operation is successfully registered.
    2. Implement configurable thread pool for execution (based on env var).
  2. Define Chunks and Determine Start/End Record IDs
    1. Design a mechanism to segment records into manageable chunks based on their identifiers.
    2. Determine the start and end record IDs for each chunk to facilitate incremental mapping.
  3. Create Database Table for Chunk Representation
    1. Establish a dedicated table within the database to represent and store the segmented chunks.
    2. Define the table structure to accommodate chunk-related information such as chunk ID, operation ID, start record ID, end record ID, etc.
  4. Persist Chunks to the Database Table:
    1. Develop the logic to save the identified chunks into the newly created table in the database.
    2. Ensure that each chunk's relevant information, including start and end record IDs, is accurately stored.
  5. Update Operation Status to DATA_MAPPING:
    1. Modify the status of the registered MARC migration operation to indicate that the data mapping process has commenced.
    2. Change the operation status to "DATA_MAPPING" to signify the transition to the mapping phase.

Additional info:
POC for defining chunks and mapping: https://github.com/folio-org/mod-entities-links/commits/MODELINKS-84-long-term-poc/

Acceptance Criteria:

  • covered by unit tests
     


 Comments   
Comment by Viacheslav Kolesnyk [ 07/Feb/24 ]

Note for testing: after some time GET endpoint should return operation with status “data mapping“

Comment by Valery_Pilko [ 08/Feb/24 ]

Verified on Snapshot environment.
GET endpoint returns new status: "data_mapping"
See attached screenshot:

Generated at Thu Feb 08 22:31:54 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.