Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Copy ETL CLI and Migrations modules into Linked Data repositories

Description

What we need:

  1. We need to create our version of the CLI and Migration modules

  2. The history of changes must remain if possible

  3. We also need to keep all the dependencies on other modules

 

Acceptance criteria:

  1. We can build the module: for example, by using the local stack

  2. The modules must have all required dependencies

  3. Our ETL pipeline can pull it for the build

Environment

None

Potential Workaround

None

Checklist

hide

TestRail: Results

Activity

Show:

Victor Bodrov August 11, 2023 at 2:35 AM
Edited

The code I used in the demo added to this branch
The fix for repeated execution of ETL process is here

Victor Bodrov August 9, 2023 at 1:22 PM
Edited

Need verification by someone who has localstack locally.
Steps

  1. Run docker-compose from mod-linked-data to deploy postgres database locally

  2. Run LinkedDataApplication from mod-linked-data to fill the databse with initial data

  3. Clone linked-data-etl-localstack and deploy the environment using the commands:
    make initialize-tools
    make update-localstack
    make good-morning

  4. Run the pipeline (check "Running the pipeline" in the guide)

  5. Check that the database public schema was populated with the data (e.g. resources table)

Done

Details

Assignee

Reporter

Labels

Priority

Story Points

Sprint

Development Team

Citation

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs

Created August 7, 2023 at 8:30 AM
Updated February 8, 2024 at 7:50 PM
Resolved August 16, 2023 at 12:54 PM
TestRail: Cases
TestRail: Runs