Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Index Linked Data via Kafka in Search Index (temporary solution)

Description

What we need:

We have data from the ETL, which is saved in the DB. We must introduce a temporary solution for the upcoming demo that will allow us to index data coming from the ETL.

Notes:

  • This is a temporary solution, that we plan to reconsider after the Thin Thread

Solution Options:

  1. We can create an endpoint where we can initiate the process of reindexing 

  2. We can index data when the process is started

Acceptance Criteria:

  1. The same data that is passed in ETL is indexed in the newly created Search Index (created for the Linked Data)

  2. We can use Search functionality for the data passed in the ETL.

Environment

None

Potential Workaround

None

Attachments

4

is blocked by

Checklist

hide

TestRail: Results

Activity

Show:

Tetiana TymoshenkoSeptember 28, 2023 at 9:02 AM

Tested on missing links env

Build version: #86

Test cases and evidences attached.

Search results before and after reindex have been compared to make sure data becomes indexed and searchable. 

Tetiana TymoshenkoSeptember 27, 2023 at 12:01 PM

Records are still not searchable after reindex initiating:

E.g. "http://bibfra.me/vocab/marc/isbn": ["9781645271796"]
    "http://bibfra.me/vocab/marc/lccn": ["2019022750"]

Christmas shopaholic
"http://bibfra.me/vocab/marc/isbn": ["9780593132821"],
    "http://bibfra.me/vocab/marc/lccn": ["2019023953"]

etc. Above records are from DB and also can be found on Marva Load page.

, can you please take a look? Thanks.

Tetiana TymoshenkoSeptember 21, 2023 at 8:08 AM

Flag added

Testing is blocked by https://folio-org.atlassian.net/browse/MODLD-115

Asadbek MuminovSeptember 12, 2023 at 10:07 AM
Edited

How to reindex data that is passed fro ETL. 

  1. Make `PUT` request with empty body to the endpoint `/reindex`
  2. The endpoint returns 204 No Content and starts processing of reindexing 
  3. Process takes all data from db and send them to kafka topic for the given tenant
  4. After some time, all data should be searchable 

We need to test whether all data is being retrieved from db -> send to Kafka -> create index -> and become searchable 

Done

Details

Assignee

Reporter

Tester Assignee

Labels

Priority

Story Points

Sprint

Development Team

Citation

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs

Created July 31, 2023 at 10:51 AM
Updated February 8, 2024 at 7:50 PM
Resolved September 29, 2023 at 12:34 PM
TestRail: Cases
TestRail: Runs