Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Data-import process creating incorrect topics in Kafka

Description

Environment: MCPT

Description:

An issue has been identified in the data-import process for authority records in our Kafka system. The current implementation is creating new Kafka topics for each individual tenant, which has resulted in topics like "mcpt.cs00000001_0001.authorities.authority" to "mcpt.cs00000001_0062.authorities.authority".

Expected behavior:t
The system should always create and use a generic topic named "mcpt.ALL.authorities.authority" for all tenants.

Moreover, it was found that after manually deleting the topic "mcpt.cs00000001_0001.authorities.authority" from kafdrop, the following data-import run on tenant 'cs00000001_0001' re-created the same incorrect topic again.

We need to correct this so that the data-import process consistently uses the single expected topic "mcpt.ALL.authorities.authority".

CSP Request Details

None

CSP Rejection Details

None

Potential Workaround

None

Checklist

hide

Activity

Show:

Oleksii Petrenko March 25, 2024 at 7:38 AM

Deployed to Poppy BF env. Please proceed with verification

Oleksii Petrenko March 24, 2024 at 7:33 AM

Guys, please add CSP request details

Viacheslav Kolesnyk February 14, 2024 at 10:25 AM
Edited

Dev note: Use KafkaUtils from folio-service-tools which checks if tenant should be used. Delete existing method in KafkaUtils (mod-entities-links) or whole class if not needed. Verify with unit test on EventProducer, check topic in producer record. Check also other modules topics creation (mod-search, mod-inventory-storage), create appropriate tasks

Done

Details

Assignee

Reporter

Labels

Priority

Story Points

Sprint

Development Team

Spitfire

Fix versions

Release

Poppy (R2 2023) Service Patch #3

RCA Group

Data related (ex. Can be detected with large dataset only)

CSP Approved

Yes

Affected releases

Poppy (R2 2023)

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs
Created February 13, 2024 at 4:25 PM
Updated August 12, 2024 at 12:30 PM
Resolved February 28, 2024 at 8:20 AM
Configure
TestRail: Cases
TestRail: Runs

Flag notifications