Batch Importer (Bib/Acq) (UXPROD-47)

[UXPROD-2929] NFR: Increase security of Kafka for Data Import and PubSub Created: 02/Mar/21  Updated: 12/Jul/21  Resolved: 12/Jul/21

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: R2 2021
Parent: Batch Importer (Bib/Acq)

Type: New Feature Priority: P2
Reporter: Ann-Marie Breaux (Inactive) Assignee: Ann-Marie Breaux (Inactive)
Resolution: Done Votes: 0
Labels: NFR, data-import, epam-folijet, r2-2021-at-risk, security, security-reviewed, testing
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Cloners
is cloned by UXPROD-2935 NFR: Increase security of Kafka for m... Closed
Defines
defines UXPROD-47 Batch Importer (Bib/Acq) Analysis Complete
is defined by MODDATAIMP-435 Review DevOps work, test, and create ... Closed
is defined by MODPUBSUB-171 Provide properties for Kafka security... Closed
is defined by MODPUBSUB-182 Provide properties for Kafka security... Closed
Duplicate
is duplicated by MODPUBSUB-54 SPIKE: Design Security for pub-sub Closed
Relates
relates to UXPROD-2931 NFR: Increase security of Kafka for R... Closed
Epic Link: Batch Importer (Bib/Acq)
Front End Estimate: Very Small (VS) < 1day
Front End Estimator: Ann-Marie Breaux (Inactive)
Front-End Confidence factor: Medium
Back End Estimate: Large < 10 days
Back End Estimator: Oleksii Kuzminov
Development Team: Folijet
PO Rank: 113
Cap Plan Fix Version (DO NOT CHANGE): R2 2021

 Description   

Next step: Ann-Marie Breaux plan a meeting the week of 19 April with team leads and POs and Vasily to discuss implementation, and draft spikes and stories

Latest documentation: https://folio-org.atlassian.net/wiki/pages/viewpage.action?spaceKey=FOLIJET&title=Enabling+SSL+and+ACL+for+Kafka

Current situation or problem: In Iris, Data import has migrated most of its transactions (but not all) to direct Kafka connections instead of mod-pubsub.

There were some concerns raised in the community regarding how secure the direct connection will be. To address these concerns, the new solution was designed: https://folio-org.atlassian.net/wiki/display/DD/Temporary+Kafka+security+solution.
The solution was reviewed and approved by the Security group and Tech Council.

Multi-tenancy on Kafka's side is implemented for the modules differently, so it will take time to make the changes in them that unify the multi-tenancy approach.
However, the direct Kafka connections should be secured in R1, so a simplified version of the solution is proposed for now.

In scope

  • Add module-level Kafka user credentials support to Data import and PubSub modules. The credentials should be provided to all producers and consumers of a module with other Kafka client settings. Changes in PubSub are required since once Kafka authentication and authorization are enabled, the PubSub will need to pass through them as well.
  • Add TLS (Transport Layer Security) support to the same modules. Same here, the settings should be provided to all producers and consumers of a module with other Kafka client settings.

Out of scope
This work is also needed for ElasticSearch and Remote Storage, but those applications/modules are managed by other dev teams

Proposed solution/How it could be implemented:

  • ModuleDescriptor should be updated to include the new Kafka settings: TLS, and, for now, user credentials (the credentials later could be injected to container a different way, for instance, as EnvironmentVariables)
  • Update a class that represents Kafka config
  • Update a class(es) that creates and assigns the config to Kafka producers and consumers
  • Test the updates

Links to additional info
https://folio-org.atlassian.net/wiki/display/DD/Temporary+Kafka+security+solution

Questions



 Comments   
Comment by Ann-Marie Breaux (Inactive) [ 03/Mar/21 ]

Hi Oleksii Kuzminov This was discussed at Tech Council today, and they confirmed it can be delayed until R2. So less time pressure for the estimate and stories. Let's get our R1 releases out and then deal with this.

Comment by Ann-Marie Breaux (Inactive) [ 31/Mar/21 ]

Hi Vasily Gancharov - could you review the description here and outline the first steps for covering this requirement? If any questions or discussion needed, please contact Oleksii Kuzminov. Thank you!

Comment by Ann-Marie Breaux (Inactive) [ 21/Apr/21 ]

Oleksii Kuzminov Please see MSEARCH-105 Closed and MSEARCH-106 Closed for the Falcon team security stories. We may be able to borrow some of this or use as templates for the security stories that we'll need to create and handle in R2. Thank you!

Comment by Ann-Marie Breaux (Inactive) [ 17/May/21 ]

Per Vladimir Shalaev DevOps will be doing most of the work; per Oleksii Kuzminov Once the DevOps work is done, then we will need to do some checking and maybe small tasks. Current design does not take PubSub into account. (Right now, quickMARC edits is the only PubSub piece that touches Data Import, and that is being changed to Kafka direct. Otherwise, PubSub is only being used by Circulation/Resource Access)

Comment by Ann-Marie Breaux (Inactive) [ 17/May/21 ]

Vladimir Shalaev If there is a feature/task for DevOps, please link that Jira to this one, so that we can track progress

Comment by Ann-Marie Breaux (Inactive) [ 18/May/21 ]

Blocking this whole feature for Data Import, until we have feedback from DevOps

Comment by Ann-Marie Breaux (Inactive) [ 19/May/21 ]

Related EBSCO DevOps story: https://rally1.rallydev.com/#/79944863724d/iterationstatus?detail=%2Fuserstory%2F605482544976

Generated at Fri Feb 09 00:28:00 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.