[FOLIO-3173] Support secure Kafka transport on ref envs Created: 25/May/21  Updated: 05/May/23

Status: Open
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Story Priority: P4
Reporter: Jakub Skoczen Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks MSEARCH-105 Use TLS for Kafka connection Closed
Sprint: DevOps Sprint 120, DevOps Sprint 121, DevOps Sprint 116, DevOps Sprint 122, DevOps Sprint 123, DevOps Sprint 124, DevOps Sprint 115, DevOps Sprint 117, DevOps Sprint 118, Kitfox: sprint 165, DevOps Sprint 119
Development Team: FOLIO DevOps

 Description   

As described in https://folio-org.atlassian.net/wiki/display/~mage.air/Kafka+Security TLS transport will be the basis on which Kafka auth and tenant seperation is going to be built.

While we don't need ref envs to provide secure Kafka (including ACLs) as these are ephemeral dev envs we will need to make sure that the new transport is supported. E.g either using pregenerated certs or by ignoring them.



 Comments   
Comment by Jakub Skoczen [ 22/Jun/21 ]

Vladimir Shalaev Just to clarify – the TLS support will still optional right? E.g no module will require TLS and refuse to work if TLS transport was not enabled. Please clarify.

Comment by Vladimir Shalaev [ 22/Jun/21 ]

Correct

Modules can be configured to use or not to use TLS

Comment by Jakub Skoczen [ 08/Jul/21 ]

John Malconian are there any updates on this?

Comment by John Malconian [ 15/Jul/21 ]

In order to enable TLS-based Kafka authentication all FOLIO modules that consume or produce Kafka topics directly will require additional Kafka-client SSL configuration which will vary between Vert.x-based and Spring-based modules. IMO, this job should be broken into at least two parts:

1. Enable SSL and PLAINTEXT connectivity between Kafka server and clients. This will allow us to add the appropriate SSL connection options to each client module one at a time.

2. Once all client modules have been migrated to SSL and verified to work properly, we can enforce TLS-based authentication.

Comment by Vladimir Shalaev [ 15/Jul/21 ]

The tasks to configure modules are already implemented

Comment by Marc Johnson [ 20/Jul/21 ]

Vladimir Shalaev

The tasks to configure modules are already implemented

It seems that MSEARCH-105 Closed were required for mod-search to support configuration for this.

Does that mean that all of the other modules that use Kafka also need to accept those same environment variables?

If so, at least some of the affected modules (mod-inventory-storage, mod-inventory) have not implemented these changes yet and AFAIK no work has been prioritised to do so.

(I thought that it was stated in the Tech Leads meeting that no changes were needed to individual modules to make this change).

Comment by Jakub Skoczen [ 10/Aug/21 ]

John Malconian Drif Abdenour Kafka SSL env vars are already specified in the MD for https://github.com/folio-org/mod-remote-storage/blob/master/descriptors/ModuleDescriptor-template.json.

Next steps:

  • configure Kafka to work in both PLAINTEXT and SSL modes John Malconian will look into this next week
  • enable SSL connectivity in FOLIO modules: mod-remote-storage, mod-inventory-storage Drif Abdenour will provide a full list of modules
  • enable SSL authentication in Kafka (disables PLAINTEXT connections)
Comment by Marc Johnson [ 10/Aug/21 ]

Jakub Skoczen

enable SSL connectivity in FOLIO modules: mod-remote-storage, mod-inventory-storage

How can this be done if the module does not support the environment variables needed for this (as I believe is the case for mod-inventory-storage)?

Comment by Vladimir Shalaev [ 16/Aug/21 ]

Issues linked to https://folio-org.atlassian.net/browse/UXPROD-2929 are ment to add security configuration support to all modules with kafka usage

 

Comment by Drif Abdenour [ 17/Aug/21 ]

This is the list of the modules that need the SSL configuration
mod-search : ready to accept env variables, it contains the new variables in module descriptor
mod-remote-storage: ready to accept env variables, it contains the new variables in module descriptor
mod-pubsub: ready to accept env variables, it DOES NOT contain the new variables in module descriptor
mod-source-record-storage : ready to accept env variables, it DOES NOT contain the new variables in module descriptor
mod-source-record-manager: ready to accept env variables, it DOES NOT contain the new variables in module descriptor
mod-inventory: ready to accept env variables, it DOES NOT contain the new variables in module descriptor
mod-inventory-storage : does not support the environment variables
mod-data-import : ready to accept env variables, it DOES NOT contain the new variables in module descriptor

Comment by Kateryna Senchenko [ 18/Aug/21 ]

Hi Drif Abdenour,

mod-source-record-storage, mod-source-record-manager, mod-inventory and mod-data-import DO support env variables - it was added in folio-kafka-wrapper lib that is used in the specified modules, so every Kafka security prop that was required in  MODPUBSUB-171 Closed https://folio-org.atlassian.net/wiki/pages/viewpage.action?spaceKey=~mage.air&title=Kafka+Security can be passed for those modules.

Comment by Drif Abdenour [ 18/Aug/21 ]

Hi Kateryna Senchenko , Thank you for clarifying this , i already edited my comment . 

Comment by Marc Johnson [ 18/Aug/21 ]

Jakub Skoczen Craig McNally Jeremy Huff Zak Burke

As described in https://folio-org.atlassian.net/wiki/display/~mage.air/Kafka+Security TLS transport will be the basis on which Kafka auth and tenant seperation is going to be built.

This document is in a personal workspace and marked as draft. Should this be considered the official policy for Kafka security in FOLIO? And if so, should it be moved to the section of the decision log that includes Kafka documentation?

Comment by Jakub Skoczen [ 24/Aug/21 ]

Marc Johnson I think it's a good idea to move it out from the personal workspace.

Comment by Jakub Skoczen [ 24/Aug/21 ]

Drif Abdenour are you able to pick this ticket up and do the following:

  • reconfigure Kafka provisioned through folio-ansible on ref envs to support SSL
  • generate keys using the appropriate tools and place them in the keystores
  • configure modules that support the SSL env vars with Kafka SSL config
Comment by Vladimir Shalaev [ 09/Sep/21 ]

Jakub Skoczen , Marc Johnson

I was planning to move it to public space as soon as we finish testing anfd create documentation for setting up different types of environments.

Comment by Marc Johnson [ 09/Sep/21 ]

Vladimir Shalaev

Thanks for responding to my question.

I was planning to move it to public space as soon as we finish testing anfd create documentation for setting up different types of environments.

Is the document intended to be implementation guidance for folks or an official policy on recommended Kafka Security (or both)?

Comment by Vladimir Shalaev [ 09/Sep/21 ]

I expect it to be a guidance for configuring secure Kafka connections when needed. And particular hoster/installation can decide if they need it or not.

 

In general that's not an invention - this is simply usage of existing security mechanisms provided by Kafka and kafka bundled library.

Comment by Jakub Skoczen [ 09/Sep/21 ]

@malc there is a question from Aliaksei Luhavy regarding this ticket on where to store certs generated for the modules?

Generated at Thu Feb 08 23:26:09 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.