[FOLIO-3173] Support secure Kafka transport on ref envs Created: 25/May/21 Updated: 05/May/23 |
|
| Status: | Open |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Story | Priority: | P4 |
| Reporter: | Jakub Skoczen | Assignee: | Unassigned |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||
| Sprint: | DevOps Sprint 120, DevOps Sprint 121, DevOps Sprint 116, DevOps Sprint 122, DevOps Sprint 123, DevOps Sprint 124, DevOps Sprint 115, DevOps Sprint 117, DevOps Sprint 118, Kitfox: sprint 165, DevOps Sprint 119 | ||||||||
| Development Team: | FOLIO DevOps | ||||||||
| Description |
|
As described in https://folio-org.atlassian.net/wiki/display/~mage.air/Kafka+Security TLS transport will be the basis on which Kafka auth and tenant seperation is going to be built. While we don't need ref envs to provide secure Kafka (including ACLs) as these are ephemeral dev envs we will need to make sure that the new transport is supported. E.g either using pregenerated certs or by ignoring them. |
| Comments |
| Comment by Jakub Skoczen [ 22/Jun/21 ] |
|
Vladimir Shalaev Just to clarify – the TLS support will still optional right? E.g no module will require TLS and refuse to work if TLS transport was not enabled. Please clarify. |
| Comment by Vladimir Shalaev [ 22/Jun/21 ] |
|
Correct Modules can be configured to use or not to use TLS |
| Comment by Jakub Skoczen [ 08/Jul/21 ] |
|
John Malconian are there any updates on this? |
| Comment by John Malconian [ 15/Jul/21 ] |
|
In order to enable TLS-based Kafka authentication all FOLIO modules that consume or produce Kafka topics directly will require additional Kafka-client SSL configuration which will vary between Vert.x-based and Spring-based modules. IMO, this job should be broken into at least two parts: 1. Enable SSL and PLAINTEXT connectivity between Kafka server and clients. This will allow us to add the appropriate SSL connection options to each client module one at a time. 2. Once all client modules have been migrated to SSL and verified to work properly, we can enforce TLS-based authentication. |
| Comment by Vladimir Shalaev [ 15/Jul/21 ] |
|
The tasks to configure modules are already implemented |
| Comment by Marc Johnson [ 20/Jul/21 ] |
It seems that
Does that mean that all of the other modules that use Kafka also need to accept those same environment variables? If so, at least some of the affected modules (mod-inventory-storage, mod-inventory) have not implemented these changes yet and AFAIK no work has been prioritised to do so. (I thought that it was stated in the Tech Leads meeting that no changes were needed to individual modules to make this change). |
| Comment by Jakub Skoczen [ 10/Aug/21 ] |
|
John Malconian Drif Abdenour Kafka SSL env vars are already specified in the MD for https://github.com/folio-org/mod-remote-storage/blob/master/descriptors/ModuleDescriptor-template.json. Next steps:
|
| Comment by Marc Johnson [ 10/Aug/21 ] |
How can this be done if the module does not support the environment variables needed for this (as I believe is the case for mod-inventory-storage)? |
| Comment by Vladimir Shalaev [ 16/Aug/21 ] |
|
Issues linked to https://folio-org.atlassian.net/browse/UXPROD-2929 are ment to add security configuration support to all modules with kafka usage
|
| Comment by Drif Abdenour [ 17/Aug/21 ] |
|
This is the list of the modules that need the SSL configuration |
| Comment by Kateryna Senchenko [ 18/Aug/21 ] |
|
Hi Drif Abdenour, mod-source-record-storage, mod-source-record-manager, mod-inventory and mod-data-import DO support env variables - it was added in folio-kafka-wrapper lib that is used in the specified modules, so every Kafka security prop that was required in
|
| Comment by Drif Abdenour [ 18/Aug/21 ] |
|
Hi Kateryna Senchenko , Thank you for clarifying this , i already edited my comment . |
| Comment by Marc Johnson [ 18/Aug/21 ] |
|
Jakub Skoczen Craig McNally Jeremy Huff Zak Burke
This document is in a personal workspace and marked as draft. Should this be considered the official policy for Kafka security in FOLIO? And if so, should it be moved to the section of the decision log that includes Kafka documentation? |
| Comment by Jakub Skoczen [ 24/Aug/21 ] |
|
Marc Johnson I think it's a good idea to move it out from the personal workspace. |
| Comment by Jakub Skoczen [ 24/Aug/21 ] |
|
Drif Abdenour are you able to pick this ticket up and do the following:
|
| Comment by Vladimir Shalaev [ 09/Sep/21 ] |
|
I was planning to move it to public space as soon as we finish testing anfd create documentation for setting up different types of environments. |
| Comment by Marc Johnson [ 09/Sep/21 ] |
|
Thanks for responding to my question.
Is the document intended to be implementation guidance for folks or an official policy on recommended Kafka Security (or both)? |
| Comment by Vladimir Shalaev [ 09/Sep/21 ] |
|
I expect it to be a guidance for configuring secure Kafka connections when needed. And particular hoster/installation can decide if they need it or not.
In general that's not an invention - this is simply usage of existing security mechanisms provided by Kafka and kafka bundled library. |
| Comment by Jakub Skoczen [ 09/Sep/21 ] |
|
@malc there is a question from Aliaksei Luhavy regarding this ticket on where to store certs generated for the modules? |