[FOLIO-2905] Set kafka host and port for mod-inventory-storage Created: 10/Dec/20  Updated: 17/Dec/20  Resolved: 16/Dec/20

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Bohdan Suprun (Inactive) Assignee: Ian Hardy
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks UXPROD-2696 Remote Storage Integration (Dematic t... Closed
Relates
relates to MODINVSTOR-620 Domain events sending in inventory-st... Closed
Sprint: DevOps: Sprint 104
Development Team: FOLIO DevOps

 Description   

Falcon team developing search engine and this requires mod-inventory-storage to send domain events to kafka message broker.

Because of this, all reference environment have to set KAFKA_HOST and KAFKA_PORT env variables in the mod-inventory-storage docker container.

Sample values for dev scratch envs (the scratch envs does not require this change, they are using global secrets config, and it is visible for all containers):
KAFKA_HOST: kafka
KAFKA_PORT: 9092



 Comments   
Comment by Marc Johnson [ 11/Dec/20 ]

Craig McNally Jakub Skoczen Zak Burke VBar

Is this the kind of design decision that we would like folks to document? Maybe because it is similar to how the database is configured at the moment and as I understand it, there is an architectural blueprint topic for addressing some of the limitations of this, e.g. the same Kafka instance has to be used for all tenants.

Comment by Marc Johnson [ 11/Dec/20 ]

Bohdan Suprun

the scratch envs does not require this change, they are using global secrets config, and it is visible for all containers

Is there a separate issue for changing the configuration of the scratch environment builds to specify these environment variables?

Comment by Marc Johnson [ 11/Dec/20 ]

Bohdan Suprun

all reference environment have to set KAFKA_HOST and KAFKA_PORT env variables in the mod-inventory-storage docker container

Given there are no parameters for credentials, does this mean that it has been decided that access to Kafka is unauthenticated?

Comment by Sergiy Vysotskiy [ 11/Dec/20 ]

Jakub Skoczen Could this task be delivered before 18th of December 2020, as it is blocks completing certain part of remote-storage feature of Firebird?
Thanks in advance.

https://folio-org.atlassian.net/browse/UXPROD-2696

Comment by Bohdan Suprun (Inactive) [ 11/Dec/20 ]

Hi Marc Johnson,

Is there a separate issue for changing the configuration of the scratch environment builds to specify these environment variables?

The scratch envs do not require any changes because these properties available in secrets store that shared for all the containers (the properties are used for pub/sub and I decided to reuse them).

Given there are no parameters for credentials, does this mean that it has been decided that access to Kafka is unauthenticated?

I'm not sure if credentials required, because kafka usually not exposed to the public and used internally. But maybe Mikhail Fokanov has better answer for this.

Comment by Marc Johnson [ 11/Dec/20 ]

Sergiy Vysotskiy

Could this task be delivered before 18th of December 2020, as it is blocks completing certain part of Data Import feature of Folijet?

Which data import work does this block?

Please could it be linked to this issue with the appropriate (either blocks or has to be done before) relation.

Comment by Marc Johnson [ 11/Dec/20 ]

Bohdan Suprun

The scratch envs do not require any changes because these properties available in secrets store that shared for all the containers (the properties are used for pub/sub and I decided to reuse them).

Ah, I did not know that. I'm a little surprised an actively private configuration for a single module became a shared secret, maybe that's just how K8s works

I'm not sure if credentials required, because kafka usually not exposed to the public and used internally.

I believe the security audit raised concerns with our security around PostgreSQL, which is also not public. I would personally consider Kakfka to be similar.

Comment by Sergiy Vysotskiy [ 11/Dec/20 ]

Marc Johnson Thank you for your question. Sorry I did a mistake and updated my comment and question accordingly.
it blocks https://folio-org.atlassian.net/browse/UXPROD-2696

Comment by Marc Johnson [ 11/Dec/20 ]

Sergiy Vysotskiy

Thank you for your question. Sorry I did a mistake and updated my comment and question accordingly.

That's cool, thank you for clarifying that.

The remote storage work is being done by Firebird, the change this work is associated with is owned by Falcon and assigned to a developer who I believe has not joined the team yet. Maybe someone from Firebird, or Mikhail Fokanov, could associate this issue with the work from the Firebird team that relies on this change.

Given that this change is expected to be done by the 18th December, that would suggest this work is already in progress. I'm not aware of any work in mod-inventory-storage that has reached the pull request stage that introduces these changes.

Comment by Mikhail Fokanov [ 11/Dec/20 ]

Bohdan Suprun I'm not sure if credentials required, because kafka usually not exposed to the public and used internally. But maybe Mikhail Fokanov has better answer for this.

Taras Spashchenko was working on Kafka Security. Taras Spashchenko Could you jump in?

Comment by Mikhail Fokanov [ 11/Dec/20 ]

Given that this change is expected to be done by the 18th December, that would suggest this work is already in progress. I'm not aware of any work in mod-inventory-storage that has reached the pull request stage that introduces these changes.

This is the PR (it have been just created): https://github.com/folio-org/mod-inventory-storage/pull/558

The remote storage work is being done by Firebird, the change this work is associated with is owned by Falcon

This change is needed for both Falcon and Firebird stories (as mod-remote-storage and mod-search will subscribe to the events).

Comment by Ian Hardy [ 14/Dec/20 ]

may make sense to use Okapi /_/env endpoint for the Kafka variables

Comment by Marc Johnson [ 15/Dec/20 ]

Ian Hardy

may make sense to use Okapi /_/env endpoint for the Kafka variables

This is probably a good idea if we intend these environment variables to become a platform wide (defacto) standard.

cc: VBar Taras Spashchenko Jakub Skoczen Craig McNally Zak Burke

Comment by Bohdan Suprun (Inactive) [ 15/Dec/20 ]

Hi Ian Hardy,

Does it mean that some additional actions have to be done in order to get the variables from OKAPI?

Comment by Ian Hardy [ 15/Dec/20 ]

Hi Bohdan Suprun no additional actions would be required. The single reference builds (folio-snapshot, etc) use okapi for container orchestration, so okapi will apply the settings in /_/env when it launches containers. This wouldn't be the case in most any production install, and operators would set variables another way. Marc Johnson I do suppose this could have the effect of making this a defacto standard. I believe this is what pubsub already uses so I don't think its a bad thing.

Comment by Bohdan Suprun (Inactive) [ 15/Dec/20 ]

Thank you Ian Hardy!

Comment by Marc Johnson [ 15/Dec/20 ]

Ian Hardy

I do suppose this could have the effect of making this a defacto standard. I believe this is what pubsub already uses so I don't think its a bad thing.

I think the cost of changing our mind increases significantly when this configuration becomes available to all (back end) modules.

Comment by Ian Hardy [ 15/Dec/20 ]

Hi Bohdan Suprun KAFKA_HOST and KAFKA_PORT should be available in tomorrows builds.

I added them to the okapi-deployment and okapi-tenant-deploy roles in folio-ansible which set environment variables for when Okapi is being used for orchestration (as it is for the reference builds).

Comment by Ian Hardy [ 17/Dec/20 ]

Hi again Bohdan Suprun I see you're mentioning this new environment variable in the README, thank you for that. When you get a chance, will you also add a default value to the launch descriptor section of the MD? Some operators will look here to see what environment variables need to be configured.

Generated at Thu Feb 08 23:24:07 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.