2025-06-11 Environmental variable naming

1 min

Scribe

All

@Jenn Colt is next, followed by @Marc Johnson

Reminder:  Please copy/paste the Zoom chat into the notes.  If you miss it, this is saved along with the meeting recording, but having it here has benefits.

*

Environmental variables and Folio multitenant upgrade problem

Kevin

Environment Variables are inconsistently used, not using established standards, default value concerns, ambiguously named, and also a multitenant upgrade problem.

1. Env variables with dots: e.g. okapi.url, tenant.url, kong.url, am.url.

   1. This is inconsistent with the primary wording of major standards and can cause errors in some systems.

   2. Rancher supports dots in the names.

      1. Bash’s env command does not appear to expose these when executed within the pod.

   3. For mod-fqm-manager, there are lowercase variables with dots.

   4. From the standard:

      1. “It is unwise to conflict with certain variables that are frequently exported by widely used command interpreters and applications:
         … ENV …”

2. Variables are inconsistent.

   1. Some have different names in different modules, but express the same quantity.

   2. Inconsistent naming (KC_, KEYCLOAK_, ...).

3. The values provided may not accurately reflect the resources required.

   1. Some problems with increased memory (from Q → R) and JAVA_OPTIONS parameters.

      1. Many modules have to be adapted for more memory.

      2. Many out-of-memory events and heap-space events.

4. Problems with the ENV variable, could be less ambiguously named.

   1. Kafka topics use that and also so does the Elasticsearch index.

   2. It is not good to use the ENV variable for two purposes.

   3. Upgrades get broken if you don't change the value of ENV between releases on a muti-tenant system.

   4. Directly relates to Folio multitenant upgrade problem.

5. See:

   1. Open Group Base Specifications Issue 6, IEEE Std 1003.1, 2004 Edition, Environment Variables 

      1. Latest edition: https://pubs.opengroup.org/onlinepubs/9699919799/

   2. Spring Boot doc: Externalized Configuration Binding From Environment Variables

   3. Sys Ops SIG met on this 2025-05-07 Sys Ops & Management SIG Agenda and Meeting Notes.

Notes:

• Some of the ones with dots are passed in on the command line. For okapi.url there are some modules that take that as a java command line system flag

• Document distinction between java variables and env variables?

• Container environment - environment variables can get passed in or coded either way so properly formatted environment. variables get translated into the underlying. code base, or goes through yaml config file

• proposal here is for container run times to use properly formatted environmental variables?

• suggestion to follow spring naming strategy, either with dot and lower. or underscore and upper

• spring can handle both so easily can be fixed

• write up? no not needed

• Java memory options - defaults seem outdated and too small

  • Raise as ticket to the teams, no action needed by TC

  • File these as they are found

• Multi-tenant upgrade

  • ID has seen this problem, issues with Kafka between releases. Topics overwhelmed or duplicated, etc.

  • ENV is usually used to distinguish which FOLIO env to use so you can define multiple FOLIO environments, kafka envrionments

  • Generally not recommended to run MT with tenants at different versions of FOLIO, but not always possible so end up with multiple instances of kafka, etc

  • GBV runs one Kafka with multiple tenants, migrate 2 tenants per day, so they put flower name into the env variables and this prevents breaking changes in the kafka module. There was an RFC on this.

    • https://github.com/folio-org/rfcs/blob/master/text/0002-kafka-tenant-collection-topics.md

  • Current suggestion is to split into two different environmental variables

    • Doesn’t seem like a good idea because it contradicts current operational standard

    • Issue of running multiple backends is well understood, but maybe it needs more awareness

    • Seems like a process problem not a technical problem, more documentation/communication

    • Keep ENV variable as it is, splitting requirements a more in-depth investigation to see if it is really better

    • Sysops shouldn’t have to discover and document, it should be discovered before that by teams. But realistically the first who runs into the problem writes the ticket. The teams should document updates to descriptors? Make part of definition of done?

    • Bugfest is where these problems come up, then the fix needs to get documented

• Much of this seems like documentation vs technical decisions

-

Zoom Chat