Mixed-Version FOLIO Deployment

1. Goal

This document explains how a single FOLIO cluster can run multiple application versions, such as Sunflower and Trillium, at the same time.

In this model, different tenants can be entitled to different application versions.

2. Overall Deployment Model

In a mixed-version deployment, the cluster runs one shared set of mgr-* components.

The shared manager layer is used by both Sunflower modules and Trillium modules.

Sunflower modules remain on the Sunflower application version. Trillium modules are deployed as a separate module set. Sidecars for both Sunflower and Trillium modules use the Trillium sidecar version.

Open

3. Setting up a Sunflower + Trillium environment

4. Important Environment Rules

4.1. Deploying multiple flower releases in same cluster (Currently supported)

• All manager components must use the same ENV value

• All sidecars must use the same ENV value as the mgr-* components

• Sunflower module containers - ENV may be the same as the shared mgr-* ENV, or it may be different

• Trillium module containers - Must use a different ENV value from the Sunflower module containers
  
  

  Example only:

  mgr-* ENV=folio all sidecars ENV=folio Sunflower module ENV=folio-sunflower Trillium module ENV=folio-trillium

4.2. FUTURE - Deploying multiple CSPs of a flower release in the same cluster (Not supported currently)

• All components (mgr-* components, module containers & sidecars) use the same ENV value

• All components use the same SECURE_STORE_ENV value.

Example only:

5. API Calls: Kong to Module Routing

Client, Stripes, and external API calls enter the cluster through Kong.

Kong routes the request to the correct module version by matching route expressions. In a mixed-version deployment, route expressions must include tenant-specific checks so the same API path can be routed to different module versions for different tenants.

Open

5.1. Tenant-Specific Kong Routes

Set the following on mgr-applications:

When this is enabled, mgr-applications consumes entitlement events from mgr-tenant-entitlements and updates Kong route expressions.

For ENTITLE and UPGRADE events, the tenant is added to the route expression for the entitled module.

For REVOKE events, the tenant is removed from the route expression.

5.2. Example Kong Route Expressions

A generic tenant-agnostic route may look like this:

After tenant1 is entitled to the module, the route is updated in place:

After tenant2 is also entitled to the same module version, the route becomes:

5.3. Existing Tenants

Enabling APIGW_TENANT_CHECKS_ENABLED=true does not automatically scan existing tenant entitlements.

Existing tenant entitlements must be processed after the flag is enabled so Kong routes are updated with tenant-specific route expressions.

6. API Calls: Module to Module Routing Through Sidecars

Module-to-module calls do not go directly from one module container to another module container. They go through sidecars.

The source module sends the request through its sidecar. The source sidecar resolves the destination module route and forwards the request to the correct target sidecar/module.

In a mixed-version deployment, the same destination interface may be provided by both a Sunflower module and a Trillium module. The sidecar must resolve the destination based on the tenant's entitled application version.

Open

6.1. Tenant-Scoped Sidecar Routing

Set the following on all sidecars:

When this setting is enabled, sidecars resolve egress routes per tenant.

The sidecar uses the request tenant, from X-Okapi-Tenant, to select the routing table built from that tenant's entitled applications.

Without tenant-scoped egress routing, a sidecar will use a non-tenant-specific egress route table. That is not sufficient when multiple versions of the same provider module exist in the same cluster.

6.2. How the Sidecar Builds Tenant Egress Routes

When tenant-scoped routing is enabled, the sidecar:

1. Tracks tenant entitlement state.

2. Loads the tenant's entitled applications from mgr-tenant-entitlements.

3. Requests egress bootstrap information from mgr-applications for those application IDs.

4. Builds an egress route cache for that tenant.

5. Uses that tenant-specific route cache for module-to-module calls.

Discovery and entitlement events keep the sidecar route state up to date.

In tenant-scoped mode, unresolved egress requests can fall through to the gateway when gateway forwarding is enabled (SIDECAR_FORWARD_UNKNOWN_REQUESTS=true)

7. Kafka Message Routing - Only supports multiple flower releases in

Kafka topic routing in a mixed-version deployment has two different cases:

1. Regular module-to-module Kafka messages produced by modules.

2. Shared manager events produced by the shared mgr-* components.

The behavior is different for each case.

7.1. Regular Module-to-Module Kafka Messages

Regular module-produced Kafka messages are naturally separated by module ENV.

All Sunflower modules should use the same Sunflower module ENV value. All Trillium modules should use a different Trillium module ENV value.

When a Sunflower module emits an event, the event is sent to a Kafka topic prefixed with the Sunflower module ENV.

When a Trillium module emits an event, the event is sent to a Kafka topic prefixed with the Trillium module ENV.

For example:

Because the topic prefixes are different, Sunflower module events are not sent to Trillium module topics, and Trillium module events are not sent to Sunflower module topics.

Open

This provides version-level separation for regular module events as long as Sunflower and Trillium modules use different ENV values.

7.2. Events Emitted by Shared mgr-* Components

Shared manager events are different.

The cluster runs only one shared set of mgr-* components. Those components use the shared manager ENV.

The shared mgr-* components publish manager-owned events using the shared manager ENV. Modules that consume those events must subscribe to the shared manager topics, even if the module's own ENV is different.

Open

7.2.1. Why Topic Pattern Overrides Are Needed

By default, a module looks for Kafka topics prefixed with its own ENV value.

If a Trillium module has ENV=folio-trillium

but the shared managers have: ENV=folio

then manager events are published to topics prefixed with folio, not folio-trillium.

Without overrides, the Trillium module may attempt to consume from a topic that does not exist.

For manager-owned events, both Sunflower and Trillium module versions should consume from the same Kafka topics emitted by the shared mgr-* components.

7.2.2. Required Topic Pattern Overrides

Apply these overrides to any module version whose ENV is different from the shared mgr-* and sidecar ENV.

Because Sunflower and Trillium modules use different ENV values, they belong to different Kafka consumer groups. As a result, both the Sunflower containers and the Trillium containers receive the manager events published by the shared mgr-* components.

Example only:

7.3. Kafka Tenant Filtering

Set the following in both Sunflower and Trillium versions of the relevant modules:

Apply to:

• mod-roles-keycloak

• mod-users-keycloak

• mod-scheduler

Kafka tenant filtering ensures a module processes only messages for tenants where that module or application version is enabled.

Without tenant filtering, both Sunflower and Trillium versions of a module could consume the same shared manager event and process messages for tenants that are not entitled to that version.

8. Notes and Caveats

Module-specific environment variables are not covered in this document.

Different versions of the same module may require different environment variables. For example, the Sunflower version of mod-data-import requires AWS_URL, AWS_REGION, and AWS_BUCKET, while the Trillium version require S3_URL, S3_REGION, and S3_BUCKET.

Refer to the module README file or module descriptor for each module version to identify version-specific environment variables.

9. Challenge - Running multiple CSPs of the Same Flower Release in One Cluster

This document mainly describes running different flower releases, such as Sunflower and Trillium, in the same cluster.

There is an additional challenge when running multiple CSPs of the same flower release in one cluster.

In a flower release, the versions of all modules are usually bumped. Between CSPs, however, only some module versions may change

Example Scenario:

• Sunflower CSP1 application set contains mod-foo-1.0 and mod-bar-1.0.

• mod-foo sends kafka messages to mod-bar.

Open

Sunflower CSP2 application set contains: mod-foo-1.0, mod-bar-2.0

In this example, mod-foo did not change between CSP1 and CSP2, but mod-bar changed.

The question is: how should mixed-version deployment work when the producer module version is shared, but the consumer module version changed?

Option 1: One Instance of mod-foo, Two Instances of mod-bar

In this model, the cluster runs:

• one mod-foo-1.0 runtime, with ENV=sf

• one mod-bar-1.0 runtime, with ENV=sf

• one mod-bar-2.0 runtime, with ENV=sf-csp2

Open

Problem:

mod-foo-1.0 runs with a single ENV value. Therefore, it publishes Kafka messages only to one ENV-prefixed topic namespace.

As a result, only mod-bar V1.0 consuming from that topic namespace receives the message. The other mod-bar version (mod-bar V20) will not receive any message.

Conclusion: This option will not work.

Option 2: Two Instances of mod-foo, Two Instances of mod-bar

n this model, the cluster runs:

• one mod-foo-1.0 runtime for the CSP1 application set

• one mod-foo-1.0 runtime for the CSP2 application set

• one mod-bar-1.0 runtime

• one mod-bar-2.0 runtime

Open

This allows each mod-foo-1.0 runtime to use the correct CSP-specific module ENV and publish Kafka messages to the matching topic namespace.

Problem:

If a tenant is entitled to mod-foo-1.0 and mod-bar-2.0, how does Kong gateway know to route that tenant's external API requests to the CSP2-specific runtime of mod-foo-1.0?

This is not straightforward because both mod-foo runtimes have the same module ID.

Conclusion: This option will not work.

Option 3: One Instances of mod-foo, Two Instances of mod-bar, Same ENV for mod-bar-2.0

In this model, the cluster runs:

• one mod-foo-1.0 runtime, with ENV=sf

• one mod-bar-1.0 runtime, with ENV=sf

• one mod-bar-2.0 runtime, with ENV=sf

Open

Problem:

Both mod-bar runtimes consume from the same Kafka topic and is also be part of the same Kafka consumer group.

For example, assume a tenant T2 is entitled to mod-bar-2.0. A Kafka message for that tenant may still be consumed by the mod-bar-1.0 runtime instead of the mod-bar-2.0 runtime.

This is a problem if the message processing logic changed between mod-bar-1.0 and mod-bar-2.0.

If the message processing logic did not change between the two versions, then this may not cause a functional problem.

Conclusion:
This option may work in many cases as message processing logic may not change between CSPs. But this is not a foolproof solution.