Spike Overview

EUREKA-846: Investigate tenant not enabled errors during entitlement (mod-lists/mod-fqm)ClosedPreview

1. Purpose and Scope

This document describes how the Eureka-based FOLIO platform supports running multiple versions of the same application in a single Kubernetes cluster, where tenants can:

• Share some modules at the same version (e.g. mod-a-2.0.0), and

• Use different versions of other modules (e.g. mod-b-1.0.0 vs mod-b-1.0.1) per tenant.

2. Business Requirement

As a multi-tenant hoster, we must:

• Migrate only selected tenants to a new critical patch (CSP-2) while others stay on a stable release (CSP-1).

• Run multiple releases of the same application in a single cluster.

• Allow tenants on different releases to share modules where versions are identical, and diverge where versions differ.

Example requirement:

• Tenant foo (CSP-1) runs app-xyz-1.0.0

• Tenant bar (CSP-2) runs app-xyz-1.0.1

• Both tenants share mod-a-2.0.0

• foo uses mod-b-1.0.0, bar uses mod-b-1.0.1

This must be achieved without cross-tenant leakage

3. Data Model and Entitlement Isolation

The current architecture already supports this pattern at the data and entitlement layer.

3.1 Application Entitlements

Table: entitlement
Primary key: (tenant_id, application_id)

This allows each tenant to be entitled to a specific application version:

Effect:

• mgr-tenant-entitlements treats entitlement for foo/app-xyz-1.0.0 and bar/app-xyz-1.0.1 as independent.

3.2 Module Entitlements

Table: entitlement_module
Composite key: (module_id, tenant_id, application_id)

This supports both shared and version-divergent modules.

Shared module (mod-a-2.0.0):

• Logically “shared”, but represented as two entitlement records, so actions on one tenant do not affect the other.

Version-specific module (mod-b):

Effects:

• Tenant isolation: Every entitlement and module-entitlement record includes tenant_id, so tenants are isolated.

• Version isolation: Different module IDs (mod-b-1.0.0 vs mod-b-1.0.1) prevent collisions.

• Shared modules: “Sharing” means multiple tenants have entitlements for the same module_id and version.

Conclusion: From a persistence and entitlement perspective, this scenario is supported .

4. Discovery, Sidecars, Kafka, and Secrets

4.1 Service Discovery and Module IDs

Eureka rely on module IDs including the version (e.g. mod-b-1.0.0) as a convention for uniquely identifying module instances.​

• Different module IDs → distinct discovery entries and URLs.

• Example discovery registry entries:

  • mod-b-1.0.0 → http://mod-b-ramsons:8081

  • mod-b-1.0.1 → http://mod-b-sunflower:8081

When mgr-tenant-entitlements resolves “where is mod-b-1.0.1 for tenant bar?”, it gets the explicit Sunflower URL; no guessing based on environment naming is required.​

4.2 What Is Sent to Sidecars

Sidecars are configured via application.properties and environment variables, and they receive:

• Kafka configuration (topics, bootstrap)

• Secure store configuration (backend type, connection)

• Environment name (ENV) and optional patterns for tenant collections / topic regexes

Discovery information per sidecar is not pushed inline in the entitlement payload; instead, modules and sidecars rely on:

• Kafka events that carry tenant and module identifiers.

• Ingress / Kong / Egress for HTTP routing and service discovery.

5. Kafka and Secure Store Behaviour in This Pattern

5.1 Kafka Topics (Environment-Level, Not Tenant-Level)

Sidecars and manager services use environment-level topics, not per-tenant topics.

Relevant parameters (example):

• mp.messaging.incoming.discovery.topic → ${ENV:folio}.discovery

• mp.messaging.incoming.entitlement.topic → ${ENV:folio}.entitlement

• mp.messaging.incoming.logout.topic → regex-based

Behaviour:

• Events for both foo and bar are published to the same environment topic (e.g. folio.entitlement).​

• Each sidecar filters messages at the module and tenant level:

  • mod-a-2.0.0 sidecar:

    • Processes entitlement events for (tenant=foo, module=mod-a-2.0.0) and (tenant=bar, module=mod-a-2.0.0).

  • mod-b-1.0.0 sidecar:

    • Listens to the same topic but only processes events for module_id=mod-b-1.0.0.

  • mod-b-1.0.1 sidecar:

    • Processes only module_id=mod-b-1.0.1.

This pattern allows shared topics, with logical isolation via filtering.​

5.2 Secure Store (Shared Backend, Namespaced Keys)

Sidecars are configured to a single secure store backend (AWS SSM, Vault, filesystem, etc.) and use naming conventions for isolation.

Key parameters:

• secret-store.type

• secret-store.aws-ssm.* / secret-store.vault.* / secret-store.fssp.*

Key naming convention:

• ${ENV}_${TENANT}_${KEY}, e.g.:

  • folio_foo_mod-a-secret

  • folio_bar_mod-a-secret

  • folio_foo_mod-b-secret

  • folio_bar_mod-b-secret

Implications:

• Isolation is achieved via keys and prefixes, not by switching secret-store configuration per tenant.

• All required secrets for both CSP-1 and CSP-2 must be accessible in the configured store.

6. The ENV Mismatch Problem Observed

6.1 Configuration Used

mgr-applications:

Release-agnostic and correct: we want mgr-applications to be independent of specific CSP environments.

Modules (e.g. mod-lists, mod-fqm):

6.2 Symptoms

• Entitlement of platform-minimal, platform-complete, and app-acquisitions succeeded.

• Entitlement of app-edge-complete, app-dcb, and app-fqm failed with:

  • Flow CANCELLED due to:

    • HttpTimeoutException calling http://mod-lists-3-1-5.eureka-sunflower:8082/_/tenant

  • mod-lists logged:

    • TenantNotEnabledException: Application is not enabled for tenant: diku when calling http://fqm/version.

6.3 Likely Root Cause

With ENV=sunflower for modules and ENV=folio for mgr-applications:

• Discovery / routing for some modules happened under one logical environment, while entitlements and Kafka events were scoped under another.

• Sidecars and modules attempted to call services (mod-fqm) that were not enabled (or not entitled) in the effective environment/tenant combination.

• When ENV was set to folio for modules too, everything aligned (mgr-apps, Kafka topics, and module discovery) and entitlements succeeded.

Conclusion:

• The failure is not in the entitlement data model, but in ENV-based environment separation conflicting with discovery and routing expectations.

7. Target Architecture: Multi-Version in One Cluster

The architecture we want:

• Single cluster.

• One logical Eureka environment (ENV=folio) for this cluster.

• Multiple application versions and module versions coexisting:

  • app-xyz-1.0.0 + mod-b-1.0.0 (CSP-1)

  • app-xyz-1.0.1 + mod-b-1.0.1 (CSP-2)

• Shared modules (e.g. mod-a-2.0.0) reused across tenants.

7.1 Design Principles

1. Tenant isolation via data model (entitlement and entitlement_module).

2. Version isolation via module ID (module name includes version, as per Okapi convention).​

3. Single environment namespace per cluster for discovery and Kafka (ENV=folio).

4. Sidecars filter by tenant and module ID rather than relying on ENV differences.

7.2 Proposed Pattern

1. Keep ENV=folio for:

   • mgr-applications

   • mgr-tenant-entitlements

   • All module deployments and their sidecars

2. Represent version differences only in module IDs and services:

   • mod-b-1.0.0 vs mod-b-1.0.1

3. Let mgr-tenant-entitlements and mgr-applications:

   • Entitle each tenant to the correct application and module versions.

   • Resolve discovery URLs per module_id, not per ENV.

4. Kafka:

   • Use shared environment topics (folio.entitlement, folio.discovery).

   • Sidecars filter by tenant + module ID (already supported).

5. Secure Store:

   • Keep SECURE_STORE_ENV=folio everywhere.

   • Maintain secrets per tenant and module version via naming convention.

This pattern matches the in-place migration example:

By ensuring that Sunflower carries a new version (even patch, e.g. 1.0.1), the platform’s version-based routing and discovery can distinguish the two paths cleanly.

8. Recommended Configuration

8.1 Environment Variables

mgr-applications:

mgr-tenant-entitlements:

All modules (including different versions):

8.2 Versioned Module IDs and Discovery

• Ensure module IDs follow the name-version convention, e.g. mod-b-1.0.0 and mod-b-1.0.1.​

• Register each module version separately in discovery:

  • mod-b-1.0.0 → http://mod-b-1-0-0:8081

  • mod-b-1.0.1 → http://mod-b-1-0-1:8081

• Make sure mgr-tenant-entitlements and mgr-applications always refer to the full module ID.

9 Sidecar Logic for Multi-Version Support

The existing sidecar implementation already provides the necessary guards:

9.1 Version-Specific Entitlement Loading

• On startup, a sidecar loads entitlements only for its own module.id:

  • TenantService#loadTenantsAndEntitlements() calls the tenant entitlement manager with the sidecar’s module ID.

  • Only tenants entitled to that exact module ID are enabled.

• module.id is defined as ${module.name}-${module.version} in application.properties.

Result:

• mod-b-1.0.0 sidecar enables only tenants entitled to mod-b-1.0.0 (e.g. foo).

• mod-b-1.0.1 sidecar enables only tenants entitled to mod-b-1.0.1 (e.g. bar).

9.2 Runtime Event Filtering

• The entitlement consumer (e.g. TenantEntitlementConsumer#consume()) processes Kafka events and checks:

  • tenantService.isAssignedModule(moduleId)

• If the event’s moduleId does not equal the sidecar’s own module.id, the event is ignored; tenant state is not updated.

Result:

• Even though all entitlement changes go to the same folio.entitlement topic, each sidecar sees only its own module-version events.

9.3 Traffic Enforcement

• The TenantFilter intercepts all incoming HTTP requests:

  • Reads the x-okapi-tenant header.

  • Checks whether that tenant is in the locally enabled tenant set for this sidecar.

• If a request for tenant bar hits the mod-b-1.0.0 sidecar:

  • Tenant bar is not enabled in that sidecar.

  • The filter rejects the request with 400 Bad Request and UNKNOWN_TENANT.

Result:

• Even if routing misconfigurations send tenant bar traffic to the wrong module version, the sidecar blocks the request at the edge.

10. Summary

• The current Eureka/FOLIO architecture already supports multi-version applications per tenant in one cluster through its entitlement schema and versioned module IDs.

• The failures observed are due to ENV-based separation causing discovery and routing mismatches, not due to limitations in entitlements.

• The recommended solution is to:

  • Use a single ENV per cluster (folio),

  • Distinguish module versions by module ID, and Let entitlements and sidecar filtering drive which tenant uses which module version.

  • The Flag KONG_TENANT_CHECKS_ENABLEDshould be true for mgr-application to support multiple versions of the same module. EUREKA-789: POC - Version-specific routes in KongClosedPreview

Kafka topic routing with multiple module versions

From recent discussions and comments, we identified that the main concern for hosters is how to route messages correctly to the appropriate Kafka topic when multiple versions of the same module are running in parallel. In particular, the same logical topic may be processed by different module versions, which can lead to incompatibilities if not properly isolated.

How this is handled with Okapi

When you need to run different versions of a module that use Kafka topics, you should follow the approach described in the RFC “Kafka tenant collection topics”, section Migration to New Module Version:
https://github.com/folio-org/rfcs/blob/master/text/0002-kafka-tenant-collection-topics.md

Key rule:

• Incompatible module versions must be separated at the Kafka topic level.

Common pattern:

• Use the ENV environment variable to populate the [environment] part of the Kafka topic name, for example:

  • ENV=nolana for one FOLIO release.

  • ENV=orchid for another release.

Migration example:

• A new version of a module is introduced and rolled out gradually.

• The new producer and consumer use a new environment value (for example, sunflower).