EUREKA-899 - Application-Scoped Sidecar Bootstrap & Tenant-Scoped Egress Routing

Status: POC — In Code Review
Ticket: EUREKA-899 – POC: Application-scoped sidecars bootstrap
Repositories / branches: mgr-applications @ EUREKA-899_v2 · folio-module-sidecar @ EUREKA-899_v2
Default behavior: OFF. The new path is gated by a single sidecar flag and is fully backward-compatible when disabled.

1. TL;DR

In a multi-version FOLIO environment, different versions of the same application can be entitled to different tenants at the same time. The module sidecar previously resolved egress routes (module → its required modules) globally by module id, so an egress call could be routed to the wrong provider version for the calling tenant.

This change makes egress resolution tenant-aware:

mgr-applications gains a bootstrap API that can be scoped to a set of application ids, so it returns only the providers belonging to those applications.
folio-module-sidecar fetches the application ids entitled to each tenant, asks mgr-applications for an application-scoped egress bootstrap, and stores one egress routing table per tenant. At request time it resolves the route table by X-Okapi-Tenant.

The feature is behind a new flag, routing.tenant-scoped.enabled (default false). When off, behavior is identical to today.

2. Background & problem

As part of running FOLIO in a multi-version environment, multiple versions of the same application can be registered in mgr-applications and entitled to different tenants simultaneously, e.g.:

Application version	Entitled to tenant	Module versions

Application version	Entitled to tenant	Module versions
`app-platform-minimal-2.0.53`	Tenant A	`mod-users-19.5.4`, `mod-users-keycloak-3.0.13`
`app-platform-minimal-2.1.8`	Tenant B	`mod-users-19.6.0`, `mod-users-keycloak-4.0.2`

Where it broke:

folio-module-sidecar — At both bootstrap entry points (startup and Kafka entitlement events), the sidecar called mgr-applications by module id only. It had no awareness of which application version was entitled to a given tenant, so it could not build correct per-tenant route tables. Egress to required/optional modules could resolve to the wrong version.
mgr-applications — Discovery, application, and module data are stored globally with no tenant context, so the bootstrap endpoint could not return tenant-scoped routing information.

3. Goals

Make sidecar egress routing resolve to the provider version that belongs to the calling tenant.
Let mgr-applications return a bootstrap restricted to a supplied application scope.
Keep the change opt-in and backward-compatible (flag-gated, off by default).
Refresh a tenant's routing on the relevant entitlement lifecycle events.

4. Solution overview

---
config:
  layout: elk
---
flowchart LR
    subgraph TE[mgr-tenant-entitlements]
      ENT["GET /entitlements?tenant={t}"]
      KAFKA(["Kafka: ${ENV}.entitlement"])
    end
    subgraph AM[mgr-applications]
      ING["GET /modules/{id}/bootstrap<br/>(ingress: self only)"]
      EGR["POST /modules/{id}/bootstrap<br/>(egress, scoped by applicationIds)"]
    end
    subgraph SC[folio-module-sidecar]
      EBS[EgressBootstrapService]
      TBL["Per-tenant egress tables<br/>ConcurrentHashMap&lt;tenant, routes&gt;"]
      LOOK[EgressRoutingLookup]
    end

    EBS -->|distinct applicationIds| ENT
    EBS -->|"POST applicationIds, returns requiredModules"| EGR
    EBS --> TBL
    KAFKA -->|ENTITLE/REVOKE/UPGRADE| EBS
    SC -->|startup, once| ING
    LOOK -->|resolve by X-Okapi-Tenant| TBL

The flow, in words:

For a given tenant, the sidecar asks mgr-tenant-entitlements which applications are entitled to that tenant (GET /entitlements?tenant={t}) and reduces them to a distinct set of application ids.
It calls mgr-applications POST /modules/{id}/bootstrap with { "applicationIds": [...] }. mgr-applications resolves the module's required/optional providers only within those applications and returns their discovery info.
The sidecar stores the resulting routes as that tenant's egress table.
At request time, EgressRoutingLookup picks the table for the request's X-Okapi-Tenant and resolves the egress route from it.

5. `mgr-applications` — the bootstrap API

The bootstrap interface (module-bootstraps) is bumped to v1.3 and gains an explicit ingress and scoped-egress split. The legacy global endpoint is unchanged.

Method	Path	Purpose	Body	Response	Status codes

Method	Path	Purpose	Body	Response	Status codes
`GET`	`/modules/{id}`	Unchanged. Full, globally-resolved bootstrap (module + all required providers across all applications).	–	`moduleBootstrap`	200 / 404 / 500
`GET`	`/modules/{id}/bootstrap`	Ingress. The module's own discovery/routes only; `requiredModules` is always empty.	–	`moduleBootstrap`	200 / 404 / 500
`POST`	`/modules/{id}/bootstrap`	Egress (scoped). Required/optional providers resolved within the supplied application scope only; the self module is not returned.	`egressBootstrapRequest`	`egressBootstrap`	200 / 400 / 404 / 500

Behavior & business rules

Ingress (GET …/bootstrap) returns only the target module's own provided interfaces; requiredModules is always empty.
Egress (POST …/bootstrap) restricts required-module resolution to providers whose owning application is in applicationIds. Providers outside scope are simply omitted (can yield an empty requiredModules).
404 — if the requested module itself is not present in the supplied scope (or does not exist), egress returns 404. Enforced in service logic after the scope-filtered query (EntityNotFoundException → 404).
400 — empty/missing applicationIds returns 400. Enforced by bean validation on the request DTO (@NotNull @Size(min = 1)) before the controller runs.
Dedup / version selection — required modules are filtered to the interfaces the module actually requires/optionally-requires and deduplicated by module name, keeping the highest version (same rule as GET /modules/{id}).
No tenant context in mgr-applications. Scoping is applied purely as a SQL filter over the existing v_module_bootstrap view by application id — there is no DB schema or view change and no tenant awareness on the server side. Tenant→applications mapping lives entirely in the sidecar/TE.

6. `folio-module-sidecar` — tenant-scoped egress routing

When routing.tenant-scoped.enabled=true:

Ingress is loaded once at startup, globally, via GET /modules/{id}/bootstrap (the ingress endpoint). It is not per-tenant. (When the flag is off, the sidecar uses the legacy full GET /modules/{id} bootstrap instead.)
Egress is resolved per tenant. Each tenant's table is built from the applications entitled to it and held in memory.

How a tenant's egress table is built

GET /entitlements?tenant={t} → distinct applicationIds → POST /modules/{id}/bootstrap { applicationIds } → store the returned requiredModules as routes for that tenant.

A tenant with no applications entitled to it stores an empty table (no bootstrap call).
Tables are stored in a ConcurrentHashMap keyed by tenant name inside EgressRoutingLookup.

When tables are (re)built — the trigger model

There are two complementary refresh mechanisms:

Trigger	Source	What it does

Trigger	Source	What it does
Membership change	`EntitlementsEvent` (internal, derived from the enabled-tenant set)	Reconciles per-tenant tables against the enabled-tenant set — builds newly-enabled tenants, drops revoked tenants. Fires at startup and on every tenant enable/disable.
Version change (UPGRADE)	Kafka `${ENV}.entitlement` → `TenantEntitlementConsumer`	On an `UPGRADE` for this module, rebuilds the affected tenant's egress table — a version change that does not alter tenant membership. (`ENTITLE` does not trigger this consumer path; new-tenant builds come from the membership reconcile above.)
Required-module discovery	`RoutingService.onDiscovery()`	When a required (egress) module's discovery changes, refreshes all enabled tenants' egress tables.

Request-time resolution

EgressRoutingLookup reads X-Okapi-Tenant, looks up that tenant's table, and resolves the egress route from it. If the tenant header is missing/unknown, or the table has not been built yet (startup window), there is no match and the request falls through the routing chain (see error behavior below).

7. The new flag(s)

The feature is controlled by one primary flag, with a coupled secondary flag so the common case "just works."

Env var	Config property	Default	Purpose

Env var	Config property	Default	Purpose
`SIDECAR_TENANT_SCOPED_ROUTING_ENABLED`	`routing.tenant-scoped.enabled`	`false`	Primary flag. Enables tenant-scoped egress routing. When off, egress routing keeps its existing global behavior, unchanged.
`SIDECAR_FORWARD_UNKNOWN_REQUESTS`	`routing.forward-to-gateway.enabled`	Defaults to the value of the primary flag (`${SIDECAR_FORWARD_UNKNOWN_REQUESTS:${SIDECAR_TENANT_SCOPED_ROUTING_ENABLED:false}}`)	Forwards unresolved egress to the gateway instead of returning `404`.
`SIDECAR_FORWARD_UNKNOWN_REQUESTS_DESTINATION`	`routing.forward-to-gateway.destination`	`http://api-gateway:8000`	Gateway URL used for forwarded unresolved egress.

Why the coupling matters. Because tenant tables are built asynchronously, there is a startup window where a tenant's egress is unresolved. Defaulting forward-to-gateway to the tenant-scoped flag means that, by simply enabling tenant-scoped routing, unresolved egress is forwarded to the gateway rather than failing with 404. An operator can still override the gateway-fallback behavior explicitly via SIDECAR_FORWARD_UNKNOWN_REQUESTS.

8. Runtime behavior & edge cases

Startup (flag on)

Ingress routes loaded once via GET /modules/{id}/bootstrap.
Per-tenant egress tables built for every currently-enabled tenant (reconcile against the enabled-tenant set).
During the build window, egress for a not-yet-built tenant is unresolved → forwarded to gateway (fallback default-on) or 404 (if fallback explicitly disabled).

Entitlement lifecycle

Tenant enabled → membership reconcile builds its table.
Tenant disabled / revoked → its table is dropped.
Application upgraded for a tenant (UPGRADE) → that tenant's table is rebuilt (no membership change).
Required-module discovery change → all enabled tenants' tables refreshed.

Error & failure behavior

Unresolved egress (unknown/missing tenant, or table not built yet): no per-tenant match. With gateway fallback on (the default when tenant-scoped routing is enabled), the request is forwarded to the gateway; otherwise the chain returns 404 Route is not found.
Bootstrap or entitlement-fetch failure during a table build: logged at warn, the tenant's previously built table is retained, and the build is retried on the next entitlement/discovery/upgrade event. No request-time error is introduced by a failed rebuild. (Caveat: a tenant that has never built successfully simply stays absent until a later event succeeds — see open questions.)

9. Open questions and topics for discussion

Scale without a server-side cache. With no cache in mgr-applications, every sidecar startup and every refresh issues a fresh DB-backed POST /modules/{id}/bootstrap. A fleet-wide discovery change triggers refreshAllTenants() across many sidecars at once — a potential thundering-herd against mgr-applications/DB. There is currently no batching, jitter, or backoff.
Refresh blast radius. refreshAllTenants() rebuilds every known tenant on any required-module discovery change, regardless of whether a tenant is actually affected. Should it be scoped to only tenants to which the changed module's application is entitled?
Startup-window correctness. In tenant-scoped mode without gateway fallback, egress for a not-yet-built tenant returns 404. Is gateway forwarding always acceptable as the interim answer, or do some flows need startup to block until tables are warm?
Cross-instance consistency. Replicas can hold divergent tables during refresh windows, and the in-memory table swap is not atomic with reads. Is eventual consistency acceptable for routing, and what convergence-time bound do we want?

10. Future improvements

Caching (the headline follow-up)

Server-side cache in mgr-applications (the missing acceptance criterion). Add a Caffeine/@Cacheable layer on the scoped egress resolution keyed on (moduleId, normalized-sorted applicationIds). Sorting/normalizing the id set is essential for stable keys. Pair it with event-driven eviction on discovery/entitlement changes plus a TTL backstop so a missed event self-heals. Weigh key cardinality (the number of distinct app-set combinations) against caching at the per-(module, application_id) grain and composing in memory.
Sidecar-side TTL/backstop refresh on per-tenant tables, so a dropped Kafka event eventually self-corrects rather than relying solely on event-driven rebuilds.
Atomic table swap in the sidecar (build-then-publish, or replace the whole map reference) to remove the brief old/new coexistence window.

Load & efficiency

Batch endpoints — a multi-module bootstrap call and/or batched entitlement lookups to cut the N round-trips during refreshAllTenants().
Scoped refresh — only refresh tenants to which the changed module's application is entitled.
Refresh storm control — add jitter/backoff to fleet-wide refreshes.
Push vs. pull — evaluate mgr-applications publishing precomputed per-scope bootstrap deltas instead of the current pull model, improving cross-instance convergence.

Resilience

Retry-with-backoff on buildEgress() failures (today it logs and waits for the next event).

11. Links

Ticket: EUREKA-899
Feature docs in-repo: mgr-applications/docs/features/module-bootstrap.md, folio-module-sidecar/docs/features/tenant-scoped-egress-routing.md