1 Spike Overview

Spike Overview

This spike evaluates two approaches for how a module can obtain its entitlement information via the sidecar:

Approach A – Proxy mode:
The sidecar acts purely as a proxy, forwarding the module’s entitlement request to the manager components ( mgr-tenant-entitlements) and returning the response.
Approach B – Local mode (preferred):
The sidecar serves entitlement information directly from its own in-memory state, without calling manager components per request.

The local mode is preferred because it reduces latency, avoids exposing powerful admin tokens on each request, and is consistent with the sidecar’s role as the runtime “source of truth” for enabled tenants.

1. Sidecar Entitlement retrieval (startup & refresh)

The sidecar determines which tenants it is entitled to serve by calling manager services during its initial startup sequence and then periodically via a scheduled task.

Initial call (TE)
- Service: Manager Tenant Entitlements (TE)
- Purpose: Retrieve the list of tenant IDs entitled for this module.
- Endpoint:
  GET [TE_URL]/entitlements/modules/{moduleId}
  Example:
  http://mgr-tenant-entitlements:8080/entitlements/modules/mod-foo-1.0.0
Secondary call (TM)
- Service: Manager Tenant Manager (TM)
- Purpose: Resolve tenant IDs into tenant names, which the sidecar uses for routing and validation.
- Endpoint:
  GET [TM_URL]/tenants?query=(id == "...")

2. Sidecar Event-driven updates

To stay up to date in real time, the sidecar subscribes to a Kafka topic that publishes entitlement changes.

Topic: entitlement
Consumer: TenantEntitlementConsumer
Logic (for events where moduleId matches the sidecar’s module ID):
- ENTITLE / UPGRADE: add the tenant to the internal enabledTenants set.
- REVOKE (and similar): remove the tenant from the enabledTenants set.

3. Sidecar Security for manager REST calls

All communication with manager components (TE, TM, AM) is secured using an administrative token:

Authentication: Keycloak client_credentials flow in the master realm
Token source: ServiceTokenProvider.getAdminToken()
Transmission: the admin token is sent in the X-Okapi-Token header on every request to manager services.

3. Sidecar Scheduled “lazy” refresh

A scheduled task keeps the sidecar’s entitlement state reconciled over time.

Default schedule: every 10 minutes
Behavior: the cron task only triggers a refresh lazily, when a request passes through TenantFilter after the 10‑minute interval has elapsed.
Purpose: act as a “safety net” in case Kafka events are missed or there are synchronization issues.

Configuration

Property (default):
tenant-service.reset-task.cron-definition = 0 0/10 * * * ?
Location: src/main/resources/application.properties
Override: can be overridden via TENANT_SERVICE_RESET_TASK_CRON_DEFINITION (environment variable).

5. Sidecar: Why TE needs special handling

Requests to the Tenant Entitlements (TE) service require master-realm admin credentials. The standard EgressRequestHandler cannot be used because it obtains tokens for the current tenant (the module tenant), not the master realm.

To support TE calls securely, we introduce a specialized handler that explicitly uses the admin token.

6. Approach A – Proxy via TenantEntitlementRequestHandler

6.1 Specialized TenantEntitlementRequestHandler

This handler ignores the tenant context of the incoming request, fetches the master admin token, and forwards the request to TE using that token.


java
@Named("tenantEntitlementRequestHandler")
@ApplicationScoped
@RequiredArgsConstructor
public class TenantEntitlementRequestHandler implements RoutingEntryHandler {
  private final ServiceTokenProvider serviceTokenProvider;
  private final RequestForwardingService requestForwardingService;
  private final PathProcessor pathProcessor;

  @Override
  public Future<Void> handle(ScRoutingEntry routingEntry, RoutingContext rc) {
    return serviceTokenProvider.getAdminToken()
      .compose(adminToken -> {
        // Force the X-Okapi-Token to be the master admin token
        rc.request().headers().set(OkapiHeaders.TOKEN, adminToken);
        
        var updatedPath = pathProcessor.cleanIngressRequestPath(rc.request().path());
        var absUri = routingEntry.getLocation() + updatedPath;

        // Forward using the egress client (which handles TLS if needed)
        return requestForwardingService.forwardEgress(rc, absUri);
      });
  }
}

6.2 StaticRoutingLookup for TE

Because TE routes are not provided via bootstrap data, we add a static routing lookup that directs all /entitlements requests to the configured TE URL.


@ApplicationScoped
public class TenantEntitlementRoutingLookup implements RoutingLookup {
  private final String teUrl;

  public TenantEntitlementRoutingLookup(TenantEntitlementClientProperties properties) {
    this.teUrl = properties.getUrl();
  }

  @Override
  public Future<Optional<ScRoutingEntry>> lookupRoute(String path, RoutingContext rc) {
    if (path.startsWith("/entitlements")) {
      var entry = new ScRoutingEntry();
      entry.setLocation(teUrl);
      entry.setModuleId("mgr-tenant-entitlements"); // Symbolic
      return Future.succeededFuture(Optional.of(entry));
    }
    return Future.succeededFuture(Optional.empty());
  }
}

6.3 Registering the handler in the chain

In RoutingConfiguration.java, the TE handler is wired into the chain after the basic egress handler and before the not‑found handler.



@Named
@ApplicationScoped
public ChainedHandler teEgressHandler(
    TenantEntitlementRoutingLookup lookup,
    @Named("tenantEntitlementRequestHandler") RoutingEntryHandler handler,
    PathProcessor pathProcessor) {
  return new RoutingHandlerWithLookup(lookup, handler, pathProcessor);
}

// In chainedHandler() method:
return basicIngressHandler
    .next(basicEgressHandler)
    .next(teEgressHandler) // The new "secret" route for the module
    .next(notFoundHandler);

6.4 Why this proxy approach is secure and effective

Authorization: getAdminToken() ensures the sidecar uses its own admin credentials when calling TE.
No bootstrap dependency: TE does not need to be declared as a required module; the sidecar uses static configuration.
Transparency: From the module’s perspective, it simply calls GET /entitlements/modules/{my-id} on the sidecar and receives the TE response.

7. Approach B – Local Entitlement Interceptor (preferred)

Because the sidecar already maintains the authoritative set of enabled tenants (used to allow or block requests), returning entitlement information directly from the sidecar’s memory is faster, more reliable, and reduces load on manager services.

7.1 Exposing the data in TenantService

Add a method to TenantService that exposes the current list of enabled tenants in the expected format.


// In TenantService.java
public List<String> getEnabledTenants() {
  return List.copyOf(enabledTenants);
}

7.2 LocalEntitlementHandler

This handler intercepts a specific egress path from the module. When the module calls /entitlements/modules/{my-id}, the sidecar responds immediately from its in-memory state, without performing any network calls.



@ApplicationScoped
@RequiredArgsConstructor
public class LocalEntitlementHandler implements ChainedHandler {

  private final TenantService tenantService;
  private final ModuleProperties moduleProperties;
  private final JsonConverter jsonConverter;

  @Override
  public Future<Boolean> handle(RoutingContext rc) {
    String path = rc.request().path();
    String myEntitlementPath = "/entitlements/modules/" + moduleProperties.getId();

    // If the module is asking for its own entitlements
    if (path.equals(myEntitlementPath)) {
      List<String> tenants = tenantService.getEnabledTenants();
      
      // Construct a response that looks exactly like the MTE service response
      // so the module's existing client code doesn't have to change.
      Map<String, Object> response = Map.of(
        "entitlements", tenants.stream()
            .map(t -> Map.of("tenantId", t, "moduleId", moduleProperties.getId()))
            .toList(),
        "totalRecords", tenants.size()
      );

      rc.response()
        .putHeader("Content-Type", "application/json")
        .setStatusCode(200)
        .end(jsonConverter.toJson(response));
        
      return Future.succeededFuture(true); // Request handled locally!
    }

    return Future.succeededFuture(false); // Continue to next handler in chain
  }
}

7.3 Insert into the chain

In RoutingConfiguration.java, the local handler is inserted before the basic egress handler so that local requests are intercepted first.


// In RoutingConfiguration.java
@Named
@ApplicationScoped
public ChainedHandler chainedHandler(
    @Named("basicIngressHandler") ChainedHandler ingressHandler,
    @Named("localEntitlementHandler") ChainedHandler localHandler, // New
    @Named("basicEgressHandler") ChainedHandler egressHandler,
    ...) {
    
  return ingressHandler
      .next(localHandler)  // Check if it's a local request first
      .next(egressHandler) // Otherwise, try to route it outside
      .next(notFoundHandler);
}

7.4 Advantages of the local interceptor

Zero latency: No HTTP call to manager components; responses are served from memory.
No credentials on this path: No master admin token is required for the module’s entitlement requests.
Consistency: The module sees exactly the set of tenants that the sidecar is already using for routing.
Resilience: The module continues to function even if TE is unavailable, as long as the sidecar cache is valid.

8. Security boundaries for the local interceptor

In a zero‑trust setup, we must ensure the module cannot escalate its privileges by querying entitlements for other modules.

8.1 Identity verification (strict path matching)

The sidecar is deployed 1:1 with a module and knows its own ID (moduleProperties.getId()). The interceptor should only serve requests for that ID.

Example allowed:
- Request: GET /entitlements/modules/mod-foo-1.0.0 (where mod-foo is the current module)
- Action: return local data (200 OK)
Example blocked:
- Request: GET /entitlements/modules/mod-admin-9.9.9 (another module’s ID)
- Action: return 403 Forbidden or 404 Not Found

8.2 Information minimization

The TenantService only stores a list of tenant identifiers. It does not store:

Master admin credentials
Keycloak client secrets
Database connection strings
Entitlements of other modules

The module only receives the list of tenants it is already authorized to serve, mirroring the state that the sidecar uses internally.

8.3 Hardened LocalEntitlementHandler example



@Override
public Future<Boolean> handle(RoutingContext rc) {
  String path = rc.request().path();
  String myId = moduleProperties.getId();
  
  // 1. Only intercept if it looks like an entitlement request
  if (path.startsWith("/entitlements/modules/")) {
    
    // 2. Security Check: Is the module asking about ITSELF?
    if (path.endsWith("/" + myId)) {
      // SUCCESS: Return the local cached list of enabled tenants
      return serveLocalEntitlements(rc);
    } else {
      // SECURITY FAILURE: Module is trying to probe other modules
      rc.response().setStatusCode(403).end("Access Denied: You can only query your own entitlements.");
      return Future.succeededFuture(true); // Handled (blocked)
    }
  }
  return Future.succeededFuture(false); // Not an entitlement request, continue chain
}

9. Initialization race condition and readiness

A common race condition in sidecar architectures occurs if the module asks for entitlements before the sidecar finishes its first loadTenantsAndEntitlements() run. In that case, the module might receive an empty list and fail to start.

To handle this reliably, we combine:

A synchronous “wait until ready” behavior in the local handler, and
Standard health checks and module‑side retries.

9.1 “Wait-for-first-load” logic in LocalEntitlementHandler

Instead of returning an empty list, the handler waits until initialization is complete.


// Logic inside LocalEntitlementHandler
@Override
public Future<Boolean> handle(RoutingContext rc) {
  // ... path matching logic ...
  
  // Return a future that only completes when the sidecar is actually ready
  return tenantService.getInitializationFuture()
    .compose(unused -> {
      serveLocalEntitlements(rc);
      return Future.succeededFuture(true);
    })
    .onFailure(err -> {
      rc.response().setStatusCode(503).end("Sidecar is still initializing...");
    });
}

If initialization has completed successfully: return entitlements with 200 OK.
If initialization is still running: “park” the request until the future completes (non‑blocking in Vert.x).
If initialization failed: return 503 Service Unavailable, prompting the module to retry.

9.2 Sidecar health check

The sidecar exposes /admin/health. In a typical Kubernetes deployment, the module container should wait for the sidecar’s health to be UP before starting its main process.

Ensure ModuleHealthCheck reflects tenant/entitlement initialization state (e.g., report DOWN or STARTING until the first successful load completes).

9.3 Module retry strategy

The module should implement a retry policy for calls to its local sidecar:

Call: GET http://localhost:8081/entitlements/modules/{id}
On 503 or an unexpected empty list (when data is expected): wait (e.g., 1 second) and retry
Give up after a reasonable timeout (e.g., 30 seconds)

9.4 Avoid circular dependencies

Direct callbacks from the sidecar to the module (for example, sidecar calling the module’s /admin/health) are discouraged because they:

Make the sidecar aware of module‑internal APIs
Introduce circular dependencies (module waits for sidecar; sidecar calls module)

A cleaner pattern is:

Sidecar reports readiness via /admin/health
Module and orchestration environment use that signal plus retry logic on the entitlement endpoint.

Regarding security, the module already has visibility into all tenants in the cluster because it is subscribed to a topic that receives messages from every tenant, and each message payload includes the tenant name. In other words, the module inherently learns about all tenants from the messages it consumes, independent of any additional entitlement lookups.

Implementation plan

Use a static handler to retrieve tenant information from the sidecar’s in-memory structures instead of calling the MGR entitlement service for each request.
Provide a dedicated service/library that modules use to query the sidecar for the list of tenants they are entitled to.
For the first iteration (at least until the Trillium release), do not introduce any additional caching on the module side.
On each incoming message, the module will:
- Invoke the library to fetch the current list of entitled tenants from the sidecar.
- Compare the tenant from the message payload with the returned list.
- If the tenant is not present in the list, silently skip processing of that message.

Spike - Filter Kafka messages by tenant entitlements