https://folio-org.atlassian.net/browse/EUREKA-847

Objective: Recommend approaches for the gaps identified so far in the Kafka formalization design, structured per-question so additional gaps discovered can be incorporated as new sections without needing to rework the document.

The Kafka formalization design proposes that modules declare their Kafka producer/consumer usage in their module descriptors. Two gaps have been identified so far where that mechanism doesn't obviously apply: sidecars, which have no module descriptor at all, and manager services, which have module descriptors but aren't wrapped by any application descriptor that the entitlement/dependency-checking system would examine.

Reference Material: Kafka Topics Management Infrastructure Module - Design Document

In scope:

• How sidecars should declare their Kafka interactions

• How manager services should declare their Kafka interactions

• Registration/validation mechanics for both

Out of scope:

• Central topic management (health-check timing relative to "topics ready," whether notification failure fails entitlement, HA visibility into all instances of a service) — tracked in https://folio-org.atlassian.net/browse/EUREKA-848.

• Fully specifying platform-type application enforcement logic in mgr-tenant-entitlements (introduced below, not delivered by this spike).

• Mgr-tenant-entitlements (MTE) processing logic for non-sidecar virtual-module content kinds (permissions-only, data-bearing) — introduced below for completeness, not delivered by this spike.

• Actually wrapping mgr-* manager service module descriptors in a platform application descriptor (Question #2) — proposed here, implementation deferred as follow-on work.

Give the sidecar its own module descriptor (requires + kafkaTopics only). Introduce a new virtual application for the sidecar (e.g. app-folio-sidecar) with dependencies declared, entitled per tenant like any regular application. For the three manager services, bundle their module descriptors into a new app-mgr-services application with a new platform application type, reserved for genuine deployment-wide singletons (Keycloak, Kong, the Kafka broker, and now manager services), and auto-checked against every tenant's dependencies with no explicit per-tenant entitlement requirement. Getting there requires a new mechanism for entitlement of applications with type=platform, as well as softening the requirement that discovery info is required for every application.

Sidecars presently consume events from several topics:

• {{env}}.discovery (from mgr-applications) - Updates/changes to discovery information

• {{env}}.entitlement (from mgr-tenant-entitlements) - Updates/changes to tenant entitlements

• {{env}}.{{tenant}}.mod-login-keycloak.logout (from mod-login-keycloak) - Logout events informing the sidecar to invalidate AuthN cache entries for the associated session.

Since the sidecars do not have their own module descriptor, this presents a gap in the Kafka Topics Management Infrastructure Module - Design Document.

Each application whose module(s) rely on the sidecar declares the sidecar's fixed set of Kafka topics directly in its own application descriptor.

Pros

• No new descriptor type or registration path — reuses the application descriptor mechanism as-is.

Cons

• A sidecar's Kafka needs don't vary by which application its module belongs to, so every application descriptor would redeclare the same fixed set of topics — pure duplication.

• Duplicated copies can drift out of sync with each other and with reality.

• Conflates "what does this application consist of" with "what does the platform's sidecar infrastructure need" — two unrelated concerns.

Assume sidecars are homogeneous within a given deployment and declare their Kafka interactions once, in a single descriptor scoped to the deployment rather than to any particular application or module.

Pros

• Simple: one artifact per deployment, no per-application duplication.

Cons

• Ties the declaration to the deployment rather than to the sidecar software version — the thing that actually changes when declared Kafka interactions change.

• Assumes homogeneity that conflicts with the platform's direction of supporting multiple releases concurrently in one deployment, where different tenants may need different sidecar versions running side by side.

• Not viable if/when heterogeneous sidecar versions become necessary.

Give the sidecar its own module descriptor — shaped similar to an edge module descriptor (exists purely for dependency checking, not for declaring interfaces/routes or permissions/capabilities) — and register it the same way any module dependency is registered, via a wrapping application descriptor.

• requires: the interfaces of mgr-tenant-entitlements, mgr-applications, etc. that the sidecar depends on.

• kafkaTopics: consumer-only entries for the three topics identified above, using the schema already proposed in the design doc. No producer role is needed today, but the schema shouldn't preclude one later.

• No provides, no permissionSets — the sidecar doesn’t provide any interfaces, handlers, or permissions/capabilities.

• Versioned per sidecar release.

This approach reuses the existing module descriptor schema. There is no need for a new descriptor type or validation logic.

The sidecar module descriptor is wrapped in a virtual-application descriptor — an application descriptor providing no real modules, existing solely to carry a dependency declaration (or other non-runtime content, see Virtual Module Contents/Purposes). The idea of a virtual-application is not new; it’s been a topic of discussion several times by the architecture team, and have even been mentioned to broader audiences (e.g. WOLFcon) as something we may introduce in the future.

Real applications (e.g. app-acquisitions, etc.) would declare a normal application-level dependency on the sidecar virtual-application. The existing application-level dependency validation, performed by the tenant entitlement manager would not require changes to support this.

If a module descriptor declares an empty provides section, discovery information should not be required for entitlement to succeed. This is considered safe because there’s no need to create routes in the API gateway for modules which don’t provide an HTTP interface. Presently discovery information is required for all backend modules (yes, even edge modules - here it looks like we’re registering bogus discovery info to get around the check). This will need to change, and actually makes for a cleaner implementation anyways.

Operators are responsible for keeping descriptor, discovery, and container image versions in sync today; runtimes don't self-report. Sidecars already check in with manager services during bootstrap (They load tenant/entitlement data from manager services on startup with a periodic resync, default every 5 minutes). That said, it seems reasonable to have the sidecars include their actual running version in these requests for the purpose of validating expected vs actual versions (reported version vs. version implied by the tenant's entitled app-sidecar dependency). While not a strict requirement, this should be considered. If we choose to do this, it probably makes sense to make it feature flagged, defaulting to permissive/warn in the logs, with a stricter option (e.g. fail bootstrap) available to operators.

Pros

• Reuses the existing module descriptor schema and validation engine (requires/provides checking), with direct precedent in edge modules — no new validation required.

• One descriptor per sidecar version, referenced by dependency rather than duplicated — less maintenance overhead compared to some other options

• Compatible with the multi-release-per-deployment direction.

• Generalizes cleanly: the same virtual-application/virtual-module mechanism also covers unrelated known use cases (settings-permissions, reference-data) without new machinery. See Virtual Module Contents/Purposes

• Minimal (no?) new MTE enforcement logic required for the sidecar case itself — dependency checking of application dependencies is unchanged from today.

Cons

• Depends on virtual-applications, a concept that isn't formally specified yet.

• Several mechanics still need verification against the real codebase before this is final (see Open Issues).

Manager services (mgr-tenant-entitlements, mgr-applications, mgr-tenants) emit Kafka events too — e.g. mgr-tenant-entitlements publishes events for system user creation and capability/capabilitySet creation. Unlike sidecars, they do have real module descriptors which provide interfaces, but don’t require any! Furthermore, these module descriptors are not present in any application descriptor(s). That suggests inter-manager-service interactions sit outside dependency checking entirely today. Simply adding kafkaTopics to these descriptors would literally have effect at all.

Do nothing structurally new — just extend each mgr-* module descriptor with a kafkaTopics block.

Pros

• Trivial to implement; no new registration or bootstrap concerns.

Cons

• Given the empty requires and no presence in an application, nothing in the entitlement/dependency-checking pipeline currently examines these descriptors in a way that would validate or enforce kafkaTopics either. This doesn't actually close the gap — it just adds an unchecked field next to other unchecked fields (requires).

Bundle the three mgr-* module descriptors into a new app-mgr-services application descriptor, typed platform (introduced below), registered with mgr-applications. This allows the platform to perform the same dependency-checking (and, once added, kafkaTopics-checking) logic on manager services as it already performs on everything else.

Introduce a type field on application descriptors:

This is the only new taxonomy this spike proposes. app-sidecar (Question #1) doesn't need it — the default (unspecified/regular) behavior already suffices there. app-mgr-services is the motivating case for platform: manager services are genuinely deployment-wide singletons, unlike the sidecar, which must support multiple concurrent versions across tenants.

Registering app-mgr-services looks circular at first glance (mgr-applications registering a descriptor that describes itself), but isn't: registration is informational and doesn't gate mgr-applications' own startup. An operator or bootstrap script can call its existing registration API once the process is up.

To avoid depending on external automation for this step, mgr-applications could instead self-register app-mgr-services idempotently on its own startup, from a location given by an env var pointing to the location of the application descriptor.

Considered: Today manager service locations are hard-wired via environment variables. The idea of replacing this with standard this with a the API gateway's location, and let normal application + discovery registration populate gateway routes so they could reach each other like any other module.

Dismissed: API Gateway route population lives in mgr-tenant-entitlements, and the route expressions are tenant-scoped by design (to support the multi-release-per-deployment direction). Platform-scoped applications like app-mgr-services have no tenant context, so we’d have to be able to support a mix of both tenant-agnostic and tenant-aware routes, which introduces scope creep and complexities instead of simplifying things, which was the intent of this idea.

Considered: mgr-tenant-entitlements' descriptor declares real permissionSets despite having no wrapping application, so something plausibly consumes it for permission-registration purposes (e.g. Keycloak role/permission setup) independently of the application/entitlement system — if so, that might be a simpler than the platform-application wrapping above.

Dismissed : No such independent mechanism exists. The manager services have module descriptors, but I don’t see evidence that these descriptor are ever registered, or used (at least not yet!)

Because platform-scoped applications (e.g. app-mgr-services, app-keycloak) are automatically factored into every tenant's dependency resolution without an explicit per-tenant entitlement record, a tenant-scoped entitlement request can interfere with a platform-scoped entitlement that's in progress. Rather than blocking one side while the other completes (which would hold open a connection/thread across hops (load balancers, API gateways) with their own timeouts), collisions are resolved with a non-blocking exclusive lock: a platform-scoped entitlement takes an exclusive lock (e.g. a Postgres pg_try_advisory_lock or SELECT ... FOR UPDATE NOWAIT on a sentinel row) for the duration of its operation. Any tenant-scoped operation performs a non-blocking check before proceeding. On contention (the lock is taken), either side fails fast with 409 Conflict and a Retry-After header rather than waiting. Retrying is the caller's responsibility (expected to be automation in most cases). The MTE does not retry internally. Tenant-scoped operations never lock against each other, only against an in-progress platform-scoped operation. Since platform-scoped operations are typically brief (no tenant-API calls for ordinary deployment-wide infrastructure changes), the exclusive window is expected to be short. The one exception is the coupled upgrade scenario, which bundles real tenant-scoped application installs into the same call as the platform-scoped change and can hold the lock for longer than the simple case. For example app-keycloak + app-platform-minimal (due to required changes in mod-*-keycloak).

Pros

• Registration is not actually circular, just needs a deliberate bootstrap step (explicit registration or possibly a self-loading mechanism).

• Doesn't require any change to how mgr-* services actually communicate at runtime.

Cons

• Yet another application descriptor to maintain and register

Modules declare their Kafka producer/consumer usage via two new ModuleDescriptor fields, producerKafkaTopics and consumerKafkaTopics. Real topic names embed placeholders — confirmed against three independent modules (folio-module-sidecar, mod-search, mod-data-export-spring) — most commonly {{env}} (a deployment-wide constant) and {{tenant}} (varies per tenant, often wildcard-subscribed via a real regex pattern at the messaging-library level). The schema captures these as opaque tokens; it does not attempt to resolve or understand them.

The two-field split (rather than one kafkaTopics field with a role discriminator described in the original proposal) mirrors the existing schema sections — InterfaceDescriptor.json (provides) and InterfaceReference.json (requires/optional). This allows for version cardinality to differ by role. (Note: mgr-applications' own moduleDescriptor.json is currently just a stub({"type": "object"}). The fully-fledged ModuleDescriptor schema hasn't been migrated from the OKAPI repository yet. That's a separate, issue only mentioned here for completeness/clarity.)

There is no optional field on the producer side — see rationale section for details.

provides/requires in the real Okapi schema aren't one shape with a role discriminator — they're two distinct types (InterfaceDescriptor, InterfaceReference) in two distinct arrays. That split is what lets Okapi give them different version cardinality (single vs. space-delimited multi-value) without conditional schema logic. producerKafkaTopics/consumerKafkaTopics mirrors this directly, for the same reason, and picks up the same benefit: each side's schema enforces its own rules without needing if/then validation (which isn't clean in JSON Schemas anyway).

A side effect: role disappears entirely. It's now implicit in which array an entry lives in, same as requires/provides never needed a role field.

Started as name, renamed to topic for specificity. pattern was considered and rejected: application.yml/application.properties often have a property called topic-pattern already, which holds escaped raw Java regex (e.g. (${folio.environment}\.)(.*\.)authorities\.authority). Naming our field "pattern" would invite authors to paste that regex in directly — exactly what the {{env}}/{{tenant}} token convention exists to avoid.

• String, not float or array. A float can't distinguish "1.10" from "1.1" (both collapse to 1.1). An array was considered but rejected in favor of consistency with how versions are specified elsewhere in the ModuleDescriptor schema.

• Major.minor only, no patch. The official Okapi schema documents both InterfaceDescriptor.version and InterfaceReference.version as "major.minor format". However, it’s worth noting that InterfaceComparisonUtils.versionParts tolerates an undocumented third patch segment. In other words the implementation is more permissive than the published contract. Only noted here for completeness. Major and minor should be sufficient.

• No event-schema mechanism. There is presently no way to declare the actual shape of the event payload — version is a convention-enforced promise (major = breaking, minor = compatible) with nothing to validate it against. That's a real limitation, but not a new one: API interface versioning has the exact same gap. Formalizing event-schema declaration is a future consideration, not part of this work.

optional controls entitlement's failure mode when a consumer's dependency isn't satisfied. There's no equivalent "optional producing" concept — nothing about entitlement fails because a producer entry goes unmatched. Splitting the two arrays let this asymmetry fall out naturally instead of needing a field that's meaningless on one side.

Checked every boolean field in the real Okapi schemas (RoutingEntry.delegateCORS, LaunchDescriptor.dockerPull, Permission.visible, EnvEntry.required) — none are nullable. A primitive boolean in the Java model can never actually hold null, so allowing it in the schema wouldn't reflect anything the real serialization could produce. Keeping it a plain two-state boolean means consuming code never needs a null-check or a defensive Boolean.TRUE.equals(x) idiom.

Some of the fields present in the original proposal are absent here (e.g. config, etc.). This is because they pertain to topic centralized topic management, which is out of scope for this document.

The entitlement-time validator normalizes every topic field by replacing each {{...}} token with one canonical wildcard marker, then compares the normalized strings for equality. It never resolves {{env}} to the deployment's real value, never substitutes a real tenant, and never needs to understand what any token name means. This is deliberately the only thing the validator checks — see below for what's explicitly excluded.

Version compatibility is not exact-match. It reuses InterfaceComparisonUtils's existing semver-style rule (same major required; the consumer's declared version is satisfied by an equal-or-newer producer minor/patch) — the same logic already applied to requires/provides — rather than a Kafka-specific comparison.

Two real cases forced a decision here, both resolved the same way — one entry per literal/pattern shape, never alternation or optionality embedded inside a single entry:

• mod-search subscribes to inventory.(instance|holdings-record|item|bound-with) via one regex with alternation (application.yml, events listener) → modeled as four separate consumerKafkaTopics entries.

• mod-data-export-spring subscribes via ${ENV:folio}.(.*\.)?data-export.job.update — a genuinely optional tenant segment → modeled as two separate entries ({{env}}.{{tenant}}.data-export.job.update and {{env}}.data-export.job.update), rather than inventing a {{token?}} optional-marker syntax.

This keeps the shape-comparison mechanism simple: it only ever parses opaque {{...}} tokens, never alternation or optionality syntax. The downside is verbosity, but the tradeoff seems worth it.

Introducing checks which go beyond the simple shape comparison were considered. The mgr-tenant-entitlements knows for instance what {{env}} is, and can check {{tenant}} against the target tenant in the entitlement call. However, this was rejected as there’s a discrepancy - the topics are declared in static module descriptors. Performing these more sophisticated checks during entitlement requires deployment/runtime configuration information. Not only does the entitlement manager lack the full picture, but runtime configuration can change at any time, including after entitlement. To illustrate the point, let’s say mod-foo has a feature flag which controls whether it publishes to {{env}}.{{tenant}}.foo or {{env}}.foo. The module knows about that feature flag, but mgr-tenant-entitlements doesn’t. Even if it did, it wouldn’t have any awareness of the flag being changed after entitlement.

Instead of declaring topics, declare events instead. This avoids the problem of placeholders/tokens, and puts us on a path that’s more closely aligned with longer term goals of tighter, more strict enforcement of async/event contracts.

See Feedback on EUREKA-847: Event Identifier as Dependency Unit in Kafka Topics Management

• 1.1 Add producesEvents / consumesEvents to ModuleDescriptor.

• 1.2 Add scope (regular/platform) to ApplicationDescriptor.

• 1.3 Generalize the tenant-agnostic-routing check: KongGatewayService.buildTenantHeaderExpression and ApisixRouteFactory.buildVars treat a module as tenant-agnostic when its owning application has scope: platform, not only when isMgrComponent() is true.

• 1.4 Release a new version and bump it in the consuming repos' pom.xml.

• 2.1 Author the sidecar's module descriptor.

  • requires: mgr-applications / mgr-tenant-entitlements / mgr-tenants interfaces (matches its existing AM_CLIENT_URL/TE_CLIENT_URL/TM_CLIENT_URL REST dependencies).