Overview

The Sidecar architectural pattern is a design pattern used in software engineering to enhance or extend the functionality of a main application by attaching a secondary application or service to it. This secondary application, known as the "sidecar," runs alongside the main application and provides supporting features such as monitoring, logging, configuration, or networking services.

In this pattern, the sidecar is tightly coupled with the main application, sharing the same lifecycle and always deployed together. However, the sidecar's runtime environment remains independent, allowing it to be developed and maintained separately from the main application.

The Sidecar pattern is commonly used in microservices architectures, where each microservice can have its sidecar to handle cross-cutting concerns. This allows developers to implement these concerns consistently and avoid duplicating code across services. It also simplifies the main application code by offloading auxiliary tasks to the sidecar.

GitHub repository: https://github.com/folio-org/folio-module-sidecar

folio-module-sidecar uses Quarkus as a framework:

1. It allows building native image
   (only non-FIPS-compliant docker images because some of the BouncyCastle FIPS libraries: Quarkus FIPS Security)

2. FIPS-compliant

3. Lightweight and quick (compared to other solutions like Spring Boot Native Image, Envoy)

4. Java-based

5. Vertx under the hood, providing general router for ingress and egress requests and WebClient for external calls

folio-module-sidecar provides the following functionality:

1. Ingress requests handling
   incoming requests from API Gateway or another module to the current Folio module

2. Egress requests handling
   outcoming requests from the current Folio module to another Folio module

3. Authentication
   based on x-okapi-token or Authorization header

4. Authorization
   based on Keycloak policies and permissions

5. Self-requests
   based on x-okapi-sidecar-signature header value

6. User impersonation
   Consortia support and cross-tenant requests

7. HTTP Transactional logging

Sidecar initializing and runtime

Bootstrap process

During startup, folio-module-sidecar calls the following modules according to the sequence diagram:

Open

Sidecar bootstrap (initialization) flow

Sidecar, as shown on the diagram, calls mgr-components to be part of the Eureka system:

1. It calls mgr-applications to construct egress and egress request caches, that store routing entries from module descriptor by id and for other related module descriptors, defined by optional and required interface dependencies.

   1. For egress requests there is a support to call API Gateway if the egress routing entry is not matched, it is controlled via environment variables: SIDECAR_FORWARD_UNKNOWN_REQUESTS and SIDECAR_FORWARD_UNKNOWN_REQUESTS_DESTINATION which by default pointing to the Kong API Gateway.

2. It calls mgr-tenant-entitlements and mgr-tenants to find out for which tenants sidecar is enabled, requests, containing tenant id, that is not cached in sidecar will be rejected with 400 Bad Request Application is not enabled for tenant: {{tenantName}}

Sidecar Runtime

During runtime, folio-module-sidecar receives events from the message bus (Apache Kafka) containing:

1. Tenant entitlement, revoke, and upgrade events. This information is used to update the active tenant's cache to support the following cases:

   1. During the application uninstallation (revoke) - all related sidecars must be disabled for the affected tenant

   2. During the application upgrade process - all upgraded (new) modules must be enabled for the tenant and all deprecated modules must be disabled for the affected tenant

2. Discovery information change information:

   1. Sidecar can update an egress request location cache if discovery information is changed in mgr-applications

3. Sidecar also caches authorized requests for the JWT using session_state claim if present until the token is expired:

   1. If a user performs logout or logout all operations - folio-module-sidecar can clean affected caches and forbid the next request using cached JWT

Sidecar filters

The filter approach is a way to intercept and process HTTP requests and responses before they reach a point when the request must be forwarded to the underlying Folio module. In request, they are responsible for all functionality and executed in chain.

Ingress request filters in details

1. SelfRequestFilter

Description

This filter is responsible for validating if a request is a module self-request. These types of requests are allowed without authentication and authorization.

Skip Conditions

Never

2. KeycloakSystemJwtFilter

Description

Retrieves and parses system token, issued by keycloak in another sidecar. This functionality was added to support module-to-module requests when the system user is not defined by the module or not all permissions are present in the existing user JWT, including tenant and module permissions.

This token is issued using client credential flow, Keycloak client is created during tenant and realm initialization in mgr-tenants and controlled by the following environment variables:

1. KC_SERVICE_CLIENT_ID Tenant-specific client ID for authenticating module-to-module requests.

2. service client password is stored in SecureStore: AWS Parameter Store, HashiCorp Vault or provided via an ephemeral (dev mode) store configured by system properties.

Skip conditions

1. matched routing entry with interfaceType is equal to system

2. matched routing entry does not require any permission

3. a request does not contain a value in X-System-Token header

3. KeycloakJwtFilter

Description

Parses the JWT from x-okapi-token or Authorization headers.

Also, support “dummy” token for mod-pubsub:

Skip conditions

1. A routing entry matched with interfaceType is equal to system, but interface id is not equal to _timer

4. KeycloakTenantFilter

Description

1. Resolves a tenant name from system JWT and user JWT.

2. Allows cross-tenant requests, if environment variable ALLOW_CROSS_TENANT_REQUESTS for sidecar is set to true.

3. Validates that x-okapi-token and x-system-token are issued for the same tenant (skipped if #2 is defined)

4. Validates that x-okapi-tenant is the same as resolved tenant id (skipped if #2 is defined)

Skip conditions

1. an interface id is not equal to _timer

2. interfaceType is equal to system

3. Self request

4. Routing entry without required permissions

5. TenantFilter

Description

Check if a tenant in x-okapi-tenant header is in the cache with active tenants.

Skip conditions

1. Tenant install request (interface _tenant v1.0, v1.1, or v2.0)
   It is done to allow installation requests from mgr-tenant-entitlements to enable modules, an event with the enabled application for a tenant is received only when a response is received from _tenant API call

6. KeycloakImpersonationFilter

Description

1. Retrieves user from mod-users-keycloak

2. If found the filter obtains an access token from Keycloak using impersonation: Server Administration Guide: Impersonating a user
   Impersonation call is done using Keycloak token endpoint and grant_type equal to urn:ietf:params:oauth:grant-type:token-exchange

3. Sets the obtained impersonated JWT to x-okapi-token header

Skip conditions

1. The tenant name or JWT is not present in the request

2. A token issuer is equal to x-okapi-tenant header

7. KeycloakAuthorizationFilter

Description

Authorizes JWT in Keycloak using grant_type equal to urn:ietf:params:oauth:grant-type:uma-ticket and scope equal to {routingEntryStaticPath}#{httpMethod}

Skip conditions

1. matched routing entry with interfaceType is equal to system

2. matched routing entry does not require any permission

8. SidecarSignatureFilter

Description

Adds a unique sidecar signature to x-okapi-sidecar-signature header. This signature is used to identify self-request from module

Skip conditions

Never

9. DesiredPermissionsFilter

Description

1. Removes the X-Okapi-Permissions header from the request.

2. Retrieves the userId from the request headers.

3. If the userId is not found, it logs a debug message and returns the context as is.

4. If the request has desired permissions, it fetches the user permissions using mod-users-keycloak and populates the X-Okapi-Permissions header with these permissions.

Skip conditions

Never

Egress Request Filters in details

1. SidecarSignatureRemover

Description

Removes self-request signature from x-okapi-sidecar-signature header

Module Prefix strategies

The module prefix strategies allow to adjust sidecar for different discovery URLs in the deployment

It can be defined using the environment variable: SIDECAR_MODULE_PATH_PREFIX_STRATEGY with the following values: PROXY, STRIP, and NONE

1. PROXY is used when routing between Kong, Sidecar, and Module requires a path prefix in the location URL:

   ## Module sidecar location https://sidecar-foo.module-subnet.example.com/sc-foo ## Folio module location https://mod-foo.module-subnet.example.com/mod-foo

2. STRIP is used when routing between Kong, Sidecar requires a path prefix in the location URL, but Sidecar and Module can be routed to each other using a private subnet:

3. NONE is used when routing between Kong, Sidecar, and Module does not require a path prefix in the location URL

Module prefix is based on the environment variable: MODULE_NAME, which is necessary for sidecar

Signing Key Rotation

Sidecars must be aware of the new signing keys. This will be done automatically by calling “/protocol/openid-connect/certs“ once again if it receives a token with a new signing key id. But it will be done only once per hour, as currently there is a line: jwtAuthContextInfo.setForcedJwksRefreshInterval($KC_FORCED_JWKS_REFRESH_INTERVAL); which means that forced refresh (if the key is not found) will be performed only once per specified value in minutes (by default it is one hour).

This logic is provided by the following libraries:

• https://github.com/folio-org/applications-poc-tools/tree/master/folio-auth-openid

• https://smallrye.io/docs/smallrye-jwt/index.html

 User: The user initiates the process by logging into Keycloak.

1. Keycloak (keycloak): Keycloak authenticates the user and returns a JWT token with a new signing key ID.

2. User to Sidecar: The user requests "Module A" via the sidecar, providing the JWT token.

3. Sidecar Processing:

   • The sidecar checks the signing key ID in the JWT token.

   • If the signing key ID is unknown and the last JWKS refresh was over an hour ago, the sidecar fetches the new signing keys from Keycloak and updates its local cache. It then forwards the user's request to "Module A".

   • If the signing key ID is known, or if a JWKS refresh occurred within the last hour, the sidecar forwards the user's request to "Module A" without fetching new keys.

   • If the signing key ID is unknown and a JWKS refresh occurred less than one hour ago, the sidecar rejects the user's request to avoid too frequent key updates.

4. Module A (moduleA): "Module A" receives and processes the request forwarded by the sidecar.

Sidecar communication example

The example is based on the 2 simple modules in app-platform-minimal - mod-notes and mod-users, when the user can create a note and mod-notes will retrieve metadata from mod-users

Open

Deployment

folio-module-sidecar allows the building of a Docker container using OpenJDK base container or native image.

• Building And Running JVM Based docker container

• Building and Running Native docker container

Java-based Docker image

Minimal memory requirements

RAM requirements are calculated from two things: java process RAM and containers OS ram usage. This values should be configured separately and container limit should include both values.

• JAVA_OPTIONS

  • -XX:+UseZGC -Xms64m -Xmx64m -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m

• Docker Container’s OS

  • 128m

Algorithm for calculation

• Module’s memory / 4 and it will be a Xms and Xmx numbers

• Xms and Xmx numbers should not be less then 64m and not higher then 256m

• Metaspace size should be 96m or higher

• Container limit should be settled as Xms value + Container’s overhead value ((Module’s memory / 4) >64m <256m + 128m + 128m)

Example

mod-inventory 2048m RAM

Xmx and Xms value = 2048/4 = 512m

512m is higher than 256m, so the final value for Xmx and Xms will be 256m

• JAVA_OPTIONS should be like-XX:+UseZGC -Xms256m -Xmx256m -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m

• Container limit should be 256m + 128m + 128m = 512m

Minimal CPU Requirements

CPU allocation depends on container’s load but not less then 0.2 CPU

Native Docker Image

To be implemented