Overview

The Sidecar architectural pattern is a design pattern used in software engineering to enhance or extend the functionality of a main application by attaching a secondary application or service to it. This secondary application, known as the "sidecar," runs alongside the main application and provides supporting features such as monitoring, logging, configuration, or networking services.

In this pattern, the sidecar is tightly coupled with the main application, sharing the same lifecycle and always deployed together. However, the sidecar's runtime environment remains independent, allowing it to be developed and maintained separately from the main application.

The Sidecar pattern is commonly used in microservices architectures, where each microservice can have its sidecar to handle cross-cutting concerns. This allows developers to implement these concerns consistently and avoid duplicating code across services. It also simplifies the main application code by offloading auxiliary tasks to the sidecar.

GitHub repository: https://github.com/folio-org/folio-module-sidecar

folio-module-sidecar uses Quarkus as a framework:

1. It allows building native image
   (only non-FIPS-compliant docker images because some of the BouncyCastle FIPS libraries: Quarkus FIPS Security)

2. FIPS-compliant

3. Lightweight and quick (compared to other solutions like Spring Boot Native Image, Envoy)

4. Java-based

5. Vertx under the hood, providing general router for ingress and egress requests and WebClient for external calls

folio-module-sidecar provides the following functionality:

1. Ingress requests handling
   incoming requests from API Gateway or another module to the current Folio module

2. Egress requests handling
   outcoming requests from the current Folio module to another Folio module

3. Authentication
   based on x-okapi-token or Authorization header

4. Authorization
   based on Keycloak policies and permissions

5. Self-requests
   based on x-okapi-sidecar-signature header value

6. User impersonation
   Consortia support and cross-tenant requests

7. HTTP Transactional logging

Sidecar initializing and runtime

Bootstrap process

During startup, folio-module-sidecar calls the following modules according to the sequence diagram:

@startuml

title "Sidecar bootstrap flow"
hide footbox

autonumber

skinparam sequenceMessageAlign right
skinparam sequenceMessageAlign right
skinparam responseMessageBelowArrow true

participant "Sidecar" as sidecar #5CCCCC
participant "mgr-applications" as mgrApplications #FFD073
participant "mgr-tenants" as mgrTenants #FFD073
participant "mgr-tenant-entitlements" as mgrTenantEntitlements #FFD073


sidecar -> mgrApplications ++#FFD073: GET /modules/:moduleId
note left of mgrApplications
//Retrieves bootstrap information for sidecar.//
//This information includes set of module routing//
//entries and a set of egress routing entries//
//based on required and optional interfaces,//
//specified in the module descriptor//
end note
sidecar <-- mgrApplications --: 200 Ok


sidecar -> mgrTenantEntitlements ++#FFD073: GET /entitlements/modules/:moduleId
note left of mgrTenantEntitlements
//Retrieves tenant entitlements(tenantId + applicationId) by module id//
end note
sidecar <-- mgrTenantEntitlements -- : 200 Ok


sidecar -> mgrTenants ++#FFD073: GET /tenants?query=id==(:tenantIds)
note left of mgrTenants
//Retrieves tenant information in batches based//
//on data retrieved from "mgr-tenant-entitlements"//
end note
sidecar <-- mgrTenants -- : 200 Ok

@enduml

Sidecar, as shown on the diagram, calls mgr-components to be part of the Eureka system:

1. It calls mgr-applications to construct egress and egress request caches, that store routing entries from module descriptor by id and for other related module descriptors, defined by optional and required interface dependencies.

   1. For egress requests there is a support to call API Gateway if the egress routing entry is not matched, it is controlled via environment variables: SIDECAR_FORWARD_UNKNOWN_REQUESTS and SIDECAR_FORWARD_UNKNOWN_REQUESTS_DESTINATION which by default pointing to the Kong API Gateway.

2. It calls mgr-tenant-entitlements and mgr-tenants to find out for which tenants sidecar is enabled, requests, containing tenant id, that is not cached in sidecar will be rejected with 400 Bad Request Application is not enabled for tenant: {{tenantName}}

Sidecar Runtime

During runtime, folio-module-sidecar receives events from the message bus (Apache Kafka) containing:

1. Tenant entitlement, revoke, and upgrade events. This information is used to update the active tenant's cache to support the following cases:

   1. During the application uninstallation (revoke) - all related sidecars must be disabled for the affected tenant

   2. During the application upgrade process - all upgraded (new) modules must be enabled for the tenant and all deprecated modules must be disabled for the affected tenant

2. Discovery information change information:

   1. Sidecar can update an egress request location cache if discovery information is changed in mgr-applications

3. Sidecar also caches authorized requests for the JWT using session_state claim if present until the token is expired:

   1. If a user performs logout or logout all operations - folio-module-sidecar can clean affected caches and forbid the next request using cached JWT

Sidecar filters

The filter approach is a way to intercept and process HTTP requests and responses before they reach a point when the request must be forwarded to the underlying Folio module. In request, they are responsible for all functionality and executed in chain.

Ingress request filters in details

1. SelfRequestFilter

Description

This filter is responsible for validating if a request is a module self-request. These types of requests are allowed without authentication and authorization.

Skip Conditions

Never

2. KeycloakSystemJwtFilter

Description

Retrieves and parses system token, issued by keycloak in another sidecar. This functionality was added to support module-to-module requests when the system user is not defined by the module or not all permissions are present in the existing user JWT, including tenant and module permissions.

This token is issued using client credential flow, Keycloak client is created during tenant and realm initialization in mgr-tenants and controlled by the following environment variables:

1. KC_SERVICE_CLIENT_ID Tenant-specific client ID for authenticating module-to-module requests.

2. service client password is stored in SecureStore: AWS Parameter Store, HashiCorp Vault or provided via an ephemeral (dev mode) store configured by system properties.

Skip conditions

1. matched routing entry with interfaceType is equal to system

2. matched routing entry does not require any permission

3. a request does not contain a value in X-System-Token header

3. KeycloakJwtFilter

Description

Parses the JWT from x-okapi-token or Authorization headers.

Also, support “dummy” token for mod-pubsub:

When a module (for example mod-circulation) registers events in mod-pubsub the request is sent without x-okapi-token (as we don't pass it during tenant install), mod-pubsub-client lib adds a default "dummy" token to
such requests and therefore sidecar should be able to process such request.

Skip conditions

1. A routing entry matched with interfaceType is equal to system, but interface id is not equal to _timer

4. KeycloakTenantFilter

Description

1. Resolves a tenant name from system JWT and user JWT.

2. Allows cross-tenant requests, if environment variable ALLOW_CROSS_TENANT_REQUESTS for sidecar is set to true.

3. Validates that x-okapi-token and x-system-token are issued for the same tenant (skipped if #2 is defined)

4. Validates that x-okapi-tenant is the same as resolved tenant id (skipped if #2 is defined)

Skip conditions

1. an interface id is not equal to _timer

2. interfaceType is equal to system

3. Self request

4. Routing entry without required permissions

5. TenantFilter

Description

Check if a tenant in x-okapi-tenant header is in the cache with active tenants.

Skip conditions

1. Tenant install request (interface _tenant v1.0, v1.1, or v2.0)
   It is done to allow installation requests from mgr-tenant-entitlements to enable modules, an event with the enabled application for a tenant is received only when a response is received from _tenant API call

6. KeycloakImpersonationFilter

Description

1. Retrieves user from mod-users-keycloak

2. If found the filter obtains an access token from Keycloak using impersonation: Server Administration Guide: Impersonating a user
   Impersonation call is done using Keycloak token endpoint and grant_type equal to urn:ietf:params:oauth:grant-type:token-exchange

3. Sets the obtained impersonated JWT to x-okapi-token header

Skip conditions

1. The tenant name or JWT is not present in the request

2. A token issuer is equal to x-okapi-tenant header

7. KeycloakAuthorizationFilter

Description

Authorizes JWT in Keycloak using grant_type equal to urn:ietf:params:oauth:grant-type:uma-ticket and scope equal to {routingEntryStaticPath}#{httpMethod}

Skip conditions

1. matched routing entry with interfaceType is equal to system

2. matched routing entry does not require any permission

8. SidecarSignatureFilter

Description

Adds a unique sidecar signature to x-okapi-sidecar-signature header. This signature is used to identify self-request from module

Skip conditions

Never

9. DesiredPermissionsFilter

Description

1. Removes the X-Okapi-Permissions header from the request.

2. Retrieves the userId from the request headers.

3. If the userId is not found, it logs a debug message and returns the context as is.

4. If the request has desired permissions, it fetches the user permissions using mod-users-keycloak and populates the X-Okapi-Permissions header with these permissions.

Skip conditions

Never

Egress Request Filters in details

1. SidecarSignatureRemover

Description

Removes self-request signature from x-okapi-sidecar-signature header

Module Prefix strategies

The module prefix strategies allow to adjust sidecar for different discovery URLs in the deployment

It can be defined using the environment variable: SIDECAR_MODULE_PATH_PREFIX_STRATEGY with the following values: PROXY, STRIP, and NONE

1. PROXY is used when routing between Kong, Sidecar, and Module requires a path prefix in the location URL:

   ## Module sidecar location
   https://sidecar-foo.module-subnet.example.com/sc-foo
   
   ## Folio module location
   https://mod-foo.module-subnet.example.com/mod-foo

2. STRIP is used when routing between Kong, Sidecar requires a path prefix in the location URL, but Sidecar and Module can be routed to each other using a private subnet:

   ## Module sidecar location
   https://sidecar-foo.module-subnet.example.com/sc-foo
   
   ## Folio module location
   https://mod-foo:8081/

3. NONE is used when routing between Kong, Sidecar, and Module does not require a path prefix in the location URL

   ## Module sidecar location
   https://sidecar-foo.module-subnet.example.com
   
   ## Folio module location
   https://mod-foo.module-subnet.example.com

Module prefix is based on the environment variable: MODULE_NAME, which is necessary for sidecar

Signing Key Rotation

Sidecars must be aware of the new signing keys. This will be done automatically by calling “/protocol/openid-connect/certs“ once again if it receives a token with a new signing key id. But it will be done only once per hour, as currently there is a line: jwtAuthContextInfo.setForcedJwksRefreshInterval($KC_FORCED_JWKS_REFRESH_INTERVAL); which means that forced refresh (if the key is not found) will be performed only once per specified value in minutes (by default it is one hour).

This logic is provided by the following libraries:

• https://github.com/folio-org/applications-poc-tools/tree/master/folio-auth-openid

• https://smallrye.io/docs/smallrye-jwt/index.html

private final int jwksRefreshInterval = ${value from `KC_JWKS_REFRESH_INTERVAL`};
private final int forcedJwksRefreshInterval = ${value from `KC_FORCED_JWKS_REFRESH_INTERVAL`};

var jwtAuthContextInfo = new JWTAuthContextInfo(issuerUri + "/protocol/openid-connect/certs", issuerUri);
jwtAuthContextInfo.setForcedJwksRefreshInterval(this.jwksRefreshInterval);
jwtAuthContextInfo.setJwksRefreshInterval(this.forcedJwksRefreshInterval);
var jwtParser = new DefaultJWTParser(jwtAuthContextInfo);
tokenParsers.put(issuerUri, jwtParser);
return jwtParser;

 User: The user initiates the process by logging into Keycloak.

1. Keycloak (keycloak): Keycloak authenticates the user and returns a JWT token with a new signing key ID.

2. User to Sidecar: The user requests "Module A" via the sidecar, providing the JWT token.

3. Sidecar Processing:

   • The sidecar checks the signing key ID in the JWT token.

   • If the signing key ID is unknown and the last JWKS refresh was over an hour ago, the sidecar fetches the new signing keys from Keycloak and updates its local cache. It then forwards the user's request to "Module A".

   • If the signing key ID is known, or if a JWKS refresh occurred within the last hour, the sidecar forwards the user's request to "Module A" without fetching new keys.

   • If the signing key ID is unknown and a JWKS refresh occurred less than one hour ago, the sidecar rejects the user's request to avoid too frequent key updates.

4. Module A (moduleA): "Module A" receives and processes the request forwarded by the sidecar.

Sidecar communication example

The example is based on the 2 simple modules in app-platform-minimal - mod-notes and mod-users, when the user can create a note and mod-notes will retrieve metadata from mod-users

@startuml

title "Create a note (positive flow)"

autonumber
skinparam responseMessageBelowArrow true

participant "Sidecar (mod-notes)" as scNotes #5CCCCC
participant "Sidecar (mod-users)" as scUsers #5CCCCC

participant Keycloak              as keycloak #FFD073
participant "mod-users"           as modUsers #876ED7
database    "mod-users-db"        as usersDb #006363

participant "mod-notes"           as modNotes #876ED7
database    "mod-notes-db"        as notesDb #006363


[-> scNotes: Create a note\n//POST /notes//

activate scNotes #5CCCCC
scNotes -> scNotes: Resolve incoming request as ingress request
note left
  //If route is not found in ingress routes cache -//
  //it will be considered as egress call//
end note

scNotes -> scNotes: Verify that request is not a self-request
scNotes -> scNotes: Skip parsing of system JWT,\nbecause request does not contain it
note left
  //System JWT used as a backup option for//
  //all module-to-module requests without specified system user//
  //in module descriptor//
end note
scNotes -> scNotes: Parse user JWT\n//(validation also performed at this stage)//
alt JWKs are not cached by ""kid"" from JWT
  scNotes -> keycloak: get jwk certificates\n//GET /realms/{tenantId}/protocol/openid-connect/certs//
  activate keycloak #FFD073

  return jwk certificates
end

alt cross-tenant requests disabled
  scNotes -> scNotes: validate that ""X-Okapi-Tenant"" header value\nmatches with JWT issuer tenant
end

scNotes -> scNotes: validate that ""X-Okapi-Tenant""\nheader value is in enabled tenants cache
scNotes -> scNotes: Skip impersonation filter,\nbecause it's not required

alt required permission cache entry is missing by key\n""notes.item.post#{tenantId}#{userId}#{sessionId}#{expirationTime}""
  scNotes -> keycloak: Authorize JWT using token endpoint\n//POST /realms/{tenantId}/protocol/openid-connect/certs//
  note left
  Request body parameters:
    //- grant_type:urn: ietf:params:oauth:grant-type:uma-ticket//
    //- permission: /notes#POST//
    //- audience: {tenantId}-login-application//
  //Request body headers://
    //- Authorization: Bearer {userJwt}//
  end note
  activate keycloak #FFD073
  return RPT token
  scNotes -> scNotes: Cache user authorization data with key:\n//notes.item.post#{tenantId}#{userId}#{sessionId}#{expirationTime}//
else cache value found
  scNotes -> scNotes: Verify that request authorized using cache value
end

scNotes -> scNotes: Add self-request signature
scNotes -> scNotes: Skip desired permission population,\nbecause routing entry does not contain them

scNotes -> modNotes: Forward request: //POST /notes//
note left
//Request will contain the following headers://
 //- X-Okapi-User-Id: {userId}//
 //- X-Okapi-Request-Id: {sidecarNotesGeneratedRequestId}//
 //- X-Okapi-Url: {resolvedNotesSidecarUrl}//
 //- X-Okapi-Token: {userJwt}//
 //- X-Okapi-Sidecar-Signature: {sidecarNotesSelfRequestSignature}//
end note

activate modNotes #876ED7
|||
modNotes -> scNotes: Get user by id: //GET /users/{id}//
activate scNotes #876ED7

scNotes -> scNotes: Resolve incoming request\nas egress request (module-to-module)
scNotes -> scNotes: Remove self-request signature\nbecause it is not a self-request
scNotes -> scNotes: Resolve sidecar location by module id
note left
  //Information comes from ""mgr-applications""//
  //from bootstrap information//
end note

alt system JWT is not cached for tenant
  scNotes -> keycloak: Get system JWT\n//POST /realms/{{tenant}}/protocol/openid-connect/token//
  activate keycloak #FFD073
  note left
    //Request body parameters://
    //  client_id: {client_id}//
    //  client_secret: {client_secret}//
    //  grant_type: client_credentials//
  
    //System JWT have access to all endpoint in Folio platform//
  end note
  return JWT
  scNotes -> scNotes: Cache system JWT for ""tenantId""
  else system JWT is cached
  scNotes -> scNotes: Populate System JWT from cache
end

scNotes -> scUsers: Forward request: //GET /users/{id}//
note left
//Request will contain the following headers://
 //- X-Okapi-User-Id: {userId}//
 //- X-Okapi-Module-Id: mod-users-{modUsersVersion}//
 //- X-Okapi-Request-Id: {scNotesRequestId}//
 //- X-Okapi-Token: {userJwt}//
 //- X-System-Token: {systemJwt}//
end note

activate scUsers #5CCCCC
  scUsers -> scUsers: Resolve incoming request as ingress request
  note left
    //If route is not found in ingress routes cache -//
    //it will be considered as egress call//
  end note
  
  scUsers -> scUsers: Verify that request is not a self-request
  scUsers -> scUsers: Verify that request does not\ncontain system-generated JWT
  note left
    //System JWT used as a backup option for//
    //all module-to-module requests without specified system user//
    //in module descriptor//
  end note
  scUsers -> scUsers: Parse user JWT\n//(validation also performed at this stage)//
  
  note left
    //""user_id"" is extracted from JWT, if token is parsed successfully//
    //it will be populated only if initial request does not contain ""X-Okapi-User-Id"" in headers//
  end note
  alt JWKs are not cached by ""kid"" from JWT
    scUsers -> keycloak: get jwk certificates\n//GET /realms/{tenantId}/protocol/openid-connect/certs//
    activate keycloak #FFD073
  
    return jwk certificates
  end
  
  alt cross-tenant requests disabled
    scUsers -> scUsers: validate that ""X-Okapi-Tenant"" header value\nmatches with JWT issuer tenant
  end
  
  scUsers -> scUsers: validate that ""X-Okapi-Tenant""\nheader value is in enabled tenants cache
  scUsers -> scUsers: Skip impersonation filter,\nbecause it's not required
  
  alt required permission cache entry is missing by key\n""users.item.get#{tenantId}#{userId}#{sessionId}#{expirationTime}""
    scUsers -> keycloak: Authorize User JWT using token endpoint\n//POST /realms/{tenantId}/protocol/openid-connect/certs//
    note left
    Request body parameters:
      //- grant_type:urn: ietf:params:oauth:grant-type:uma-ticket//
      //- permission: /users/{id}#GET//
      //- audience: {tenantId}-login-application//
    //Request body headers://
      //- Authorization: Bearer {userJwt}//
    end note
    activate keycloak #FFD073
    return 403 Forbidden
    scUsers -> scUsers: Cache user authorization data with key:\n//users.item.get#{tenantId}#{userId}#{sessionId}#{expirationTime}//
  else cache value found
    scUsers -> scUsers: Verify that request authorized using cache value
  end
  
  alt user JWT is not authorized
    alt required permission cache entry is missing by key\n""users.item.get#{tenantId}#{sessionId}#{expirationTime}""
      scUsers -> keycloak: Authorize System JWT using token endpoint\n//POST /realms/{tenantId}/protocol/openid-connect/certs//
        activate keycloak #FFD073
        note left
        Request body parameters:
          //- grant_type:urn: ietf:params:oauth:grant-type:uma-ticket//
          //- permission: /users/{id}#GET//
          //- audience: {tenantId}-login-application//
        //Request body headers://
          //- Authorization: Bearer {systemJwt}//
        end note
        return RPT
      scUsers -> scUsers: Cache user authorization data with key:\n//users.item.get#{tenantId}#{sessionId}#{expirationTime}//
    else cache value found
      scUsers -> scUsers: Verify that request authorized using cache value
    end
  end
  
  scUsers -> scUsers: Add self-request signature
  scUsers -> scUsers: Skip desired permission population,\nbecause routing entry does not contain them
  scUsers -> modUsers: Forward request: //GET /users/{id}//
  note left
  //Request will contain the following headers://
   //- X-Okapi-User-Id: {userId}//
   //- X-Okapi-Request-Id: {scNotesRequestId}/{scUsersRequestId}//
   //- X-Okapi-Url: {resolvedUsersSidecarUrl}//
   //- X-Okapi-Token: {userJwt}//
   //- X-Okapi-Sidecar-Signature: {sidecarUsersSelfRequestSignature}//
  end note
  activate modUsers #876ED7
  
  modUsers -> usersDb: Find user by id
    activate usersDb #006363
    return User by id
  
  return User by id
return User by id

|||
return User by id
modNotes -> notesDb: Save a note entity

activate notesDb #006363
return Saved note

return Created note
return Created note

@enduml

Deployment

folio-module-sidecar allows the building of a Docker container using OpenJDK base container or native image.

• Building And Running JVM Based docker container

• Building and Running Native docker container

Java-based Docker image

Native Docker Image

To be implemennted