Kubernetes Example Deployment

Overview

Setting Up the Environment

Prerequisites:

• Kubernetes Cluster (system for automating deployment, scaling, and management of containerized applications)

• PostgreSQL (RDBMS used by Keycloak, Kong Gateway, Eureka modules)

• Apache Kafka (distributed event streaming platform)

• HashiCorp Vault (identity-based secret and encryption management system)

• Keycloak (Identity and Access Management)

• Kong Gateway (API Gateway)

• MinIO (Enterprise Object Store is built for production environments, OPTIONAL)

• Elasticsearch or OpenSearch(enterprise-grade search and observability suite)

To set up Eureka Platform you should already have Kubernetes Cluster installed. Then just create a new Namespace within K8s Cluster to assign and manage resources granularity for your Eureka deployment.

You can have your cluster nodes on premise in local data center or adopt any cloud provider (i.e. AWS, Azure, GCP and so on) most suitable for you to meet planned or not planned resource demand.

Eureka Platform depends on a bunch of 3rd party services (listed above) for its expected operation. Some of these services (PostgreSQL, Apache Kafka, OpenSearch, Hashicorp Vault) can be deployed as standalone servces outside of cluster namespace but others mostly never depoloyed outside.

For initial Eureka deployment you will need about 30Gb of RAM. Such setup incorporates all mentioned 3rd party services in one kubernetes namespace.

It may require some extra resources (RAM, CPU, HDD Disk Space, HDD IOPS) to be assigned to destination Kubernetes Cluster in case prerequisites services are deployed in to the same cluster namespace.

Also in case you are going to have Consortia deployment it also needs extra resources to be assigned.

In case you make decision to have everything in one place please pay attention for HDD IOPS required by PostgreSQL/OpenSearch/ApacheKafka services.

PostgreSQL RDBMS should be installed to cluster namespace first since its the prerequisite for Kong Gateway and Keycloak Identity Manager.

Apache Kafka service is used by Eureka for internal communication between modules and very important to keep it in a good shape.

HashiCorp Vault stores all secrets used within Platform. AWS SSM Parameters are also supported as secrets' storage now.

Keycloak service provides authentication and authorization (granting access) for any kind of identities (users, roles, endpoints).

Kong Gateway as API Gateway routes requests to modules and provides access to Eureka REST APIs.

MinIO object storage keeps data for some modules to be used during platform operation.

Elasticsearch instance contains huge amount of information and indexes it for a fast search. It is very important to look after appropriate level of performance for this service. Also can be installed outside of Kubernetes Cluster.

Cluster setup

Lets assume you are going to set up Eureka Platform development environment on Kubernetes Cluster. To meet resource scalability ease during workload spikes it worth to use Cloud Services like EKS (AWS), AKS (Azure), GKE (GCP).

In the same time to control cloud vendor lock and cut down expences we are going to deploy all prerequisite services into the one cluster namespace except OpenSearch instance :)

To deploy prerequisite services we would recommend to adopt following Container (Docker) Images and Helm Charts:

PostgreSQL container Image: hub.docker.com/bitnami/postgresql , Helm Chart: github.com/bitnami/charts/postgresql

Apache Kafka container Image: hub.docker.com/bitnami/kafka, Helm Chart: github.com/bitnami/charts/kafka

Hashicorp Vault container Image: hub.docker.com/bitnami/kafka, Helm Chart: github.com/bitnami/charts/vault

Keycloak container Image: hub.docker.com/folioci/folio-keycloak, Helm Chart: github.com/bitnami/charts/keycloak, values for values.yaml: github.com/folio-org/pipelines-shared-library/…/keycloak.tf, Git Repository github.com/folio-org/folio-keycloak

Kong Gateway container Image: hub.docker.com/folioci/folio-kong, Helm Chart: charts/bitnami/kong, Git Repository github.com/folio-org/folio-kong

MinIO container Image: hub.docker.com/bitnami/minio Helm Chart: github.com/bitnami/charts/minio

Also we need to have Module Descriptors Registry to be in place.

Module Descriptors Registry service (MDR) represents HTTP Server that configured in Kubernetes Pod.

NOTE: A formal MDR is not a strict requirement. Any HTTP server will suffice, as long as the module descriptors are accessible from the mgr-applications service. For example, you could choose to host module descriptors in an S3 bucket configured as a static website using Amazon S3, or point directly to files in a GitHub repository, or setup an apache HTTP server, or even develop something custom.

This HTTP Server holds and distributes Modules Descriptors for Eureka Instance install and upgrade.

Module descriptor (see Module Descriptor Template) is generated during Continues Integration Flow and is put to Modules Descriptor Registry on finish.

These modules descriptors are used by Eureka install and update flows.

Deploying EUREKA on Kubernetes

Once all Prerequisites are met we can proceed with mgr-* Eureka modules deployment to cluster namespace:

• mgr-applications module:

  • Github Repository folio-org/mgr-applications

  • Container Image folioci/mgr-applications

  • Helm Chart charts/mgr-applications

  • Helm Chart variable values (./values/mgr-applications.yaml file below):

    mgr-applications: extraEnvVars: - name: MODULE_URL value: "http://mgr-applications" - name: FOLIO_CLIENT_CONNECT_TIMEOUT value: "600s" - name: FOLIO_CLIENT_READ_TIMEOUT value: "600s" - name: KONG_CONNECT_TIMEOUT value: "941241418" - name: KONG_READ_TIMEOUT value: "941241418" - name: KONG_WRITE_TIMEOUT value: "941241418" extraJavaOpts: - "-Dlogging.level.root=DEBUG -Dsecure_store=AwsSsm -Dsecure_store_props=/usr/ms/aws_ss.properties" integrations: db: enabled: true existingSecret: db-credentials kafka: enabled: true existingSecret: kafka-credentials replicaCount: 1 resources: limits: memory: 2Gi requests: memory: 1Gi

• mgr-tenant-entitlements module:

  • Github Repository folio-org/mgr-tenant-entitlements

  • Container Image folioci/mgr-tenant-entitlements

  • Helm Chart charts/mgr-tenant-entitlements

  • Helm Chart variable values(./values/mgr-tenant-entitlements.yaml file below):

    mgr-tenant-entitlements: extraJavaOpts: - "-Dlogging.level.root=DEBUG -Dsecure_store=AwsSsm -Dsecure_store_props=/usr/ms/aws_ss.properties" extraEnvVars: - name: MODULE_URL value: "http://mgr-tenant-entitlements" - name: FOLIO_CLIENT_CONNECT_TIMEOUT value: "600s" - name: FOLIO_CLIENT_READ_TIMEOUT value: "600s" - name: KONG_CONNECT_TIMEOUT value: "941241418" - name: KONG_READ_TIMEOUT value: "941241418" - name: KONG_WRITE_TIMEOUT value: "941241418" integrations: db: enabled: true existingSecret: db-credentials kafka: enabled: true existingSecret: kafka-credentials replicaCount: 1 resources: limits: memory: 2Gi requests: memory: 1Gi

• mgr-tenants module:

  • Github Repository folio-org/mgr-tenants

  • Container Image folioci/mgr-tenants

  • Helm Chart charts/mgr-tenants

  • Helm Chart variable values(./values/mgr-tenants.yaml file below):

    mgr-tenants: extraEnvVars: - name: MODULE_URL value: "http://mgr-tenants" - name: FOLIO_CLIENT_CONNECT_TIMEOUT value: "600s" - name: FOLIO_CLIENT_READ_TIMEOUT value: "600s" - name: KONG_CONNECT_TIMEOUT value: "941241418" - name: KONG_READ_TIMEOUT value: "941241418" - name: KONG_WRITE_TIMEOUT value: "941241418" extraJavaOpts: - "-Dlogging.level.root=DEBUG -Dsecure_store=AwsSsm -Dsecure_store_props=/usr/ms/aws_ss.properties" integrations: db: enabled: true existingSecret: db-credentials replicaCount: 1 resources: limits: memory: 2Gi requests: memory: 1Gi

     

Deploy mgr-* applications to Kubernetes Cluster:

Eureka deployment flow:

Get Master Auth token from Keycloak.

To run administrative REST API requests against Eureka Instance we need to get Master Access Token from Keycloak on the start.

We need to know request parameters first (consider adopting following example)

• Keycloak FQDN: keycloak.example.org

• Token Service Endpoint: /realms/master/protocol/openid-connect/token

• Client ID: folio-backend-admin-client (this is expected value and should not be changed)

• Client Secret: SecretPhrase

• Grant Type: client_credentials (Constant)

We need to save returned Master Access Token to run within any administrative REST API call later.

Register Applications Descriptors:

REST API Docs for "POST /applications" endpoint.

We need to register application descriptor in Eureka instance. Application descriptor is created from github.com/folio-org/app-platform-full/sprint-quesnelia/app-platform-full.template.json file taken from release branch.

Docs for registerApplication Rest API call - register a new application.

Descriptor is registered with CURL command and related parameters:

• Kong Gateway FQDN (http header): kong.example.org

• Auth token (http header): 'Authorization: Bearer...'

• Application Descriptor (http request body): JSON data file

Register Modules

REST API Docs for “GET /modules/discovery“ endpoint.

Once required Applications Descriptors are registered in instance we proceed with Module Discovery Flow to register modules in system.

Docs for searchModuleDiscovery Rest API call - Retrieving module discovery information by CQL query and pagination parameters.

Modules Discovery is started with CURL command and related parameters:

• Kong Gateway FQDN (HTTP header): kong.example.org

• Auth token (http header): 'Authorization: Bearer...'

• Module Discovery Info (http request body): JSON data file

Deploy Backend Modules

Now we are ready to deploy backend modules to Kubernetes Namespace with Eureka instance.

Helm Charts for modules are taken from Github repository folio-org/folio-helm-v2

Variable values for helm charts are stored in dedicated repository folder folio-org/pipelines-shared-library/resources/helm

For exmaple:

Create tenant

REST API Docs for “POST /tenants“ endpoint.

At this point we are ready to create application tenant in Eureka instance.

First we need to take a look on docs for createTenant Rest API call to create a new tenant.

Once we sure about required parameters we give Post HTTP request to create a new tenant.

In our example we create tenant with name “diku” and description “magic happens here”:

Set entitlement

REST API Docs for “POST /entitlements“ endpoint.

We have application tenant created so we can entitle registered applications to our tenant.

In other words we enable application(s) to tenant.

As usual we take a look on docs for create Rest API call to Install/enable application for tenant.

From mentioned docs we can get some info about passed parameters and returned value.

Following example shows how to enable application for our tenant without problem:

Add User

REST API Docs for “POST /users-keycloak/users“ endpoint.

On this stage we are ready to add first User to Eureka Instance to have administrative privileges later.

So checking parameters in docs for createUser Rest API call to create a new user.

Then use CURL command to run POST HTTP request against Eureka Instance:

Set User Password

REST API Docs for “POST /authn/credentials“ endpoint.

Having our user created we are free to assign him some secret password to use it on login.

Just carefully looking through docs for createCredentials Rest API call to add a new login to the system.

Create Role

REST API Docs for “POST /roles“ endpoint.

We need to create a Role to bundle Eureka administrative capabilities with our Admin User.

So accordingly to docs for createRole Rest API call to create a new role we need to run following POST HTTP reuest:

Assign Capabilities to Role

REST API Docs for “POST /roles/capabilities“ endpoint.

And then we just attach required Eureka application capabilities to our Admin Role

Using docs for createRoleCapabilities Rest API call we create a new record associating one or more capabilities with the already created role

To get a list of existing Capabilities we are going to use findCapabilities Rest API call

Add Roles to User

REST API Docs for “POST /roles/users“ endpoint.

The last step in the row is assigning Admin Role to Admin User to provide him Super Power to rule Eureka world.

So accordingly to existing docs for assignRolesToUser Rest API call to create a record associating role with user we should run CURL command like the next one:

Deploy Edge modules

• Render Ephemeral Properties

• Create config map for every edge-* module

• Deploy edge-* modules to cluster namespace

Perform Consortia Deployment (if required)

REST API Docs for “POST /consortia“ endpoint.

• Set up a Consortia Deployment with the given tenants

  • Create consortia deployment instance accordingly to docs for consortia REST API call to save consortium configuration.

    curl --location 'https://kong.example.org/consortia' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: consortium' \ --data '{ "name": "consortium", "id": "[Consortium-UUID]" }'

  • Add Consortia Central Tenant

    curl --location 'https://kong.example.org/consortia/[Consortium-UUID]/tenants' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: consortium' \ --data '{ "id": "consortium", "name": "Central office", "code": "MCO", "isCentral": true }'

  • Add Consotia Institutial Tenant

    curl --location 'https://kong.example.org/consortia/[Consortium-UUID]/tenants?adminUserId=[Admin-User-UUID]' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: consortium' \ --data '{ "id": "college", "name": "college", "code": "COL", "isCentral": false }'

Perform indexing on Eureka resources

There is comprehensive documentation piece for Search Indexing we would highly recommend to walk through to learn that magic closer.

• Re-create search index for authority resource

  • Have a look into existing docs for Resource reindex REST API call to initiate reindex for the authority records (/search/index/inventory/reindex endpoint)

    curl --location 'https://kong.example.org/search/index/inventory/reindex' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: diku' \ --data '{ "recreateIndex": true, "resourceName": "authority" }'

  • Monitoring reindex process

    • It is possible to monitor indexing process with getReindexJob REST API call. To check how many records are published to Kafka topic we may use following command

      curl --location 'https://kong.example.org/authority-storage/reindex/[reindex_job_id]' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'x-okapi-tenant: diku'

• Indexing of instance resources

  • First need to check related docs for Full Inventory Records reindex REST API call to initiate the full reindex for the inventory instance records ( /search/index/instance-records/reindex/full endpoint)

    curl --location 'https://kong.example.org/search/index/instance-records/reindex/full' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: diku' \ --data '{}'

Configure Edge modules

• Create Eureka Users for Eureka UI

  • UI modules expect respcective Users created in Eureka instance. Enough system capabilites have to be assigned to UI Users to perfrom desired level of access.

  • To have some clue how UI modules are mapped with Eureka Accounts with required capabilities please take a look into folio-org/pipelines-shared-library/resources/edge/config_eureka.yaml file

  • So we need to create extra Eureka Accounts to be used by UI Modules. For example

    • Create User Account:

      curl --location 'https://kong.example.org/users-keycloak/users' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: diku' \ --data-raw '{ "username": "admin" , "active": true , "patronGroup": "3684a786-6671-4268-8ed0-9db82ebca60b" , "type": "staff" , "personal": { "firstName": "John" , "lastName": "Doe" , "email": "noreply@ci.folio.org" , "preferredContactTypeId": "002" } }'

    • Set Password for User:

      curl --location 'https://kong.example.org/authn/credentials' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: diku' \ --data '{ "username": "admin", "userId": "[Admin-User-UUID]", "password": "SecretPhrase" }'

    • Assign Capabilities to User Account:

      curl --location 'https://kong.example.org/users/capabilities' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: diku' \ --data '{ "userId": "[User-UUID]", "capabilityIds": [ "[Eureka-Capability-01-UUID]", "[Eureka-Capability-02-UUID]", "[Eureka-Capability-03-UUID]", "[Eureka-Capability-04-UUID]", "[Eureka-Capability-05-UUID]" ] }'

    • Assign Capabilities Set to User Account

      curl --location 'https://kong.example.org/users/capability-sets?limit=5000' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: diku' \ --data '{ "userId": "[User-UUID]", "capabilityIds": [ "[Eureka-Capability-Set-01-UUID]", "[Eureka-Capability-Set-02-UUID]", "[Eureka-Capability-Set-03-UUID]" ] }'

Build FOLIO Eureka UI images

• Please obtain the source code for the Eureka UI from the GitHub repository titled "folio-org/platform-complete" on the snapshot branch. Kindly modify the configuration template file located at platform-complete/eureka-tpl/stripes.config.js in accordance with the existing values. To construct the Eureka UI (Stripes Platform), please execute a series of commands using Yarn. For building the Eureka UI (Stripes Platform) within a Container Image, it suffices to utilize the following Dockerfile available at http://github.com/folio-org/platform-complete/docker/Dockerfile.

• Please obtain the source code for the Eureka UI from the GitHub repository folio-org/platform-complete on the snapshot branch.

  $ git clone -b snapshot https://github.com/folio-org/platform-complete.git

• Kindly modify the configuration template file located at platform-complete/eureka-tpl/stripes.config.js in accordance with the existing values.

• Add to the platform-complete/package.json dependencies section the following modules

  "@folio/authorization-policies": ">=1.0.0", "@folio/authorization-roles": ">=1.0.0", "@folio/plugin-select-application": ">=1.0.0"

• To build Eureka UI (Stripes Platform), please execute a series of commands using Yarn

  $ yarn config set @folio:registry https://repository.folio.org/repository/npm-folioci/ $ yarn install

• For building the Eureka UI (Stripes Platform) within a Container Image, it suffices to utilize the following Dockerfile located at github.com/folio-org/platform-complete/docker/Dockerfile

  FROM node:18-alpine as stripes_build ARG OKAPI_URL=http://localhost:9130 ARG TENANT_ID=diku ARG CXXFLAGS="-std=c++17" RUN mkdir -p /etc/folio/stripes WORKDIR /etc/folio/stripes COPY . /etc/folio/stripes/ RUN apk upgrade \ && apk add \ alpine-sdk \ python3 \ && rm -rf /var/cache/apk/* RUN yarn config set python /usr/bin/python3 RUN yarn config set @folio:registry https://repository.folio.org/repository/npm-folioci/ RUN yarn install RUN yarn build-module-descriptors RUN yarn build output --okapi $OKAPI_URL --tenant $TENANT_ID # nginx stage FROM nginx:stable-alpine # Install latest patch versions of packages: https://pythonspeed.com/articles/security-updates-in-docker/ RUN apk upgrade --no-cache EXPOSE 80 COPY --from=stripes_build /etc/folio/stripes/output /usr/share/nginx/html COPY --from=stripes_build /etc/folio/stripes/yarn.lock /usr/share/nginx/html/yarn.lock COPY docker/nginx.conf /etc/nginx/conf.d/default.conf COPY docker/entrypoint.sh /usr/bin/entrypoint.sh ENTRYPOINT ["/usr/bin/entrypoint.sh"]

  After creating a Docker Image with the Eureka UI, it should be stored in a Container Image Repository to be accessible for deployment to the Kubernetes Cluster.

Deploy Eureka UI

• Deploy 'ui-bundle' module

  • To deploy ui-bundle image feel free to utilize folio-helm-v2/platform-complete Helm Chart.

  • Also, variable module properties can be seen in github.com/folio-org/pipelines-shared-library/resources/helm/testing.yaml and given in values/ui-bundle-diku.yaml file

    helm repo add folio-helm-v2 https://repository.folio.org/repository/folio-helm-v2/ helm repo update folio-helm-v2 helm install diku-ui-bundle --namespace=eureka -f ./values/ui-bundle-diku.yaml folio-helm-v2/platform-complete

• Configure Eureka UI parameters

  • Get Tenant Realm Name from Keycloak

    curl -X GET --location 'https://keycloak.example.org/admin/realms/[YOUR_TENANT_NAME]/clients?clientId=[YOUR_TENANT_NAME]-application' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json'

  • Put extra Tenant Configuration for Eureka UI (Stripes Platform)

    curl -X POST --location 'https://keycloak.example.org/admin/realms/[YOUR_TENANT_NAME]/clients/[YOUR_TENANT_REALM_NAME]' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --data '{ "rootUrl": "[YOUR_TENANT_URL]", "baseUrl": "[YOUR_TENANT_URL]", "adminUrl": "[YOUR_TENANT_URL]", "redirectUris": ["[YOUR_TENANT_URL]/*", "http://localhost:3000/*"], "webOrigins": ["/*"], "authorizationServicesEnabled": true, "serviceAccountsEnabled": true, "attributes": ['post.logout.redirect.uris': "/*##[YOUR_TENANT_URL]/*", login_theme: 'custom-theme'] }'

Post-Deployment Tasks

Troubleshooting and Common Issues

• InternalServerErrorException error 500. Connection refused - lack of resources.

• InternalServerErrorException error 500. Connection refused - deployment timing.

• The application is not entitled on tenant - sidecar vs Kafka
  Issue: The error The module is not entitled on tenant ... may occur during certain operations, especially during the entitlement process. You can find the full log of this issue in the related module's sidecar.
  Cause: This error happens due to communication issues between mgr-tenant-entitlement and the corresponding module, which notifies the module about the end of the entitlement process via Kafka. In some cases, the sidecar consumer connection could be marked as dead. The main reason for that is a wrong setting of Kafka heartbeat request and sidecar poll requests, which aren’t aligned with each other, or various networking issues that lead to the poll request being absent during the specific period. The module is not entitled on tenant ... errors appear due to the next module portion in the entitlement process request to the previously entitled modules and sidecar which does not receive a message via Kafka about the finish of the related module entitlement process.
  Resolution: In case a sidecar loses connection with Kafka during the entitlement process and it has not been affected by this issue, simply restart the affected module, and its sidecar will get entitlement information from the mgr-tenant-entitlement module. In case the entitlement process fails, repeat it. Nevertheless ensures a stable connection between Kafka and sidecars as well as aligns Kafka heartbeat request and sidecar poll request periods.

• The application is not entitled on tenant - sidecar vs mod-tenant-entitlement
  Issue: The error The module is not entitled on tenant ... may occur during certain operations.
  Cause: This error can happen when the mgr-tenant-entitlement and some module pods are redeployed simultaneously, and the module's sidecar becomes ready before mgr-tenant-entitlement, causing it to be unable to obtain information about the application entitlement from the MTE module and potentially providing an error response to other modules upon request.
  Resolution: To fix this issue, ensure the correct module redeployment order. If the issue occurs unexpectedly, simply restart the affected module.

• The upstream server is timing out - Kong fine-tuning
  Issue: Some API requests may result in a 504 error code with the error message "upstream server is timing out.” This problem is primarily caused by Kong and typically occurs during long operations, such as assigning a capability to a role or user.
  Cause: When a request reaches a specific module, it always goes through Kong, which has two potential points of failure: Kong's Nginx and Kong itself.
  Resolution: To address this issue, you should adjust the upstream timeout of Kong's Nginx using the KONG_NGINX_HTTP_KEEPALIVE_TIMEOUT, KONG_NGINX_UPSTREAM_KEEPALIVE, and KONG_NGINX_HTTP_KEEPALIVE_REQUESTS environment variables. Additionally, consider modifying the following Kong variables: KONG_UPSTREAM_KEEPALIVE_IDLE_TIMEOUT, KONG_UPSTREAM_KEEPALIVE_POOL_SIZE, and KONG_UPSTREAM_KEEPALIVE_MAX_REQUESTS, KONG_UPSTREAM_CONNECT_TIMEOUT, KONG_RETRIES. More information about how to perform that.

• Some capabilities/capability sets are absent - Kafka messages processing period
  Issue: If you try to assign capabilities or capability sets to a role or user immediately after the entitlement process, you may encounter an issue with their absences.
  Cause: The predefined capabilities (also known as permissions) and capability sets are created just after the application entitlement process. This process takes time to complete. Here's how it works:

  • The mgr-tenant-entitlement module sends messages to the mod-roles-keycloak via Kafka with a list of roles. This process lasts during the entitlement process as each module is enabled on a tenant.

  • The mod-roles-keycloak starts processing the messages right after it has been entitled, so it could be at the end of the entitlement process.

  • mod-roles-keycloak proceeds through the message queue in Kafka until it reaches the end.