Kubernetes Example Deployment

Overview

Setting Up the Environment

Prerequisites:

• Kubernetes Cluster (system for automating deployment, scaling, and management of containerized applications)

• PostgreSQL (RDBMS used by Keycloak, Kong Gateway, Eureka modules)

• Apache Kafka (distributed event streaming platform)

• HashiCorp Vault (identity-based secret and encryption management system)

• Keycloak (Identity and Access Management)

• Kong Gateway (API Gateway)

• MinIO (Enterprise Object Store is built for production environments, OPTIONAL)

• Elasticsearch or OpenSearch(enterprise-grade search and observability suite)

To set up Eureka Platform you should already have Kubernetes Cluster installed. Then just create a new Namespace within K8s Cluster to assign and manage resources granularity for your Eureka deployment.

You can have your cluster nodes on premise in local data center or adopt any cloud provider (i.e. AWS, Azure, GCP and so on) most suitable for you to meet planned or not planned resource demand.

Eureka Platform depends on a bunch of 3rd party services (listed above) for its expected operation. Some of these services (PostgreSQL, Apache Kafka, OpenSearch, Hashicorp Vault) can be deployed as standalone servces outside of cluster namespace but others mostly never depoloyed outside.

For initial Eureka deployment you will need about 30Gb of RAM. Such setup incorporates all mentioned 3rd party services in one kubernetes namespace.

It may require some extra resources (RAM, CPU, HDD Disk Space, HDD IOPS) to be assigned to destination Kubernetes Cluster in case prerequisites services are deployed in to the same cluster namespace.

Also in case you are going to have Consortia deployment it also needs extra resources to be assigned.

In case you make decision to have everything in one place please pay attention for HDD IOPS required by PostgreSQL/OpenSearch/ApacheKafka services.

PostgreSQL RDBMS should be installed to cluster namespace first since its the prerequisite for Kong Gateway and Keycloak Identity Manager.

Apache Kafka service is used by Eureka for internal communication between modules and very important to keep it in a good shape.

HashiCorp Vault stores all secrets used within Platform. AWS SSM Parameters are also supported as secrets' storage now.

Keycloak service provides authentication and authorization (granting access) for any kind of identities (users, roles, endpoints).

Kong Gateway as API Gateway routes requests to modules and provides access to Eureka REST APIs.

MinIO object storage keeps data for some modules to be used during platform operation.

Elasticsearch instance contains huge amount of information and indexes it for a fast search. It is very important to look after appropriate level of performance for this service. Also can be installed outside of Kubernetes Cluster.

Cluster setup

Lets assume you are going to set up Eureka Platform development environment on Kubernetes Cluster. To meet resource scalability ease during workload spikes it worth to use Cloud Services like EKS (AWS), AKS (Azure), GKE (GCP).

At the same time to control cloud vendor lock and cut down expences we are going to deploy all prerequisite services into the one cluster namespace except OpenSearch instance.

To deploy prerequisite services we would recommend to adopt following Container (Docker) Images and Helm Charts:

PostgreSQL container Image: hub.docker.com/bitnami/postgresql , Helm Chart: github.com/bitnami/charts/postgresql

Suggested PostgreSQL chart version: 13.2.19

P\S Please pay attention to this SQL initialization script (in “values.yml” file), if deployment approach differs these 2 DBs MUST BE pre-created and configured accordantly beforehand.

Apache Kafka container Image: hub.docker.com/bitnami/kafka, Helm Chart: github.com/bitnami/charts/kafka

Since Kafka is crucial part of Eureka platform, it’s highly recommended to make sure that Kafka is up & running, you’re able to create topics and push messages.

All of the above check could be done via Kafka scripts, usually located in the following location:

It’s also it worth to mention check zookeeper deployment status, make sure that both Kafka and Zookeeper are in up & running state.

Suggested Kafka chart version: 21.4.6

Hashicorp Vault container Image: hub.docker.com/bitnami/vault, Helm Chart: github.com/bitnami/charts/vault

Suggested Vault helm chart version: 0.28.0

P\S In case of AWS SSM parameters store, the following parameters MUST BE pre-created, manually or via some automation approach:
"folio-backend-admin-client", "master_mgr-applications", "master_mgr-tenant-entitlements", "master_mgr-tenants"

• folio-org/mgr-tenant-entitlements

• folio-org/mgr-tenants

• folio-org/mgr-applications

Open

*** On Folio Rancher Eureka envs, EC2 instance profile is being used for granting AWS_SSM permission.

*** the same is true for Vault setup, all of the above secrets MUST BE pre-created in the following path: {{folio}} → value of ENV environment variable (may differ, so please adjust according to your values)

Source code for AWS SSM Params store:

list of params: pipelines-shared-library/terraform/rancher/project/locals.tf at master · folio-org/pipelines-shared-library

creation: pipelines-shared-library/terraform/rancher/project/secrets.tf at master · folio-org/pipelines-shared-library

Keycloak container Image: folioci/folio-keycloak(snapshot versions), folioorg/folio-keycloak(release versions)

Helm Chart: github.com/bitnami/charts/keycloak, values for values.yaml: github.com/folio-org/pipelines-shared-library/…/keycloak.tf, Git Repository github.com/folio-org/folio-keycloak

Suggested Keycloak chart version: 21.0.4

P\S for the Keycloak deployment USE ONLY folio-keycloak docker image, default Keycloak image is missing additional configuration scripts.

Kong Gateway container Image: folioci/folio-kong(snapshot versions), folioorg/folio-kong(release versions)

Helm Chart: charts/bitnami/kong, Git Repository github.com/folio-org/folio-kong

Suggested Kong helm chart version: 12.0.11

MinIO container Image: hub.docker.com/bitnami/minio Helm Chart: github.com/bitnami/charts/minio

Suggested MinIO helm chart version: 11.8.1

OpenSearch container Image: OpenSearch/opensearch(latest versions), necessary for e.g. mod-search

Create the following secret “opensearch-credentials” before deploying and adjust as necessary for your setup:

Use the following values for the deployment. Adjust as necessary for your setup. Note the addition of the plugins section:

plugins:
enabled: true
installList:
- analysis-icu
- analysis-kuromoji
- analysis-nori
- analysis-phonetic
- analysis-smartcn

Also we need to have Module Descriptors Registry to be in place.

Module Descriptors Registry service (MDR) represents HTTP Server that configured in Kubernetes Pod.

NOTE: A formal MDR is not a strict requirement. Any HTTP server will suffice, as long as the module descriptors are accessible from the mgr-applications service. For example, you could choose to host module descriptors in an S3 bucket configured as a static website using Amazon S3, or point directly to files in a GitHub repository, or setup an Apache HTTP server, or even develop something custom.

This HTTP Server holds and distributes Modules Descriptors for Eureka Instance install and upgrade.

Module descriptor (see Module Descriptor Template) is generated during Continues Integration Flow and is put to Modules Descriptor Registry on finish.

These modules descriptors are used by Eureka install and update flows.

In case of Folio community snapshot\release docker images (Docker Hub) usage, this MDR: folio-registry.dev.folio.org/_/proxy/modules could be used for module descriptors pulling.

Deploying EUREKA on Kubernetes

Snapshot versions modules introduced by Eureka platform, can be found via platform-complete/eureka-platform.json at snapshot · folio-org/platform-complete

*****

CAUTION: as of 05/12/2025 ALL POST REST API calls to mgr-* endpoints should have(strongly suggested) Authorization token, GET API calls are working without authorization.

• mgr-applications API

• mgr-tenants API

• mgr-tenant-entitlements API

*****

To enable authorization feature, follow up instructions in [RANCHER-2180] Security-related env variables misconfigured for mgr-* in Eureka envs - FOLIO Jira

Once all Prerequisites are met, we can proceed with mgr-* Eureka modules deployment to cluster namespace:

• mgr-applications module:

  • Github Repository folio-org/mgr-applications

  • Container Image folioci/mgr-applications

  • Helm Chart charts/mgr-applications

  • Helm Chart variable values (EXAMPLE ONLY:./values/mgr-applications.yaml file, below values are subject to be adjusted according to your setup):

    mgr-applications: extraEnvVars: - name: MODULE_URL value: "http://mgr-applications" - name: FOLIO_CLIENT_CONNECT_TIMEOUT value: "600s" - name: FOLIO_CLIENT_READ_TIMEOUT value: "600s" - name: KONG_CONNECT_TIMEOUT value: "941241418" - name: KONG_READ_TIMEOUT value: "941241418" - name: KONG_WRITE_TIMEOUT value: "941241418" extraJavaOpts: - "-Dlogging.level.root=DEBUG -Dsecure_store=AwsSsm -Dsecure_store_props=/usr/ms/aws_ss.properties" integrations: db: enabled: true existingSecret: db-credentials kafka: enabled: true existingSecret: kafka-credentials replicaCount: 1 resources: limits: memory: 2Gi requests: memory: 1Gi

• mgr-tenant-entitlements module:

  • Github Repository folio-org/mgr-tenant-entitlements

  • Container Image folioci/mgr-tenant-entitlements

  • Helm Chart charts/mgr-tenant-entitlements

  • Helm Chart variable values(EXAMPLE ONLY:./values/mgr-tenant-entitlements.yaml file, below values are subject to be adjusted according to your setup):

    mgr-tenant-entitlements: extraJavaOpts: - "-Dlogging.level.root=DEBUG -Dsecure_store=AwsSsm -Dsecure_store_props=/usr/ms/aws_ss.properties" extraEnvVars: - name: MODULE_URL value: "http://mgr-tenant-entitlements" - name: FOLIO_CLIENT_CONNECT_TIMEOUT value: "600s" - name: FOLIO_CLIENT_READ_TIMEOUT value: "600s" - name: KONG_CONNECT_TIMEOUT value: "941241418" - name: KONG_READ_TIMEOUT value: "941241418" - name: KONG_WRITE_TIMEOUT value: "941241418" integrations: db: enabled: true existingSecret: db-credentials kafka: enabled: true existingSecret: kafka-credentials replicaCount: 1 resources: limits: memory: 2Gi requests: memory: 1Gi

• mgr-tenants module:

  • Github Repository folio-org/mgr-tenants

  • Container Image folioci/mgr-tenants

  • Helm Chart charts/mgr-tenants

  • Helm Chart variable values (EXAMPLE ONLY:./values/mgr-tenants.yaml file, below values are subject to be adjusted according to your setup):

    mgr-tenants: extraEnvVars: - name: MODULE_URL value: "http://mgr-tenants" - name: FOLIO_CLIENT_CONNECT_TIMEOUT value: "600s" - name: FOLIO_CLIENT_READ_TIMEOUT value: "600s" - name: KONG_CONNECT_TIMEOUT value: "941241418" - name: KONG_READ_TIMEOUT value: "941241418" - name: KONG_WRITE_TIMEOUT value: "941241418" extraJavaOpts: - "-Dlogging.level.root=DEBUG -Dsecure_store=AwsSsm -Dsecure_store_props=/usr/ms/aws_ss.properties" integrations: db: enabled: true existingSecret: db-credentials replicaCount: 1 resources: limits: memory: 2Gi requests: memory: 1Gi

     

Example of deploying mgr-* applications to Kubernetes Cluster (folio-helm-v2 is a private Helm repo, if you would like to gain an access to please reach out via Slack channel: https://open-libr-foundation.slack.com/archives/C017RFAGBK2, alternatively you may package charts on your side and use them for installation):

Eureka deployment flow:

Get Master Auth token from Keycloak.

To run administrative REST API requests against Eureka Instance we need to get Master Access Token from Keycloak on the start.

We need to know request parameters first (consider adopting following example)

• Keycloak FQDN: keycloak.example.org

• Token Service Endpoint: /realms/master/protocol/openid-connect/token

• Client ID: folio-backend-admin-client (this is expected value and should not be changed)

• Client Secret: SecretPhrase (Generated by the deployment example K8s secret reference keycloak-credentials.  E.g. the value usually used in reference envs isSecretPassword)

• Grant Type: client_credentials (Constant)

We need to save returned Master Access Token to run within any administrative REST API call later.

Register Applications Descriptors:

REST API Docs for "POST /applications" endpoint.

We need to register application descriptor in Eureka instance. Application descriptor is created from github.com/folio-org/app-platform-full/sprint-quesnelia/app-platform-full.template.json file taken from release branch.

Depending on release being delivered, application composition may differ, please ALWAYS refer to this page: FOLIO Eureka Applications - Releases - FOLIO Wiki for actual structure and the list of the applications.

If targeting the latest development version, please use snapshot branch in application’s repository, in case any release, pre-generated application descriptor is residing inside release tag (example):

Open

Each and every application descriptor MUST BE generated using folio-application-generator (for snapshot versions): folio-org/folio-application-generator: A Maven plugin to generate application descriptor from a template

Docs for registerApplication Rest API call - register a new application.

Descriptor is registered with CURL command and related parameters:

• Kong Gateway FQDN (http header): kong.example.org

• Auth token (http header): 'Authorization: Bearer...'

• Application Descriptor (http request body): JSON data file

Register Modules

REST API Docs for “POST /modules/discovery“ endpoint.

Once required Applications Descriptors are registered in instance we proceed with Module Discovery Flow to register modules in system.

Docs for searchModuleDiscovery Rest API call - Retrieving module discovery information by CQL query and pagination parameters.

Modules Discovery is started with CURL command and related parameters:

• Kong Gateway FQDN (HTTP header): kong.example.org

• Auth token (http header): 'Authorization: Bearer...'

• Module Discovery Info (http request body): JSON data file

Deploy Backend Modules

Now we are ready to deploy backend modules to Kubernetes Namespace with Eureka instance.

Helm Charts for modules are taken from Github repository folio-org/folio-helm-v2

Variable values for helm charts are stored in dedicated repository folder folio-org/pipelines-shared-library/resources/helm

For exmaple:

Create tenant

REST API Docs for “POST /tenants“ endpoint.

At this point we are ready to create application tenant in Eureka instance.

First we need to take a look on docs for createTenant Rest API call to create a new tenant.

Once we sure about required parameters we give Post HTTP request to create a new tenant.

In our example we create tenant with name “diku” and description “magic happens here”:

Set entitlement

REST API Docs for “POST /entitlements“ endpoint.

Before initiating the entitlement process, both the application and the tenant must be created or registered. This entitlement process can be understood as enabling an application for a specific tenant.
It is advisable to review the documentation related to entitlement or enabling applications for tenants. This documentation offers detailed information about the parameters required and the expected returned values.
The following example demonstrates how an application can be enabled for a tenant effectively:

Add User

REST API Docs for “POST /users-keycloak/users“ endpoint.

On this stage we are ready to add first User to Eureka Instance to have administrative privileges later.

So checking parameters in docs for createUser Rest API call to create a new user.

Then use CURL command to run POST HTTP request against Eureka Instance:

Set User Password

REST API Docs for “POST /authn/credentials“ endpoint.

Having our user created we are free to assign him some secret password to use it on login.

Just carefully looking through docs for createCredentials Rest API call to add a new login to the system.

Create Role

REST API Docs for “POST /roles“ endpoint.

We need to create a Role to bundle Eureka administrative capabilities with our Admin User.

So accordingly to docs for createRole Rest API call to create a new role we need to run following POST HTTP reuest:

Assign Capabilities to Role

REST API Docs for “POST /roles/capabilities“ endpoint.

And then we just attach required Eureka application capabilities to our Admin Role

Using docs for createRoleCapabilities Rest API call we create a new record associating one or more capabilities with the already created role

To get a list of existing Capabilities we are going to use findCapabilities Rest API call

Add Roles to User

REST API Docs for “POST /roles/users“ endpoint.

The last step in the row is assigning Admin Role to Admin User to provide him Super Power to rule Eureka world.

So accordingly to existing docs for assignRolesToUser Rest API call to create a record associating role with user we should run CURL command like the next one:

Deploy Edge modules

• Render Ephemeral Properties

• Create config map for every edge-* module

• Deploy edge-* modules to cluster namespace

P\S on Eureka based environments, edge-* modules should be pointed to Kong endpoint, below TF config snippet is used across all edge-* modules as source of additional env vars.

Perform Consortia Deployment (if required)

REST API Docs for “POST /consortia“ endpoint.

• Set up a Consortia Deployment with the given tenants

  • Create consortia deployment instance accordingly to docs for consortia REST API call to save consortium configuration.

    curl -X POST --location 'https://kong.example.org/consortia' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: consortium' \ --data '{ "name": "consortium", "id": "[Consortium-UUID]" }'

  • Add Consortia Central Tenant

    curl -X POST --location 'https://kong.example.org/consortia/[Consortium-UUID]/tenants' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: consortium' \ --data '{ "id": "consortium", "name": "Central office", "code": "MCO", "isCentral": true }'

  • Add Consotia Institutial Tenant

    curl -X POST --location 'https://kong.example.org/consortia/[Consortium-UUID]/tenants?adminUserId=[Admin-User-UUID]' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: consortium' \ --data '{ "id": "college", "name": "college", "code": "COL", "isCentral": false }'

Perform indexing on Eureka resources

There is comprehensive documentation piece for Search Indexing we would highly recommend to walk through to learn that magic closer.

• Re-create search index for authority resource

  • Have a look into existing docs for Resource reindex REST API call to initiate reindex for the authority records (/search/index/inventory/reindex endpoint)

    curl -X POST --location 'https://kong.example.org/search/index/inventory/reindex' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: diku' \ --data '{ "recreateIndex": true, "resourceName": "authority" }'

  • Monitoring reindex process

    • It is possible to monitor indexing process with getReindexJob REST API call. To check how many records are published to Kafka topic we may use following command

      curl -X GET --location 'https://kong.example.org/authority-storage/reindex/[reindex_job_id]' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'x-okapi-tenant: diku'

• Indexing of instance resources

  • First need to check related docs for Full Inventory Records reindex REST API call to initiate the full reindex for the inventory instance records ( /search/index/instance-records/reindex/full endpoint)

    curl -X POST --location 'https://kong.example.org/search/index/instance-records/reindex/full' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA...' \ --header 'Content-Type: application/json' \ --header 'x-okapi-tenant: diku' \ --data '{}'

Configure Edge modules

• Create Eureka Users for Eureka UI