Eureka Early Adopter Investigation

This documentation may provide some Curl script examples that are run as a Bash script.
The base variable represents the base URL with a trailing slash like: https://kong.folio.example/.
This URL should point to a Kong instance.
The path variable will be described for each particular step, which could be something like: applications (there is no leading slash).

The Application Descriptor should be used to describe a single application, however the Application Platform Minimal is a convenient way to provide all of the core applications via a single descriptors.
While this behavior may change in the future, the Application Platform Minimal is ideal for testing Eureka for the Early Adopter investigation.

The FOLIO Module Descriptor Registry operates using GitHub Actions via a locally hoster runner to generate pages hosted via GitHub Pages.
A series of Bash scripts are used as a minimalistic way to achieve Module Descriptor generation.

The Module Descriptors are being generated and deployed automatically using GitHub Actions.

Step 1: Build Application Descriptor has worked without any notable issues.
See the file for an example descriptor.

We later on ended up switching to a more explicitly selected set of project versions.

Please see the Selecting Versions for Investigation (Sunflower) for the recommended versions of projects and packages.

At this point the Keycloak, Kong, MGR Applications, MGR Tenants, and MGR Tenant Entitlements must be up and running as described in the Kubernetes, Rancher, and Fleet Steps.

The first attempt at getting the Access Token failed because the Kong URL was not being used.
Using the Kong URL may not be strictly required, but it is the easiest and best way to go about this entire process.

When using the above Curl command inside of a script, that script may need to be executed via the source command to ensure that the MASTER_OKEN is properly exported into the callers environment.

The Step 3: Register Applications attempts kept failing for unknown reasons.
It turns out that the default time out for the Access Token is too short for manual scripting (especially while one is perusing documentation before running a command).

This can be fixed by going to the master realm of Keycloak and selecting Realm settings.
Scroll down to Access tokens. The default value happened to be 1 minute in our case. We changed this value to something larger, like 5 minutes.

The master realm is only used for the steps up to the Step 6: Create Entitlements.
After that step the realm should now be set to the name of the tenant, such as scratch.
See Step 7: Side Car Access Token for further details.

Step 2: Keycloak Master Access Token worked while using the Kong URL.

The initial attempts failed but these failed not because of the end point, the Curl script, or the desciptor but instead due to the time out issue described in Step 2: Keycloak Master Access Token.

Step 3: Register Applications worked once the Master Access Token time out (see Step 2: Keycloak Master Access Token) was increased to something more reasonable.

We are unaware of an automated way to generate the Module Discovery and have opted to make a copy of the Application Descriptor, filtering out only the modules.
The location was then manually added, see .

The initial attempts had the paths to the Rancher Deployment Pods setup to contain the version information in the name.
This caused problems where certain characters in the name had to be replaced for the URLs to work (namely the . needed to become a -), such as http://mod-configuration-5-13-0-SNAPSHOT-139.folio-modules.svc:19002.
This led to problems with the Module Discovery script being manually created.
This was easily resolved by ensuring that the URL path to a particular pod did not have version information as part of the path, such as http://mod-configuration.folio-modules.svc:19002.

The ports chosen were the standard :8081 ports.
It turns out that the Module Discovery ports need to point to the side car ports, such as http://mod-configuration.folio-modules.svc:19002.
This process was solved by making the side car ports set to port 80 and then removing the port reference from the HTTP URL, such as http://mod-configuration.folio-modules.svc.

(This port change was ultimately reverted during later iterations where we used he standar :8081.)

Step 4: Register Module Deployment worked once we were using the correct URLs, such as http://mod-permissions.folio-modules.svc.

Our initial attempts at creating this resulted in numerous failures.
We were using unmodified base images for our deployments, such as Keycloak.
One problem turns out to be a problem because there are special modifications that are requiring and we cannot use the unmodified base images.
Another problem turns out to be a lack of environment variables being setup.

Error snippets:

Quotes from the support group:

• ”It seems you have an issue with the connection between the mgr-tenant and Keycloak. During the tenant creation process, mgr-tenant creates a new realm, client, and so on. The error states that it can't enable user management during the realm creation process”

• ”It seems you have a lack of env variables in your ConfigMap.”

• ”The only case to obtain such kind of error is a wrong image of keycloak or wrong envs in docker:”

• https://github.com/folio-org/folio-keycloak/blob/19e98e4963b3c79aabc5b2214599f1be3916c8ef/Dockerfile#L22

• “I highly recommend switching to the folioci or folioorg image (Kong as well) because those images have numerous predefined settings in line with Eureka platform expectations.”

• https://github.com/folio-org/folio-keycloak

• https://github.com/folio-org/folio-keycloak-plugins

Once we discovered and solved the problems described above, we completely dropped all of our databases.
This allowed us to make it through to other steps, like the Step 6: Create Entitlement, before we had our next set of failures.
It turns out that the database roles too must be dropped.
The reason for this is documented in RMB-1025.
Be sure to drop all relevant databases and relevant roles when waanting to completely start over with a fresh database.

The Tenant name should not have underscores in the name due to how Keycloak handles underscores as directory names.
Having underscores produces weird directory structures in Keycloak.

The Tenant name should not be called test.
The name test may have unusual behavior in Keycloak.

Step 5: Create Tenant worked from a fresh database (with all databases and roles being removed before repeating Step 1: Build Application Descriptor and so forth) while using the custom FOLIO specific images.

Our generated Tenant looks like:

We have had significant failures with the entitlements process.
This turned out to be a problem with the roles not being created in the database.
This situation is described in Step 5: Create Tenant above.

Step 6: Create Entitlements worked from a fresh database (with all databases and roles being removed before repeating Step 1: Build Application Descriptor and so forth).

A successful response looks like:

The flowId can be used to fetch additional details on how the entitlement creation process opererated.

Example request:

See the file for an example flow response.

Look in the Tenant realm (such as scratch), under the Manage -> Clients, look for and expand sidecar-module-access-client (this will be the SC_SERVICE_CLIENT_ID).
From there, look under the Credentials tab and copy the Client Secret (this will be the SC_SERVICE_CLIENT_SECRET).

When using the above Curl command inside of a script, that script may need to be executed via the source command to ensure that the SIDECAR_TOKEN is properly exported into the callers environment.

Step 7: Keycloak Side Car Access Token worked.

Be sure to now start using the Side Car Access Token from this point forward.

The patronGroup is from Reference Data derived from the following repository:

• https://github.com/folio-org/mod-users/tree/master/src/main/resources/ref-data/groups-17.3.0

Other fields like "preferredContactTypeId": "002" are copied literally from the example shown in the Kubernetes Example Deployment Add User Section documentation.
The data-raw Curl parameter was also chosen based on that example.

We had a problem with using this.
See the problem regarding patronGroup that is described below (long story short is that the patronGroup must be an empty string).

The process failed on errors like this:

The user-keycloak side cars have errors like:

This is the current state of the investigation.
The currently belief is that Kong may have some sort of problem.

The process then failed due to errors like:

This is happening because the patronGroup Reference Data is not installed in our environment.
The Kubernets Example Deployment, Add User documentation can be misleading as their example shows "patronGroup": "3684a786-6671-4268-8ed0-9db82ebca60b",.
Using that causes the above error.
The patronGroup cannot be omitted either, therefore, the patronGroup must be set to an empty string.

Additional contributing factors may be that the following environment variables needed to be set for each side car (and each side car has different values for some of these):

• ALLOW_CROSS_TENANT_REQUESTS = true

• SIDECAR_URL = http://localhost:8082 (update port number as needed.)

• SIDECAR_FORWARD_UNKNOWN_REQUESTS = true

• MODULE_NAME = (name of module, such as mod-users-keycloak)

• MODULE_VERSION = (version of module, such as 3.1.0-SNAPSHOT.141)

• MOD_USERS_BL = http://mod-users-bl:8082 (update port number as needed.)

Step 8: Create User worked once we were using the Side Car Access Token instead of the Master Access Token in addition to several Rancher environment variable changes.

A successful response looks like:

This process is failing with the following error:

Step 9: Assign User Password worked once we changed our environment to strictly follow the EBSCO ecosystem.

Existing roles may be found in the mod-keycloak reference data.

This can be done in parallel to the Step 9: Assign User Password step.
Problems at that step may not prevent performing this step.

Step 10: Create Role worked as expected.

A successful response looks like:

The capabilities added to the json file can be retrieved via a GET to /capabilities.

The following is a snippet of the Curl script being used to perform this task:

Where limit is the maximum amount of capabilities to select, such as 5000.

Initial attempt failed with this response:

The logs for mod-roles-keycloak has the stack trace, such as this snippet:

After waiting an hour so so, I ran the Curl request again with the same data and I got the following error:

The mod-roles-keycloak logs show lots of entries like the following snippet:

There is no HTTP 502 Bad Gateway on this second attempt.

Additional attempts were made to solve this problem that were unsuccessful.
However, the changes resulted in prior steps failing.

See the Problems with Entire Ecosystem Setup section for further details.

Step 11: Assign Role Capabilities worked once we changed our environment to strictly follow the EBSCO ecosystem.

We skipped the failing Step 11: Assign Role Capabilities to test and confirm if this step works or not regardless of that step.

Step 12: Assign Roles to User worked as expected.

A successful response looks like:

The Client UUID is the UUID from the id property at the top-level of the JSON response to the Curl request below.
Do not confuse the id with the clientId property.
The clientId property must not be used.