FOLIO Module Descriptor Registry

The FOLIO Module Descriptor Registry is a repository used to assist in the automated deployment of a scratch environment.
It consists of several scripts along with several special purpose branches.

The primary purpose of this repository is to provide a Eureka friendly repository and scripts for assisting in the automation of Continuous Integration / Continuous Deployment (CI/CD).
Additional scripts are also provided to assist in operating in a purely OKAPI environment for those who have not yet migrated to Eureka.

Repository Branch Structure

The Repository consists of traditional development-style branches as well as deployment branches, such CI/CD.

The `main` Branch

The main branch consist primarily of all scripts.
These are directly used by GitHub Actions or by users within their own local environments.
Most template and setting files stored in this repository are generally examples or defaults used for the snapshot branch.

Each CI/CD branch will likely contain their own template and settings files such that the main branch will eventually have none of these files.

The `snapshot` Branch

The snapshot branch consists of the generated module descriptors for each release.
The sub-directories are broken up into flower release names or other special case names like snapshot.
This branch is directly used to generate the GitHub Pages Module Descriptor Registry.

Template files and other setting files specific to the snapshot are also stored in this branch.

This branch is auto-updated regularly using GitHub Actions to load the latests snapshot releases.
These are derived from the Platform Complete repository snapshot branch.

The folio-module-descriptor-registry repository is therefore down stream of the Platform Complete repository.

The `fleet` Branch

The fleet brach consists of manifest YAML files used for deploying the snapshot release.
These manifest YAML files are generated based on the snapshot branch for the snapshot release.

The fleet branch is under active development and may be empty at this time.

Custom CI/CD Repositories

The FOLIO module Descriptor Registry is intended to provide a registry as well as the scripts to operate and utilize the registry.

It is recommended that anyone wanting to use this repository to create their own repository for storing the custom templates and settings.
Doing so would further allow for even storing any such templates and settings in a private repository.

It is recommended that any actual provide information to not be stored in the templates or settings.

For example, the build_deployments.sh script utilizes template configurations for building a Fleet Manifest YAML file.
Rather than adding the private data in an environment variable, a secret could instead be used.
This can be done by creating or modifying the template/deploy/input/main/vars.json file, such as adding the following:

  "[ENV_FROM_VAULT_SECRET]": {
    "secretRef": {
      "name": "vault-connect",
      "optional": true
    }
  },

Furthermore, using ”optional”: true is strongly encouraged so that any configuration problems never prevent the Fleet Manifest from being processed.
Instead, the environment variable or secret, if missing, can be added directly in Rancher and the containers can be restarted without having to rebuild or modify the Fleet Manifest.

GitHub Pages

This registry builds and stores the Module Descriptors in the FMDR GitHub Pages.

The top-level directories structure represents the particular flower release and any special case groupings like snapshot.

This registry, unlike OKAPI, is static and provides no dynamic functionality, such as filtering and sorting.
A -latest symbolic link is provided to point to the most recent release of any given module.

A date and time is presented at the top of each page in UTC to designate when the GitHub Page was last built.

The Snapshot Registry is intended to contain the latest CI/CD snapshot release of each module.

GitHub Actions

This registry provides GitHub Actions to operate the scripts in a CI/CD environment.

These GitHub Action scripts are designed to be both automatically run or manually run (by users with sufficient priviledges).
They accept various inputs to allow customizing the operation, such as adding DEBUG settings.

GitHub restricts operation of GitHub Actions on branches other the then main branch.

As a work-around to this, several of the scripts accept input arguments that allows for specifying a custom branch to operate off of.
In this case, the Action is being operating on the main branch but it may have checked out an alternate version of the repository at a custom branch.

The most common inputs to the GitHub Actions are:

Input Variable	Description

Input Variable	Description
`debug_mode`	Enables debugging when non-empty. Special options (space separated): `curl`, `git`, `json`, `verify`, `yarn`, `curl_only`, `git_only`, `json_only`, `verify_only`, and `yarn_only`.
`registry_branch`	The name of the branch containing the registry descriptor files, such as `snapshot`.
`script_branch`	The name of the branch containing the scripts, such as `master`.

GitHub restricts GitHub Pages in its environments section.

It is recommended to set the branch restrictions for GitHub Pages to appropriate branches, such as:

main
snapshot
gh_testing* (for testing GitHub Pages)

Be sure to restrict access to make changes to these branches to help prevent unintended execution and operations.

The following are GitHub Action scripts:

These GitHub Action scripts provide a run-name setting at the top of the script to provide details based on variables passed to the script to help better identify what and how the GitHub Action is run.

Build and Deploy a Release

This section provides a guide involving every script in this registry and the results of such scripts to manually build and deploy a release.
Many of the aspects of this may rely on third-party software beyond the scripts, such as Kubernetes, Rancher, and Fleet.
The focus of this guide is on these scripts rather than on the third-party software.

This documentation utilizes Poppy R2-2023-csp-7 as an example release for legacy OKAPI examples.

Minimum Requirements

A Local GitHub Runner, Docker Container, or native (or virtual) Linux system with GNU Bash is expected.
Other systems are not well tested.

Each script provides a description of user-space programs that are expected to be present in order to operate.
The following is a summary of all of these such user-space programs:

basename
bash
cd
curl
date (optional in some, but not all, cases)
find
git
grep
jq
ls
mkdir
sed
sleep (optional)
sort
yarn
yq

Setting up Required Repositories

There scripts are generally separate from the templates and settings.
This, therefore, results in requiring 2 distinct repositories.

The script repository, such as the main branch of folio-module-descriptor-registry.
A configuration repository, such as the snapshot branch of folio-module-descriptor-registry

The script repository directory will be checked out into the scripts directory, such as:

git clone -b main https://github.com/TAMULib/folio-module-descriptor-registry.git scripts

The configuration repository will be checked out into the data directory, such as:

git clone -b snapshot https://github.com/TAMULib/folio-module-descriptor-registry.git data

Populate Release

The release files need to be fetched and populated.

Building for either Eureka and OKAPI will be identical other than possible changes in release names, branch/tag names, and directory names.

This is done via the populate_release.sh script:

cd /path/to/data/
POPULATE_RELEASE_FLOWER="poppy" \
POPULATE_RELEASE_TAG="R2-2023-csp-7" \
POPULATE_RELEASE_REPOSITORY_PART="tags" \
POPULATE_RELEASE_FILES="install.json" \
  bash /path/to/scripts/script/populate_release.sh

This should result in the download of an install.json file.
All of the downloaded module releases are found in release/poppy/.

The script may send errors about OKAPI curl request failing, such as:

ERROR: Curl request failed for: https://folio-registry.dev.folio.org/_/proxy/modules/okapi-5.1.2 to release/poppy/okapi-5.1.2 (system code 22).

This is not a problem as there is no expected OKAPI module to fetch and can safely be ignored.

Populate the Node

The populate_release.sh script does not catch all of the dependencies.
The populate_node.sh script grabs packages from the Node/NPM/Yarn system for packages that are not in the install.json file.

This should not be needed by the legacy OKAPI process.
This has been found to be needed by the Eureka process, namely @folio/authorization-policies and @folio/authorization-roles.

Populating for Eureka

Create a /path/to/data/workspace/package.json file like this (be sure to update the /release/snapshot/ path to the appropriate release):

{
  "name": "workspace",
  "private": true,
  "version": "1.0.0",
  "workspaces": [ "*" ],
  "dependencies": { }
}

Then execute the script:

cd /path/to/data/
POPULATE_NODE_DESTINATION="${PWD}/release/snapshot/" \
  bash /path/to/scripts/script/populate_node.sh

The POPULATE_NODE_DESTINATION environment variable requires a full path.

Therefore, the example above uses ${PWD}/ to produce a full path relative to the current working directory.

Build Latest Version

The generation of the -latest release symbolic links is primarily needed by the FMDR GitHub Pages.

This script works not by select the latest based on time stamp but it instead works by expecting the versions specified in the install.json file to represent the latest release.

The BUILD_LATEST_SKIP_NOT_FOUND variable is recommended because some dependencies, such as okapi, do not have a module or release to be managed.

Building for either Eureka and OKAPI will be identical other than possible changes in release names and directory names.

Execute the script:

cd /path/to/data/
BUILD_LATEST_PATH="release/poppy/" \
BUILD_LATEST_FILES="install.json" \
BUILD_LATEST_SKIP_NOT_FOUND="y" \
  bash /path/to/scripts/script/build_latest.sh

Synchronize Repository

The repository changes can by synchronized with the upstream repository.
This step is not necessary for the local build described here.

If this is desired, then this can be built doing something like this:

cd /path/to/data/
bash /path/to/scripts/script/sync_snapshot.sh

This script will perform git commit and git push actions.

Build GitHub Pages

If the Synchronize Repository step above is performed, then the GitHub Pages may need to be re-created to provide an up to date build of the repository.

Execute the script:

cd /path/to/data/
bash /path/to/scripts/script/build_pages.sh

This script is intended to be run using a GitHub Action where the GitHub Action must build and deploy the artifact.

Locally running the script will allow for easier investigation and debugging of the generated files but will not result in the GitHub Pages being updated.
There may be a way to manually package these files and upload them.
To do that, please consult the appropriate GitHub documentation.

Build the Launch Files

The launch files are snippets of the launch descriptors from each module that are extracted and converted into a template format for the automated construction (or as a documentation guide) of the Fleet Manifest templates.

An instructions file, located in /path/to/scripts/template/launch/input/instructions.json, provides the fields to extract and how to extract them.

Building for either Eureka and OKAPI will be identical other than possible changes in release names and directory names.

Execute the script:

cd /path/to/data/
BUILD_LAUNCHES_INPUT_PATH="/path/to/scripts/template/launch/input/" \
BUILD_LAUNCHES_RELEASE_PATH="release/poppy/" \
BUILD_LAUNCHES_OUTPUT_PATH="template/deploy/input/launches/poppy" \
  bash /path/to/scripts/script/build_launches.sh

This should result in the creation of launch files for the specific release within the /path/to/data/template/deploy/input/launches/poppy/ directory.

Build Location Files

There are two or more different upstream Docker repositories used by FOLIO, folioci and folioorg.
The install.json and similar files might have package versions that exist on only one of those repositories.
The build_location.sh script provides a way to automatically detect which of the repositories has which version, called tag in Docker, to use.
The results of this detection is stored in a location file for each module.

Building for either Eureka and OKAPI will be identical other than possible changes in release names and directory names.

Building locations makes several requests to one or more of the upstream Docker repositories.
Rate limiting and blocking is a concern.
The script provides ways to reduce the number of requests.
The script uses HTTP HEAD requests to reduce the chance of being rate limited or even blocked.

The default number of requests performed at a time might be 100.
If there are more than 100 dependencies, then either increase the number of requests via BUILD_LOCATION_LIMIT or execute the script multiple times.

Setting BUILD_LOCATION_DEBUG=”y” will be especially useful here.

Execute the script:

cd /path/to/data/
BUILD_LOCATION_REPOSITORIES_PATH="/path/to/scripts/template/location/" \
BUILD_LOCATION_FILES="install.json" \
BUILD_LOCATION_FLOWER="poppy" \
  bash /path/to/scripts/script/build_location.sh

The location of every single module, by version, is stored in location/poppy/ based on the BUILD_LOCATION_FLOWER variable.

Construct the Template Files

Each institution and organization are expected to have their own unique Rancher configuration needs based on their environment.
The template files provide a way to dynamically construct the Fleet Manifest files.

With Eureka, each Application is expected to have its own Fleet Manifest YAML file.

With OKAPI, the Fleet Manifest YAML file might represent a complete flower release.

These template files are manually constructed by a user.
Basic ones are already created within template/deploy/input of the folio-module-descriptor-registry repository.

The Build Deployments documentation section of the FMDR README goes into details on how to construct these templates.

Details on configuring and using the templates are described in the FMDR: Fleet Deployment Templates.

The template/deploy/input/main/vars.json is where most of the configuration will be added.

These templates may be specific to different flower releases.
Having different repositories or different sub-directories for different releases is suggested.

A good starting point would be to copy the existing templates over:

cd /path/to/data/
mkdir -p template/deploy/{input,output}/
cp -R /path/to/scripts/template/deploy/input/* templte/deploy/input/

Build the Application Descriptor

This is a Eureka specific process, but has been adapted to be used for the legacy OKAPI.

This step requires an install.json file to exist.
Either provide one or perform the Populate Release step as described above.

Building for Eureka

Eureka is designed around utilizing Applications.
There are expected to be several Applications needing to be built.
However, at this time there are specialized Applications that consist of a set of multiple Applications all grouped as a single Application, such as app-platform-complete and app-platform-minimal.

A custom applicaiton may be built based on projects like app-platform-minimal by editing the app-platform-minimal.template.json and pom.xml (renaming files as appropriate).

The app-platform-minimal is being used for Eureka building for the purposes of this guide.

The plugin with an artifactId of folio-application-generator (and groupId of org.folio) in the pom.xml must be updated to have the configuration point to the desired registry.

For the FOLIO Module Descriptor Registry with app-platform-minimal using the snapshot release, this must be set as follows:

        <configuration>
          <templatePath>${basedir}/app-platform-minimal.template.json</templatePath>
          <moduleRegistries>
            <registry>
              <type>simple</type>
              <url>https://tamulib.github.io/folio-module-descriptor-registry/snapshot/</url>
            </registry>
          </moduleRegistries>
        </configuration>

The Application Descriptor for Eureka can now be built using the following commands:

cd /path/to/app-platform-minimal/
mvn clean package

The generated Application Descriptor may look something like this: target/app-platform-minimal-2.1.0-SNAPSHOT.json.

Building for OKAPI

OKAPI lacks a concept of an Application and therefore an Application Descriptor.
However, the build_module_discovery.sh script requires an Application Descriptor.

One can be generated using the same process as done with Eureka above (changing all packages and versions as appropriate) or one can be partially generated.
The build_application_descriptor.sh script provides a way to generate the bare minimum requirements using existing information without needing to create a custom project like app-platform-minimal.
The only thing needed by the build_application_descriptor.sh script is an install.json file.
This install.json is already provided by existing flower releases such as the one seen in the R2-2023-csp-7 branch of platform-complete (a Poppy release).

The Application Descriptor for OKAPI can be built usig the following commands:

cd /path/to/data/
BUILD_APP_DESCRIPTOR_NAME="app-poppy" \
BUILD_APP_DESCRIPTOR_VERSION="R2-2023-csp-7" \
BUILD_APP_DESCRIPTOR_OUTPUT_PATH="/path/to/files/" \
  bash /path/to/scripts/script/build_application_descriptor.sh

This should result in a file name app-poppy-R2-2023-csp-7.json to be generated in the /path/to/files/ directory.

Build the Module Discovery

Like the Application Descriptor, the Module Discovery file is used in Eureka for deploymnt.
With OKAPI, this is used to help build the Fleet Manifest file.

Building for either Eureka and OKAPI will be identical other than possible changes in release names, directory names, and the name of the Application Descriptor.

Execute the script:

cd /path/to/data/
BUILD_MOD_DISCOVERY_INPUT="/path/to/files/app-poppy-R2-2023-csp-7.json" \
BUILD_MOD_DISCOVERY_OUTPUT="/path/to/files/discovery-poppy-R2-2023-csp-7.json" \
  bash /path/to/scripts/script/build_module_discovery.sh

This should result in a file at /path/to/files/discovery-poppy-R2-2023-csp-7.json.

Build the Fleet Manifest

The Fleet Manifest file is used for automated deployment.

Building for either Eureka and OKAPI will be identical other than possible changes in release names, directory names, deployment namespace, and the name of the Module Discovery.

Execute the script:

cd /path/to/data/
BUILD_DEPLOY_DISCOVERY="../files/discovery-poppy-R2-2023-csp-7.json" \
BUILD_DEPLOY_APP="app-poppy" \
BUILD_DEPLOY_NAMESPACE="folio-modules" \
  bash ../scripts/script/build_deployments.sh

This should result in a Fleet Manifest YAML file at template/deploy/output/yaml/app-poppy.yaml.

FOLIO Module Descriptor Registry

Repository Branch Structure

The `main` Branch

The `snapshot` Branch

The `fleet` Branch

Custom CI/CD Repositories

GitHub Pages

GitHub Actions

Build and Deploy a Release

Minimum Requirements

Setting up Required Repositories

Populate Release

Populate the Node

Populating for Eureka

Build Latest Version

Synchronize Repository

Build GitHub Pages

Build the Launch Files

Build Location Files

Construct the Template Files

Build the Application Descriptor

Building for Eureka

Building for OKAPI

Build the Module Discovery

Build the Fleet Manifest

Additional Pages and Links

Additional Pages

Additional Links

FOLIO Module Descriptor Registry

Repository Branch Structure

The main Branch

The snapshot Branch

The fleet Branch

Custom CI/CD Repositories

GitHub Pages

GitHub Actions

Build and Deploy a Release

Minimum Requirements

Setting up Required Repositories

Populate Release

Populate the Node

Populating for Eureka

Build Latest Version

Synchronize Repository

Build GitHub Pages

Build the Launch Files

Build Location Files

Construct the Template Files

Build the Application Descriptor

Building for Eureka

Building for OKAPI

Build the Module Discovery

Build the Fleet Manifest

Additional Pages and Links

Additional Pages

Additional Links

The `main` Branch

The `snapshot` Branch

The `fleet` Branch