Manage Scratch Environment ver 1.2

1.INTRODUCTION

Each development team has access to a “developer scratch environment” to ensure that their changes are ready to be finalised.

Rancher is a Kubernetes management tool. Useful links:

• FOLIO rancher main URL: https://rancher.dev.folio.org

• Project UI URL: https://<Project name>.ci.folio.org

• Basic Okapi URL: https://<Project name>-okapi.ci.folio.org

• PgAdmin4 URL: https://<Project name>-pgadmin.ci.folio.org (login as user ‘chart@example.local’ or ‘user@folio.org’ password ‘SuperSecret’)

• Logs: https://logs.ci.folio.org/

Before commencing become familiar with this FAQ. Some teams also provide further general instructions.

For general assistance use the Slack channel #scratch-environments and help via your team-specific channel.

The Kitfox DevOps team wiki page has further resources regarding Rancher scratch environments.

Logging in

To have access to Rancher, you need to be a member of a FOLIO organization Team in GitHub. Go to the main Rancher URL and login with GitHub account.

Select the default Cluster and your Project. Default cluster name is folio-eks-2-us-west-2.

Project description

Every project has its own PostgreSQL, Okapi, pre-installed core backend modules, and Stripes Platform bundle. All of these containers run in the default namespace (which is named the same as the Project name). Also every Project runs Prometheus and Kafka. FOLIO modules are installed from the FOLIO Helm repository. Postgres and Kafka are installed from the Bitnami Helm repository.

Running modules

All Projects installed FOLIO Helm repository (Catalog in Rancher) which contains all backend modules, UI, pgadmin. By default the backend modules are pulled from DockerHub/folioci repository with a ‘latest’ tag. UI module is pulled from docker.dev.folio.org repository with own project latest tag <project>-latest. All modules can be managed in the App menu in Rancher, where new modules can be added or existing ones can be upgraded.

2. CREATE NEW SCRATCH ENV FOR TEAM

link: Manage Scratch Environment ver 1.2

General properties

Listed properties cannot be modified without full recreation of related resource.

Postgre DB versionManage Scratch Environment ver 1.2
DB Size

Build with parameter

Available parameters:

teamName, Action, TenantName, Repo, Plan, Change, Register, branch, platform_image

Open

Parameter teamName:

Specifies the name of environment and access parameters in Rancher. For most values admin and members of github teamName team have access to Rancher to configure module deployment.

Parameter Action:

Specifies what action will be performed.

Parameter TenantName:

Default value: diku

This parameter specifies tenantID for enviroment. Okapi will contain 2 tenants supertenant and TenantName. All modules will be registered and deployed within tenant TenantName. 

!!! NOTE !!! Module platform-complete is configured to address a specific tenant, which is determined during BUILD-UI.

Parameter Repo and branch:

Files okapi-install.json & stripes-install.json are used to create/update environment. They can be specified with repo and branch. 

Repo value stands for platform-core or platform-complete.

There is separated list of branches for each value of repo.

!!! NOTE !!! If selected branch does not contain okapi-install.json & stripes-install.json, pipeline will fail.

There are created branches in platform-core (e.g. folijet-rancher). These branches contain a list of modules required for a particular team.

Parameter platform_image:

Specified image will be used for module platform-complete. Available values are formed by BUILD-UI.

3. Build UI module from branch

To build UI from a specific branch, use this Jenkins pipeline:
https://jenkins-aws.indexdata.com/job/scratch_environment/job/BUILD-UI/

• Ensure that you are logged in to Jenkins

• Select “Build with Parameters”

• Choose your team name

• Choose the branch that you want to build from

• Then select “Build”

The tag for the image is: docker.dev.folio.org/platform-complete:team_name-build number  

for example :  docker.dev.folio.org/platform-complete:core-platform-530

How to use UI

1. go to rancher →choose your environment → apps → platform-complete

2. upgrate  module need find "answers" option →  change image 

Here is default answers for Frontend Bundle Application deployment:

resources.limits.cpu = 200m resources.limits.memory = 500Mi image.tag = <project_name>-latest ingress.enabled = true ingress.hosts[0].host = <project_name>.ci.folio.org ingress.hosts[0].paths[0] = /

In package.json set --max-old-space-size=8192 to build options.

4. Build backend module from branch

Building Backend modules

You can build your own module, and automatically deploy it with Rancher pipeline and Helm. To get started, create your own branch and modify .rancher-pipeline.yml to your needs (for example as in this pipeline).

Go to Workloads -> Pipelines, run pipeline for that branch, and then Rancher will deploy the new version of that module.

Any backend module can be built from a specific branch. Use this Jenkins pipeline:
https://jenkins-aws.indexdata.com/job/scratch_environment/job/BUILD-BACKEND/

• Ensure that you are logged in to Jenkins

• Select “Build with Parameters”

• Choose the module that you want to build

• Choose which branch that you want to build the module from

• Then select “Build”

• Select “Console Output”

• Near the end, find the full image name with the proper tag:

• Go to your environment in Rancher –> Apps –> the module that you built

• Select the vertical ellipsis ⋮ and then “Upgrade”

• In the “Answers” section, add the following:

  • Variable: image.repository

    • Value: docker.dev.folio.org/module_name

  • Variable: image.tag

    • Value: the tag obtained from the build

• Select “Upgrade”

Building Frontend

Modify sample pipeline and start build.

Here is default answers for Frontend Bundle Application deployment:

resources.limits.cpu = 200m resources.limits.memory = 500Mi image.tag = <project_name>-latest ingress.enabled = true ingress.hosts[0].host = <project_name>.ci.folio.org ingress.hosts[0].paths[0] = /

In package.json set --max-old-space-size=8192 to build options.

5.Registering modules in Okapi

Module registration runs automatically after the install or upgrade procedure. Helm uses post-install and post-upgrade hooks to run the module registration job for each module. Helm gets ModuleDescriptors from the FOLIO Registry (https://folio-registry.dev.folio.org) – it gets the latest snapshot descriptor.

Default steps for module registration:

• Pushing module descriptor

• Pushing module deployment

• Creating tenant (default ‘diku’)

• Enabling module for tenant

Docker commands can also be used to do registration manually:

• Registering a particular module (backend or UI module)

  docker run --rm -e TENANT_ID=diku -e OKAPI_URL=https://<project name>-okapi.ci.folio.org -e MODULE_NAME=<module name> docker.dev.folio.org/folio-okapi-registration

• Registering all modules (backend and UI from ‘platform-complete’ list)

  docker run --rm -e TENANT_ID=diku -e OKAPI_URL=https://<project name>-okapi.ci.folio.org -e MODULE_NAME='' docker.dev.folio.org/folio-okapi-registration

• Registering all UI modules only

  docker run --rm -e TENANT_ID=diku -e OKAPI_URL=https://<project name>-okapi.ci.folio.org -e MODULE_NAME='platform-complete' docker.dev.folio.org/folio-okapi-registration

for example:
docker run --rm -e TENANT_ID=diku -e OKAPI_URL=https://core-platform-okapi.ci.folio.org -e MODULE_VERSION=24.0.0-SNAPSHOT.999 -e MODULE_NAME=mod-inventory-storage docker.dev.folio.org/folio-okapi-registration

Apply module permissions

The last step after modules registration is to apply permissions for modules to the admin user.

• Applying permissions for all installed modules to diku_admin

  docker run --rm -e TENANT_ID=diku -e ADMIN_USER=diku_admin -e ADMIN_PASSWORD=admin -e OKAPI_URL=https://<project name>-okapi.ci.folio.org folioci/bootstrap-superuser

Environment variables

Environment variables for database and backend modules are stored in Kubernetes secrets (Workload -> Secrets) and installed by default to every Project.

Deployment tips

Some backend modules are based on SpringBoot and require more CPU to start. To deploy those applications (such as mod-agreements or mod-licenses) you need to override CPU and memory parameters. Add ‘answers’ to module deployment:

resources.limits.cpu = 500m resources.limits.memory = 600Mi

If you notice a pod restarting a lot, it may be that it is running out of memory. Usually increasing the resources.limits.memory in its yaml via Rancher will fix this. The pod limits on CPU may also be too low to allow your cluster to run reliably. If things go wrong, consider increasing resources.limits.cpu for a given pod. You may also consider removing the pod-level resources.limits.cpu limit and set a limit for your namespace instead, so as to not deplete other teams’ resources on the cluster.