[FOLIO-2118] CI-integrated continuous deployment (Q2, cluster setup) Created: 26/Jun/19  Updated: 03/Jun/20  Resolved: 02/Jul/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Umbrella Priority: P3
Reporter: Wayne Schneider Assignee: Jakub Skoczen
Resolution: Done Votes: 0
Labels: ci, platform-backlog, q2-2019, sprint49, sprint50, sprint51, sprint52, sprint53, sprint54
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks FOLIO-2119 Preview capability for PRs (Q2, devel... Closed
blocks UXPROD-1817 Preview capability for PRs (Q3, initi... Closed
blocks UXPROD-2118 Preview capability for PRs (Q4, final... Closed
blocks FOLIO-2011 initial roll out of PR preview capabi... Closed
is blocked by FOLIO-1519 Automatic loading of sample and refer... Closed
is blocked by FOLIO-2053 add AWS K8s deployment configuration ... Closed
is blocked by FOLIO-2086 Set up HA FOLIO Rancher Closed
is blocked by FOLIO-2090 Create and document AWS VPC template ... Closed
is blocked by FOLIO-1408 deploy FOLIO on Kubernetes and docume... Closed
is blocked by FOLIO-1596 enable platform builds for all ui-* m... Closed
is blocked by FOLIO-2054 Stand up cluster for CI using AWS K8s... Closed
is blocked by FOLIO-2117 Kubernetes integration (Q2, container... Closed
Cloners
clones UXPROD-1827 CI-integrated continuous deployment (... Closed
Relates
relates to OKAPI-728 Add filtering capabilities to /_/prox... Open
relates to FOLIO-1519 Automatic loading of sample and refer... Closed
relates to FOLIO-1548 SPIKE: a lighter-weight folio/testing... Closed
relates to FOLIO-1597 Add module dependency resolution qual... Closed
relates to FOLIO-2036 Include custom backend modules in PR ... Closed
relates to FOLIO-2057 SPIKE: explore AWS K8s cost models Closed
relates to OKAPI-670 Report dependency info for a module Open
relates to FOLIO-1576 folio-ansible: allow for URL deployment Closed
Sprint:
Development Team: Core: Platform

 Description   

(this issue has been split from UXPROD-1827 Closed to indicate the cluster-setup work completed in Q2 2019)

Problem statement

Current FOLIO CI infrastructure and reference environments have several limitations that make it hard to scale them the point where a fully continuos and isolated deployment of development builds (e.g from PRs and feature branches, for both UI and backend modules) is possible. Those limitations include:

  • the reference envs (e.g snapshot) are recreated in full nightly, in a top-down approach (starting from a platform and resolving its dependencies which are deployed one by one).
  • the reference envs are primarily single-tenant (the so-called 'diku' demo tenant)
  • the reference envs are constructed from artefacts based on shared branches (master) only and all those artefacts are shared at runtime by the single demo tenant

This is makes it impossible to implement certain development process capabilities (like the PR previews, see FOLIO-1993 Closed ) without a list of fairly severe limitations (see description of the PR preview PoC for a comprehensive list of those limitations at https://dev.folio.org/guides/branch-preview/)

Proposed approach

We would like to revamp the deployment and orchestration infrastructure, create a new, clustered and multi-tenant, reference environment for development and integration purposes, and update the CI processes (Jenkins, Ansible, etc) to allow for a more continuos and isolated deployment of development artefacts.

Orchestration

Based on the prior work from from Jason Root (TAMU) and Mark Stacy (Core: platform/Colorado, see FOLIO-1408 Closed ) who have experimented with various orchestration tools and we have concluded that Kubernetes (K8s) has become the defacto standard for container orchestration and it is the orchestration tool of choice across major cloud vendors. It seems very likely that many organisations will use K8s for production-ready FOLIO deployments. K8s also brings many benefits for development deployments: a rich ecosystem of tools that ease provisioning of dependencies (e.g Helm) and widely accepted practices and processes for deployment of multiple development builds.

The Core: platform has undertaken a focused effort to ease K8s integration across FOLIO platform. This effort is tracked on UXPROD-1823 Draft .

Clustered reference environment

This work is being done outside of the Platform team, see FOLIO-2053 Closed

CI process

We would like to extend the CI process in a way which allows to:

  • deploy a backend container immediately after a successful automatic build (Jenkins/CI) on the clustered reference environment. This would include both snapshot and release builds and could be scaled up to allow for a deployment of feature branch builds (at specific points in the PR lifecycle). The deployed container should be registered with Okapi running on the clustered reference environment in order to allow for it being used as a dependency in appropriate tenant configurations.
  • create an independent tenant, on the clustered reference environment, for each platform- and ui- module build (of all kinds, including release, snapshot, and feature branch) to allow for running the particular build in isolation (including sample data set isolation). In the case of ui- modules, which are not self-contained units, the build process must be able to embed the module in an appropriate platform to form a complete Stripes bundle. Such bundle should be exposed to users after a succesful build. Okapi remains responsible for providing backend service dependencies in the created tenant.

See FOLIO-2054 Closed , FOLIO-2055 Closed and FOLIO-2056 Closed

Further read

PR previews: UXPROD-1817 Closed
Kubernetes integration: UXPROD-1823 Draft
FOLIO RM and CI concepts: https://docs.google.com/document/d/1au2hG4gPekyZ_HxAU7s6sc4NOvRaVovUcBnMLa1iR7E/edit?usp=sharing


Generated at Thu Feb 08 23:18:20 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.