[FOLIO-2130] SPIKE: how to clean up unused container instances Created: 01/Jul/19  Updated: 03/Jun/20  Resolved: 26/Jul/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Jakub Skoczen Assignee: John Malconian
Resolution: Done Votes: 0
Labels: platform-backlog
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks UXPROD-1827 CI-integrated continuous deployment (... Closed
Relates
relates to FOLIO-2194 Develop script to clean up expired K8... Closed
relates to FOLIO-2055 create an Ansible role for backend-mo... Closed
Sprint: CP: sprint 68
Story Points: 3
Development Team: Core: Platform

 Description   

This is a follow up of FOLIO-2055 Closed

The goal of the SPIKE is to provide a description for the actual task.



 Comments   
Comment by Wayne Schneider [ 01/Jul/19 ]

AFAICT, there is no single Okapi API call to discover what modules are unused. You would have to follow a script something like this:

  1. Get list of tenants
  2. For each tenant, get list of modules enabled
  3. Compare list of enabled modules against the /_/discovery/modules API – modules that are visible in discovery but are not enabled for any tenant would be the candidates for cleanup

You would need to do this for all Okapi clusters that are sharing the K8s cluster to get a master list.

Cleanup would consist of undeploying the module from the K8s cluster and deleting it from the Okapi discovery service on all Okapi clusters.

Comment by Jakub Skoczen [ 02/Jul/19 ]

Wayne Schneider Adam Dickmeiss John Malconian My concern is that scripting something on top of existing Okapi APIs is error prone. Consider this scenario:

1. A new build of a backend module is done by the CI and deployed to the FOLIO cluster. The module is registered with Okapi's discovery service but not used in any tenant configuration because the no UI PR has been issued yet.

2. The process clean up runs and cleans up the seemingly unused instance

3. UI developer issues a PR that requires the particular snapshot build of the backend module but fails because it's been cleaned up

I am not sure what's the best approach here. Maybe the clean-up process should run when merging the UI module PR?

Comment by Wayne Schneider [ 08/Jul/19 ]

Maybe simply some logic that prevents the latest versions (latest release + latest snapshot) of a module from being cleaned up, regardless of whether they are used by a tenant?

Comment by John Malconian [ 25/Jul/19 ]

So I'm thinking of the following retention policies:

  • snapshots built from master: unused backend modules that are not the latest two versions released are removed. A script can determine which are two most recent versions by getting a list of modules descriptors from Okapi.
  • proper releases: We keep the three released versions of backend modules regardless of whether they are in-use by a tenant. These modules can be determined in essentially same way as the snapshots - just set 'preRelease=false' when querying Okapi.
  • branch releases: All backend branch artifacts will reside in dedicated K8 namespaces. These namespaces will be destroyed along with all associated resources that reside in the namespace when PR is closed.

A script will need to be written that determines which module containers are candidates for removal. This script will be invoked via Jenkins at a regular interval - for now, daily. The script will need to be able to query Okapi to determine which modules are in-use, the status of PRs on Github, as well as conduct K8 operations via the K8 API.

Comment by John Malconian [ 26/Jul/19 ]

Next steps: Discuss with team. Create issue for cleanup script.

Generated at Thu Feb 08 23:18:26 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.