Overrides/Supersedes 

This decision was migrated from the Tech Leads Decision Log as part of a consolidation process.  The original decision record can be found here.

RFC 

N/A

Stakeholders

• All developers, SysOps, Hosting providers

Contributors

@Raman Auramau 

Approvers

Background/Context

Introduction

A secret is anything to which you want to tightly control access, such as API keys, passwords, certificates, and other sensitive information. Encrypted storage is needed to store secrets in a safe way, as well as a service to provide secrets management capabilities.

At the moment the FOLIO platform has neither dedicated secret storage as a common module or service nor even a standard approach for secrets management.

Note: edge-common has an interface for that, there are 3 implementations as for now - Vault, AWS, and an ephemeral one for dev needs (properties file), with read access implemented only. So, this piece of edge-common can be leveraged as some basic experience though it's definitely not enough.

With that, this document is aimed to provide analysis details and answer the following questions:

• What technologies would be appropriate for this?

• What might a solution look like? Is there some abstraction layer/interface allowing for different implementations?

• What's required for a module to store/retrieve a secret?

• What's required on the infrastructure side?

Assumptions

N/A

Constraints

N/A

Rationale

Requirements

Please refer to FOLIO secrets management - Requirements for a list of identified requirements.

Analysis

Current state

Three environment types should be considered:

• Standard EC2 machine with all dockers on it (native docker, so Env Var are only available) - IndexData (snapshots, testing)

• Kubernetes (rancher envs)

  • K8S Secrets with Env Var is used there to ensure compatibility with the previous one

  • Rancher has UI for displaying & configuration

• AWS Cloud - SSM is used; application knows secret alias, IAM role is set on EC2 level

Sample

https://github.com/folio-org/edge-oai-pmh/tree/master/src/main/resources

https://github.com/folio-org/mod-search/blob/master/descriptors/ModuleDescriptor-template.json

Spring Cloud Config Server

https://cloud.spring.io/spring-cloud-config/reference/html/ Spring Cloud Config Server supports Git/SVN, Vault, JDBC database, Redis, AWS S3 as a backend for configuration properties but not AWS SSM. With that, AWS SSM is an Amazon native secret storage, and its usage for cloud-based deployment looks pretty justified. There is still particular way for thttps://rieckpil.de/resolving-spring-boot-properties-using-the-aws-parameter-store-ssm/.

Kubernetes Secrets

https://kubernetes.io/docs/concepts/configuration/secret/ - Kubernetes Secrets is a resource which is mounted into containers with a new volume type. It enables storing and managing secrets in safer and more flexible way than just putting them verbatim in a Pod definition or in a container image. Secrets can be

• mounted as data volumes or

  • mounted Secrets are updated automatically

• exposed as environment variables or

  • if a container already consumes a Secret in an environment variable, a Secret update will not be seen by the container unless it is restarted. There are third party solutions for triggering restarts when secrets change

• using imagePullSecrets (i.e. by the kubelet when pulling images for the Pod)

This mechanism looks to be really convenient and efficient to deliver secrets into containers under Kubernetes. At the same time, this is Kubernetes-specific feature, and it does not fit any containers rather than under Kubernetes managements. As far as I could find Kubernetes should not be considered as the only container orchestration tool.

Alternatives for evaluation

Option 1 - Introduce mod-secret-storage and dedicated platform API

The idea of this proposal is to introduce a new dedicated back-end module which will be able to work with a number of specific secret storages providers (from one side) and provide generic platform API for consumers (from the another one).

Draft visualization

Solution components

• FOLIO modules read secrets using sidecar agent

• secret-storage agent is a sidecar jar-library to facilitate communication with mod-secret-storage

  • pure Java enables both Vert.x and Spring support

  • operates with mod-secret-storage via platform API

  • different update support strategies can be implemented - e.g., keep secrets in RAM with some time-to-live or request them from mod-secret-storage every time

• mod-secret-storage is a dedicated module encapsulating and hiding particular Secret Storage provider details and providing a generic platform API for secrets management

  • it can also provide additional functionality like time-to-live and expiration management, enforce rotation, provide additional level of permissions verification, etc. - up to requirements

  • can support CRUD operations for secrets

• ui-secret-storage is a module providing UI for secrets management

  • if need; can be added at any time

• secret storage is a persistent storage for long-time keeping of secrets

• Devops / Administrator can use either dedicated UI and secret storage provider UI for secrets configuration and management