Overrides/Supersedes 

This decision was migrated from the Tech Leads Decision Log as part of a consolidation process.  The original decision record can be found here.

RFC 

N/A

Stakeholders

• All developers, SysOps, Hosting providers

Contributors

Raman Auramau 

Approvers

Background/Context

Introduction

A secret is anything to which you want to tightly control access, such as API keys, passwords, certificates, and other sensitive information. Encrypted storage is needed to store secrets in a safe way, as well as a service to provide secrets management capabilities.

At the moment the FOLIO platform has neither dedicated secret storage as a common module or service nor even a standard approach for secrets management.

Note: edge-common has an interface for that, there are 3 implementations as for now - Vault, AWS, and an ephemeral one for dev needs (properties file), with read access implemented only. So, this piece of edge-common can be leveraged as some basic experience though it's definitely not enough.

With that, this document is aimed to provide analysis details and answer the following questions:

• What technologies would be appropriate for this?
• What might a solution look like? Is there some abstraction layer/interface allowing for different implementations?
• What's required for a module to store/retrieve a secret?
• What's required on the infrastructure side?

Assumptions

N/A

Constraints

N/A

Rationale

Requirements

Please refer to FOLIO secrets management - Requirements for a list of identified requirements.

Analysis

Current state

Three environment types should be considered:

• Standard EC2 machine with all dockers on it (native docker, so Env Var are only available) - IndexData (snapshots, testing)
• Kubernetes (rancher envs)
  • K8S Secrets with Env Var is used there to ensure compatibility with the previous one
  • Rancher has UI for displaying & configuration
• AWS Cloud - SSM is used; application knows secret alias, IAM role is set on EC2 level

Sample

https://github.com/folio-org/edge-oai-pmh/tree/master/src/main/resources

https://github.com/folio-org/mod-search/blob/master/descriptors/ModuleDescriptor-template.json

Spring Cloud Config Server

https://cloud.spring.io/spring-cloud-config/reference/html/ Spring Cloud Config Server supports Git/SVN, Vault, JDBC database, Redis, AWS S3 as a backend for configuration properties but not AWS SSM. With that, AWS SSM is an Amazon native secret storage, and its usage for cloud-based deployment looks pretty justified. There is still particular way for thttps://rieckpil.de/resolving-spring-boot-properties-using-the-aws-parameter-store-ssm/.

Kubernetes Secrets

https://kubernetes.io/docs/concepts/configuration/secret/ - Kubernetes Secrets is a resource which is mounted into containers with a new volume type. It enables storing and managing secrets in safer and more flexible way than just putting them verbatim in a Pod definition or in a container image. Secrets can be

• mounted as data volumes or
  • mounted Secrets are updated automatically
• exposed as environment variables or
  • if a container already consumes a Secret in an environment variable, a Secret update will not be seen by the container unless it is restarted. There are third party solutions for triggering restarts when secrets change
• using imagePullSecrets (i.e. by the kubelet when pulling images for the Pod)

This mechanism looks to be really convenient and efficient to deliver secrets into containers under Kubernetes. At the same time, this is Kubernetes-specific feature, and it does not fit any containers rather than under Kubernetes managements. As far as I could find Kubernetes should not be considered as the only container orchestration tool.

Alternatives for evaluation

Option 1 - Introduce mod-secret-storage and dedicated platform API

The idea of this proposal is to introduce a new dedicated back-end module which will be able to work with a number of specific secret storages providers (from one side) and provide generic platform API for consumers (from the another one).

Draft visualization

Solution components

• FOLIO modules read secrets using sidecar agent
• secret-storage agent is a sidecar jar-library to facilitate communication with mod-secret-storage
  • pure Java enables both Vert.x and Spring support
  • operates with mod-secret-storage via platform API
  • different update support strategies can be implemented - e.g., keep secrets in RAM with some time-to-live or request them from mod-secret-storage every time
• mod-secret-storage is a dedicated module encapsulating and hiding particular Secret Storage provider details and providing a generic platform API for secrets management
  • it can also provide additional functionality like time-to-live and expiration management, enforce rotation, provide additional level of permissions verification, etc. - up to requirements
  • can support CRUD operations for secrets
• ui-secret-storage is a module providing UI for secrets management
  • if need; can be added at any time
• secret storage is a persistent storage for long-time keeping of secrets
• Devops / Administrator can use either dedicated UI and secret storage provider UI for secrets configuration and management

Pros and Cons

• mod-secret-storage provides a good abstraction layer so that AWS SSM, Vault or any other secret storage can be introduced and used behind it
• Fully FOLIO in-house implementation enables full control and any required customization, with centralized audit log etc., though it will require certain ownership and maintenance
• Good secrets protection in rest with robust Secret Storage tool(-s)
• Secrets are transferred via REST HTTP which is generally not the best practice to protect secrets in transit (can be covered by protecting with TLS connection); access to secrets is only managed by FOLIO users/permissions model
• No dependency on type of container hosting FOLIO modules because of platform API usage
• This option is more about development then devopsing

Option 2 - Inject secrets during container creation

Draft visualization

Solution components

• FOLIO modules can read secrets from sidecar agent from well-known and standard place (e.g., Environment Variables, hereafter EV) and use them
• secret-storage agent is a jar-library supporting different container type details and move secrets from container-specific paths to EV
  • it is aware about specific details of particular container types and is able to read them
    • take secrets from environment variables for clean docker
    • use environment variable or mounted data volumes for Kubernetes Secrets
    • retrieve secrets from AWS SSM for cloud-based installation
  • pure Java enables both Vert.x and Spring support
  • potentially, it can be able to support additional functionality, e.g. runtime secret updates via event/callback mechanism if this feature is supported by container type, etc.
• pre-deployment script is a part of container building process which is responsible for retrieving secrets from particular Secret Storage and injecting them into containers
  • e.g., this https://github.com/folio-org/mod-search/blob/master/descriptors/ModuleDescriptor-template.json contains now a set of env parameters; the proposal is to have there placeholders only for those parameters that are considered as secrets, and replace placeholders with real secrets in a moment before real deployment
  • secret storage is a persistent storage for long-time keeping of secrets
• Devops / Administrator uses specific secret storage provider UI for secrets management

Note: one more idea here is to allow secret-storage agent to work directly with Secret Storage provider programmatically. Then no pre-deployment script will be needed.

Pros and Cons

• Good secrets protection in rest
• Much more limited access to secrets, no regular secrets transferring via HTTP which means better in transit protection
• Pre-deployment script should have an access to Secret Storage which means you need to safely store the key to the storage with all the keys...
• Looks like no place for dedicated UI, dedicated and aggregated audit logs, no ability to update secrets via platform API
• This option is more about devopsing then development

Proposed Solution

• secret-store-agent (or client) as a pure Java library available for both Vert.x and Spring
• Module edge-api-utils (https://github.com/folio-org/edge-api-utils) currently presenting common / shared Edge API utilities for edge-common and edge-common-spring can be used as a basis for secret-store-agent and be enriched with new functionality
• secret-store-agent provides an abstract unified interface (for connecting to Secret Store and work with secrets) and an extendable number of particular implementations
• CRUD support makes sense (see cases A and B from Craig's comment below)
• >> How does authentication and authorization fit into these models? For example, can every module access every secret? does each module access the secret store with different credentials?
  • secret-store-agent in every module should be configured separately
  • each module (more precisely - each secret-store-agent) can access the secret store with different credentials; this can be managed via
    
    • Vault Token https://www.vaultproject.io/docs/concepts/tokens which can be mapped to a set of one or more attached policies. In turn, these policies control what the token holder is allowed to do within Vault. Note that usage of roottoken in production is not recommended
    • AWS IAM role
  • another option is to manage access on application level, e.g. every module has own static key (or token, or salt) that is used as a part of secret key

• Multi-tenancy support - all the secrets are to be accessible per tenant level

  • it's possible to add secrets for a new tenant without module(-s) restart
• Secrets can be cached with defined TTL by secret-store-agent (under discussion - caching is not the best practice for secrets)
• Logging and audit
  • via native Secret Store provider capabilities
  • still FOLIO-centralized option is possible if need
• Certificates (e.g., for Kafka clients) are to be provided and mounted externally as JKS while secret-store-agent can be used to manage keystore & truststore location and password
  • Enabling SSL and ACL for Kafka

Next steps

• 10 May 2021 Share this document for very early review
• 18 May 2021 Find FOLIO devops expert(-s) to review some technical capabilities and ideas
  • Current approach implemented in edge-common still looks good
• 26 May 2021 On Tech Leads meeting:
  • how important and priority this task is?
  • present the work done and current state
  • collect known requirements to FOLIO secrets management
  • receive an early feedback on which of proposed options looks more preferable
• 09 Jun 2021 Collect feedback, remarks, notes and thoughts offline
  • Raman A: aggregate them, react on potential action items
• 09 Jun 2021 Tech Leads meeting
  
  • review updates
  • Proposed next steps
    • check with DevOps regarding Kafka certs configuration
    • rename once again (sorry for that) in secret-store-agent to be widely used
    • use MODORGSTOR-33 - Implement encryption of interface credentials DRAFT to add C.UD support

Other notes and thoughts

Use cases to be addressed

• Module A creates a new secret and saves it to Secret Storage
• Module B reads a secret from Secret Storage
• Module A updates an existing secret and saves it to Secret Storage; module B is able to read an updated secret
• Module A deletes an existing secret from Secret Storage; module B is able to identify the case and throw an error

Update Folio tenant with credentials - 2 options to consider

• central store of credentials with runtime reading - less downtime, less secure, need to support consumer notification if secret is updated
  
• mount key-chain on module launch (k8s or AWS), accessible locally - through module restart only, more secure

Decision

Implications

• Pros
  • N/A
• Cons
  • N/A

Other Related Resources

FOLIO-2278 - Spike: Secret Storage OPEN

MODORGSTOR-33 - Implement encryption of interface credentials DRAFT

MODINREACH-19 - SPIKE: Investigate possible methods to securely store key/secret pairs CLOSED

MODPUBSUB-78 - Insecure use of password for pub-sub user CLOSED

List of active topics (see Secret Storage item)

History of Changes