Changes list
| Raman Auramau | Proposed solution is finally documented for review |
| Raman Auramau | Added some details about current FOLIO deployment options |
| Raman Auramau | Re-structure the document, add option 2 |
| Raman Auramau | Added some requirements, current investigation details, drafted an option with a dedicated module |
| Raman Auramau | Initial document |
Problem statement
Introduction
A secret is anything to which you want to tightly control access, such as API keys, passwords, certificates, and other sensitive information. Encrypted storage is needed to store secrets in a safe way, as well as a service to provide secrets management capabilities.
At the moment the FOLIO platform has neither dedicated secret storage as a common module or service nor even a standard approach for secrets management.
Note: edge-common has an interface for that, there are 3 implementations as for now - Vault, AWS, and an ephemeral one for dev needs (properties file), with read access implemented only. So, this piece of edge-common can be leveraged as some basic experience though it's definitely not enough.
With that, this document is aimed to provide analysis details and answer the following questions:
- What technologies would be appropriate for this?
- What might a solution look like? Is there some abstraction layer/interface allowing for different implementations?
- What's required for a module to store/retrieve a secret?
- What's required on the infrastructure side?
References
/wiki/spaces/TLG/pages/753669 (see Secret Storage item)
Requirements
Below is the list of identified requirements (
To anyone reviewing this document: feel free to specify more requirements if you have any):
- Secret types to be supported
- certificate
- API key
- login, password
- ...
what else
- Particular implementations for following secret storage providers (with capabilities to extend in future)
- HarshiCorp Vault (used for on-premises installations),
- AWS Systems Manager Parameter Store or AWS SSM (used for AWS cloud hosting),
- properties-file (for dev/test needs)
- Unified interface covering needs in secrets management
- Secrets CRUD
- Are there known benefits in comparison with manual secrets update by platform administrators? Should this be done programmatically?
- Are there specific requirements to runtime updating for secrets? I.e. should consumers be able to update secrets in runtime without restart/downtime?
- Secrets management UI
- Is it required? Technically, both AWS SSM and Vault have their own management user interfaces, Rancher provides an option to manage secrets in form of environment variables in UI forms
-
- Both Spring-Way and Vert.x-based modules
- Multi-tenancy support
Using the same credentials for every tenant is a kind of concern
- Audit capabilities
- Access to secrets should/must be logged for further audit - any specific requirements?
- Can AWS SSM and Vault logs be efficient for this, or do we need to have any dedicated log storage for that?
- ...
- ... anything else to be added?
- ...
- ... Need to review the vision and requirements with Tech Leads
Analysis
Current state
Three environment types should be considered:
- Standard EC2 machine with all dockers on it (native docker, so Env Var are only available) - IndexData (snapshots, testing)
- Kubernetes (rancher envs)
- K8S Secrets with Env Var is used there to ensure compatibility with the previous one
- Rancher has UI for displaying & configuration
- AWS Cloud - SSM is used; application knows secret alias, IAM role is set on EC2 level
Sample
https://github.com/folio-org/edge-oai-pmh/tree/master/src/main/resources
https://github.com/folio-org/mod-search/blob/master/descriptors/ModuleDescriptor-template.json
Spring Cloud Config Server
https://cloud.spring.io/spring-cloud-config/reference/html/ Spring Cloud Config Server supports Git/SVN, Vault, JDBC database, Redis, AWS S3 as a backend for configuration properties but not AWS SSM. With that, AWS SSM is an Amazon native secret storage, and its usage for cloud-based deployment looks pretty justified. There is still particular way for thttps://rieckpil.de/resolving-spring-boot-properties-using-the-aws-parameter-store-ssm/.
Kubernetes Secrets
https://kubernetes.io/docs/concepts/configuration/secret/ - Kubernetes Secrets is a resource which is mounted into containers with a new volume type. It enables storing and managing secrets in safer and more flexible way than just putting them verbatim in a Pod definition or in a container image. Secrets can be
- mounted as data volumes or
- mounted Secrets are updated automatically
- exposed as environment variables or
- if a container already consumes a Secret in an environment variable, a Secret update will not be seen by the container unless it is restarted. There are third party solutions for triggering restarts when secrets change
- using imagePullSecrets (i.e. by the kubelet when pulling images for the Pod)
This mechanism looks to be really convenient and efficient to deliver secrets into containers under Kubernetes. At the same time, this is Kubernetes-specific feature, and it does not fit any containers rather than under Kubernetes managements. As far as I could find Kubernetes should not be considered as the only container orchestration tool.
Alternatives for evaluation
Option 1 - Introduce mod-secret-storage and dedicated platform API
The idea of this proposal is to introduce a new dedicated back-end module which will be able to work with a number of specific secret storages providers (from one side) and provide generic platform API for consumers (from the another one).
Draft visualization
Solution components
- FOLIO modules read secrets using sidecar agent
- secret-storage agent is a sidecar jar-library to facilitate communication with mod-secret-storage
- pure Java enables both Vert.x and Spring support
- operates with mod-secret-storage via platform API
- different update support strategies can be implemented - e.g., keep secrets in RAM with some time-to-live or request them from mod-secret-storage every time
- mod-secret-storage is a dedicated module encapsulating and hiding particular Secret Storage provider details and providing a generic platform API for secrets management
- it can also provide additional functionality like time-to-live and expiration management, enforce rotation, provide additional level of permissions verification, etc. - up to requirements
- can support CRUD operations for secrets
- ui-secret-storage is a module providing UI for secrets management
- if need; can be added at any time
- secret storage is a persistent storage for long-time keeping of secrets
- Devops / Administrator can use either dedicated UI and secret storage provider UI for secrets configuration and management
Pros and Cons
mod-secret-storage provides a good abstraction layer so that AWS SSM, Vault or any other secret storage can be introduced and used behind it- Fully FOLIO in-house implementation enables full control and any required customization, with centralized audit log etc.,
though it will require certain ownership and maintenance - Good secrets protection in rest with robust Secret Storage tool(-s)
- Secrets are transferred via REST HTTP which is generally not the best practice to protect secrets in transit (can be covered by protecting with TLS connection); access to secrets is only managed by FOLIO users/permissions model
- No dependency on type of container hosting FOLIO modules because of platform API usage
- This option is more about development then devopsing
Option 2 - Inject secrets during container creation
Draft visualization
Solution components
- FOLIO modules can read secrets from sidecar agent
from well-known and standard place (e.g., Environment Variables, hereafter EV) and use them - secret-storage agent is a jar-library supporting different container type details
and move secrets from container-specific paths toEV- it is aware about specific details of particular container types and is able to read them
- take secrets from environment variables for clean docker
- use environment variable or mounted data volumes for Kubernetes Secrets
- retrieve secrets from AWS SSM for cloud-based installation
- pure Java enables both Vert.x and Spring support
- potentially, it can be able to support additional functionality, e.g. runtime secret updates via event/callback mechanism if this feature is supported by container type, etc.
- it is aware about specific details of particular container types and is able to read them
- pre-deployment script is a part of container building process which is responsible for retrieving secrets from particular Secret Storage and injecting them into containers
- e.g., this https://github.com/folio-org/mod-search/blob/master/descriptors/ModuleDescriptor-template.json contains now a set of env parameters; the proposal is to have there placeholders only for those parameters that are considered as secrets, and replace placeholders with real secrets in a moment before real deployment
- secret storage is a persistent storage for long-time keeping of secrets
- Devops / Administrator uses specific secret storage provider UI for secrets management
Note: one more idea here is to allow secret-storage agent to work directly with Secret Storage provider programmatically. Then no pre-deployment script will be needed.
Pros and Cons
- Good secrets protection in rest
- Much more limited access to secrets, no regular secrets transferring via HTTP which means better in transit protection
- Pre-deployment script should have an access to Secret Storage which means you need to safely store the key to the storage with all the keys...
- Looks like no place for dedicated UI, dedicated and aggregated audit logs, no ability to update secrets via platform API
- This option is more about devopsing then development
Proposed solution
The proposal is based on initially suggested options evaluation and provided comments and thoughts from Tech Leads.
- secret-store-agent (or client) as a pure Java library available for both Vert.x and Spring
- Module edge-api-utils (https://github.com/folio-org/edge-api-utils) currently presenting common / shared Edge API utilities for edge-common and edge-common-spring can be used as a basis for secret-store-agent and be enriched with new functionality
- secret-store-agent provides an abstract unified interface (for connecting to Secret Store and work with secrets) and an extendable number of particular implementations
- CRUD support makes sense (see cases A and B from Craig's comment below)
- >> How does authentication and authorization fit into these models? For example, can every module access every secret? does each module access the secret store with different credentials?
- secret-store-agent in every module should be configured separately
- each module (more precisely - each secret-store-agent) can access the secret store with different credentials; this can be managed via
- Vault Token https://www.vaultproject.io/docs/concepts/tokens which can be mapped to a set of one or more attached policies. In turn, these policies control what the token holder is allowed to do within Vault. Note that usage of root token in production is not recommended
- AWS IAM role
- another option is to manage access on application level, e.g. every module has own static key (or token, or salt) that is used as a part of secret key
Multi-tenancy support - all the secrets are to be accessible per tenant level
- it's possible to add secrets for a new tenant without module(-s) restart
- Secrets can be cached with defined TTL by secret-store-agent (under discussion - caching is not the best practice for secrets)
- Logging and audit
- via native Secret Store provider capabilities
- still FOLIO-centralized option is possible if need
- Certificates (e.g., for Kafka clients) are to be provided and mounted externally as JKS while secret-store-agent can be used to manage keystore & truststore location and password
Next steps
- Share this document for very early review
- Find FOLIO devops expert(-s) to review some technical capabilities and ideas
- Current approach implemented in edge-common still looks good
- On Tech Leads meeting:
- how important and priority this task is?
- present the work done and current state
- collect known requirements to FOLIO secrets management
- receive an early feedback on which of proposed options looks more preferable
- Collect feedback, remarks, notes and thoughts offline
- Raman A: aggregate them, react on potential action items
- Tech Leads meeting
- review updates
- Proposed next steps
- check with DevOps regarding Kafka certs configuration
- rename once again (sorry for that) in secret-store-agent to be widely used
- use to add C.UD support
Other notes and thoughts
Use cases to be addressed
- Module A creates a new secret and saves it to Secret Storage
- Module B reads a secret from Secret Storage
- Module A updates an existing secret and saves it to Secret Storage; module B is able to read an updated secret
- Module A deletes an existing secret from Secret Storage; module B is able to identify the case and throw an error
Update Folio tenant with credentials - 2 options to consider
- central store of credentials with runtime reading - less downtime, less secure, need to support consumer notification if secret is updated
- mount key-chain on module launch (k8s or AWS), accessible locally - through module restart only, more secure