FOLIO secrets management - Requirements

This page is aimed to consolidate and document all known requirements (both functional and non-functional) applicable to FOLIO secrets management.

Use cases

A user (administrator) wants to create/manage secrets through the secret storage directly and allow FOLIO modules to access those secrets (read-only access)
A user (administrator) wants to create/manage secrets through FOLIO. E.g., a module might need to store credentials to some external system (like FTP or SMTP server credentials, API keys etc. UI forms are involved as well for interaction with a user
A module wants to create/manage secrets so that a user is not even in the loop here, and the interaction is only between the module and secret storage. Examples are modules which are creating system/tenant users (like mod-pubsub), or a case with generating and storing signing keys in such a way that all instances of the mod-authtoken container can easily stay aligned.

Secret types

The following secret types are to be supported:

Secret type	Format	Who creates?	Who accesses?	How does it work now?	Additional comments
login & password	String	FOLIO resource administrator external system administrator	modules working with any resource with logo-pass	different modules use their own storages for that
API key	String	administrator system	modules working with APIs	different modules use their own storages for that
token	String	system	mod-authtoken modules working with tokens
SSL certificate	file	administrator	modules using certificate for work with Kafka		Cannot be stored in SSM, Vault etc. Encrypted S3?

Secret stores

The following secret stores are to be supported:

Secret store mechanism	Links	Environment & Comments
HarshiCorp Vault	https://www.vaultproject.io/	On-premises installations
AWS Systems Manager Parameter Store ( or just AWS SSM)	https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html	As an option for AWS cloud hosting
AWS Secrets Manager	https://aws.amazon.com/secrets-manager/	As an option for AWS cloud hosting
properties-file		For dev / test needs
environment variables		Standard EC2 machine with all dockers on it (native docker, so Env Var are only available) - IndexData (snapshots, testing)
Kubernetes Secrets		K8S Secrets with Env Var is used there to ensure compatibility with native docker deployment Rancher has UI for displaying & configuration

User interface for Secrets management

Home-grown unified web UI vs. store-specific UIs

Both AWS SSM and Vault have their own management user interfaces, Rancher provides an option to manage secrets in form of environment variables in UI forms.

Tech stack support

The following programming languages used on Folio platform are to be supported:

Java Spring way
Java Vert.x
Node.js (RA: I haven't met such modules/components by my own but have heard they exist - should they be considered?)
are there any other languages used?

Audit capabilities

Access to secrets must be logged for further audit

Option 1 - Use secret stores native audit capabilities
- https://docs.aws.amazon.com/systems-manager/latest/userguide/logging-and-monitoring.html
- https://www.vaultproject.io/docs/audit
- No development, just devops/hosting
Option 2 - FOLIO dedicated log storage
- e.g., based on mod-audit or a new one - just a unified way regardless of hosting details
- Development is required

Proposal - start now with option 1, may consider option 2 later if option 1 won't be enough.

Secret store access

Any particular FOLIO module should only access its own secrets

Notes:

check AWS ssm:GetParameters permission - it is the policy used to enforce access control and protect one service’s secrets from another (here https://aws.amazon.com/blogs/mt/the-right-way-to-store-secrets-using-parameter-store/ there is a pretty good overview of this process)
in Vault an access can be managed via roles, tokens etc. (https://www.vaultproject.io/docs/auth)

Multi-tenancy support

A tenant should only access its own secrets

Notes:

tenant are FOLIO internal term, so that it seems now that access verification on tenant level cannot be supported on Secret store level
tenant ID can be used as a part of secret name

Adding new secrets (even for a new tenant) should be available without restart

Secret update in runtime

Are there specific requirements to runtime updating for secrets? I.e. should consumers be able to update secrets in runtime without restart/downtime?

Caching secrets

Can / should secrets be cached in apps?

Concern is about storing secrets in memory - there's potential risk of data extracting from RAM, though this can be considered as a low risk / low impact
Caching reduces load on Secret Store, and improves performance & availability
- https://docs.aws.amazon.com/secretsmanager/latest/userguide/use-client-side-caching.html
- https://github.com/aws/aws-secretsmanager-caching-java - Java implementation of SSM caching client
- "To use your secrets most efficiently, you should not simply retrieve the secret value from Secrets Manager every time you need to use the credentials. Instead, use a supported Secrets Manager client component to cache your secrets and update them only when required because of rotation. AWS has created such client components for you and made them available as open-source.
- Note that caching introduce some challenges, e.g. time-to-live and notification about changes