Changes list

...

References

/wiki/spaces/TLG/pages/753669 (see Secret Storage item)

Requirements

Below is the list of identified requirements ( To anyone reviewing this document: feel free to specify more requirements if you have any):

...

• certificate
• API key
• login, password
• ... what else

...

• HarshiCorp Vault (used for on-premises installations),
• AWS Systems Manager Parameter Store or AWS SSM (used for AWS cloud hosting),
• properties-file (for dev/test needs)

...

• Are there known benefits in comparison with manual secrets update by platform administrators? Should this be done programmatically?
• Are there specific requirements to runtime updating for secrets? I.e. should consumers be able to update secrets in runtime without restart/downtime?

...

• Is it required? Technically, both AWS SSM and Vault have their own management user interfaces, Rancher provides an option to manage secrets in form of environment variables in UI forms
  

...

•  Using the same credentials for every tenant is a kind of concern

...

• Access to secrets should/must be logged for further audit - any specific requirements?
• Can AWS SSM and Vault logs be efficient for this, or do we need to have any dedicated log storage for that?

...

Please refer to FOLIO secrets management - Requirements for a list of identified requirements.

Analysis

Current state

Three environment types should be considered:

...

Option 2 - Inject secrets during container creation

Draft visualization

Solution components

• FOLIO modules can read secrets from sidecar agent from well-known and standard place (e.g., Environment Variables, hereafter EV) and use them
• secret-storage agent is a jar-library supporting different container type details and move secrets from container-specific paths to EV
  • it is aware about specific details of particular container types and is able to read them
    • take secrets from environment variables for clean docker
    • use environment variable or mounted data volumes for Kubernetes Secrets
    • retrieve secrets from AWS SSM for cloud-based installation
  • pure Java enables both Vert.x and Spring support
  • potentially, it can be able to support additional functionality, e.g. runtime secret updates via event/callback mechanism if this feature is supported by container type, etc.
• pre-deployment script is a part of container building process which is responsible for retrieving secrets from particular Secret Storage and injecting them into containers
  • e.g., this https://github.com/folio-org/mod-search/blob/master/descriptors/ModuleDescriptor-template.json contains now a set of env parameters; the proposal is to have there placeholders only for those parameters that are considered as secrets, and replace placeholders with real secrets in a moment before real deployment
  • secret storage is a persistent storage for long-time keeping of secrets
• Devops / Administrator uses specific secret storage provider UI for secrets management

...

• secret-store-agent (or client) as a pure Java library available for both Vert.x and Spring
• Module edge-api-utils (https://github.com/folio-org/edge-api-utils) currently presenting common / shared Edge API utilities for edge-common and edge-common-spring can be used as a basis for secret-store-agent and be enriched with new functionality
• secret-store-agent provides an abstract unified interface (for connecting to Secret Store and work with secrets) and an extendable number of particular implementations
• CRUD support makes sense (see cases A and B from Craig's comment below)
• >> How does authentication and authorization fit into these models? For example, can every module access every secret? does each module access the secret store with different credentials?
  • secret-store-agent in every module should be configured separately
  • each module (more precisely - each secret-store-agent) can access the secret store with different credentials; this can be managed via
    
    • Vault Token https://www.vaultproject.io/docs/concepts/tokens which can be mapped to a set of one or more attached policies. In turn, these policies control what the token holder is allowed to do within Vault. Note that usage of root token in production is not recommended
    • AWS IAM role
  • another option is to manage access on application level, e.g. every module has own static key (or token, or salt) that is used as a part of secret key

• Multi-tenancy support - all the secrets are to be accessible per tenant level

  • it's possible to add secrets for a new tenant without module(-s) restart
• Secrets can be cached with defined TTL by secret-store-agent (under discussion - caching is not the best practice for secrets)
• Logging and audit
  • via native Secret Store provider capabilities
  • still FOLIO-centralized option is possible if need
• Certificates (e.g., for Kafka clients) are to be provided and mounted externally as JKS while secret-store-agent can be used to manage keystore & truststore location and password
  • Enabling SSL and ACL for Kafka

...

• 10 May 2021 Share this document for very early review
• 18 May 2021 Find FOLIO devops expert(-s) to review some technical capabilities and ideas
  • Current approach implemented in edge-common still looks good
• 26 May 2021 On Tech Leads meeting:
  • how important and priority this task is?
  • present the work done and current state
  • collect known requirements to FOLIO secrets management
  • receive an early feedback on which of proposed options looks more preferable
• 09 Jun 2021 Collect feedback, remarks, notes and thoughts offline
  • Raman A: aggregate them, react on potential action items
• 09 Jun 2021 Tech Leads meeting
  
  • review updates
  • Proposed next steps
    • check with DevOps regarding Kafka certs configuration
    • rename once again (sorry for that) in secret-store-agent to be widely used
    • use

      Jira Legacy
      server System JIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODORGSTOR-33

      to add C.UD support

...

Other notes and thoughts

Use cases to be addressed

...