/
FSSP-2 Spike - authorization investigation and design

FSSP-2 Spike - authorization investigation and design

Spike Overview

https://folio-org.atlassian.net/browse/FSSP-2

Objective: https://folio-org.atlassian.net/browse/FSSP-1 gave us a working implementation of the secure store proxy, but it’s not yet secure itself. Before we start using this we need to design a security model which ensures secrets are only furnished to those with appropriate authorization.

Background

Folio Secure Store Proxy (FSSP) is a Quarkus-based service designed to act as a secure proxy for accessing secrets stored in various back-end systems like AWS Systems Manager Parameter Store (SSM) or HashiCorp Vault. It is intended for use within the FOLIO platform to centralize and secure access to sensitive configuration data.

FSSP provides similar functionality that can be found in Secure Stores library but it also introduces a layer of data caching to improve communication between clients and secure store back-ends. Caching helps to address an issue of excessive cost of operations when the data is changed infrequently. There are some basic operations, like “list” and “invalidate” entries, supported by FSSP to manage cached data.

To seamlessly integrate FSSP into client’s code Secure Store library has another implementation of SecureStore interfaces. This implementation talks to FSSP as if it would be just another secure store back-end system. The only thing client needs to do to start using the proxy is to change secure store type to FSSP and define a URL of the running proxy instance in the configuration.

Problem Statement

The spike aims to address security concerns when using FSSP API to populate and retrieve secrets. Internally the proxy goes to a back-end system in secured manner: currently for Vault it uses a provided token, for AWS - either IAM role or ECS container credentials. But the exposed API is not protected, thus anyone who has access to FSSP can automatically get access to secured data.

The goal is to protect FSSP from unauthorized access. Any operation has to be requested by a client with appropriate permissions. This includes forwarding requests to the underlying back-end system as well as returning already cached data. The approach has to be also FIPS compliant.

Scope

In Scope

  • A mandatory authorization check must be executed for every request, regardless of its cache-hit status.

  • The evaluation should encompass both server-side (FSSP) and client-side implications and requirements.

  • An architectural approach to consider is the decoupling of the proxy's authentication and authorization (AuthN/AuthZ) mechanism from the native access controls of the underlying secure store (e.g., HashiCorp Vault, AWS SSM).

  • Conversely, an integrated model should be considered, wherein the proxy's AuthN/AuthZ mechanisms are tightly coupled with, or delegate to, the native access controls of the secure store.

  • A formal analysis of the available architectural options is required. This analysis must include the following for each option:

    • A detailed comparison of its advantages and disadvantages (pros and cons).

    • An estimation of the relative implementation effort, and complexity.

    • An assessment of ongoing operational overhead and security posture.

    • Finally, the analysis should conclude with a recommendation for the preferred approach, supported by a clear and comprehensive justification.

Authentication approaches supported by tools/frameworks

Quarkus

The Quarkus Security framework supports multiple authentication mechanisms. Some of them can be combine to archive better protection and/or provide several alternatives.

There are authentication mechanisms are built into Quarkus, while others require an extension to be added.

Built-in authentication mechanisms

Quarkus Security provides the following built-in authentication support:

Authentication mechanisms supported through extensions

Quarkus Security also supports the following authentication mechanisms through extensions:

Authentication mechanisms overview

Authentication Mechanism

Short Description

Benefits

Drawbacks

When to Use

Authentication Mechanism

Short Description

Benefits

Drawbacks

When to Use

HTTP Basic

Standard HTTP authentication where the client sends a Base64(username:password) string in the Authorization header on every request.

  • Simplicity: Trivial to implement and test.

  • Wide Support: Supported by virtually all HTTP clients and tools (curl, Postman, etc.).

  • Insecure over HTTP: Credentials are only base64 encoded, not encrypted. Must be used with HTTPS.

  • Stateless Credential Passing: Credentials sent with every request, increasing exposure risk.

  • Poor UX: Browser prompts are clunky; logout is difficult.

  • Simple M2M APIs: For internal machine-to-machine communication where simplicity is key.

  • Prototyping/Development: Quickly securing an endpoint during early development.

  • Legacy System Integration.

Form-Based

Traditional web authentication. The user submits credentials via an HTML form. The server validates them and establishes a session, usually managed by a cookie.

  • Familiar UX: Standard login experience for web users.

  • Stateful Control: Server-side sessions can be easily invalidated (e.g., for immediate logout).

  • Stateful by Nature: Can complicate scaling in distributed/cloud-native environments. Requires sticky sessions or a distributed session cache.

  • CSRF Vulnerability: Requires protection against Cross-Site Request Forgery (CSRF).

  • Not for APIs: well-suited for non-browser clients (SPAs, mobile, M2M).

  • Traditional Web Applications: Monolithic or stateful applications serving server-rendered HTML pages.

  • When you require fine-grained, server-side control over user sessions.

JWT (Bearer Token)

The client presents a self-contained JSON Web Token (JWT) in the Authorization: Bearer <token> header. The server validates the token's signature and claims.

  • Stateless & Scalable: Ideal for microservices and distributed systems as no server-side session is needed.

  • Decoupled: Authentication can be handled by a dedicated service that issues tokens.

  • Flexible: JWT claims can carry user roles/permissions, reducing database lookups.

  • Token Revocation is Hard: Once issued, tokens are valid until they expire. A stolen token is a security risk.

  • Key Management: Securely managing signing keys is critical.

  • Token Size: Can become large if too many claims are included.

  • Stateless REST APIs: The de facto standard for securing modern APIs.

  • Single-Page Applications (SPAs) and Mobile Apps.

  • Microservices architectures.

OIDC (OpenID Connect)

An identity layer built on OAuth 2.0. Authentication is delegated to an external Identity Provider (IdP) like Keycloak, Okta, Auth0, or Google.

  • Centralized Identity: Manages users, credentials, and policies in one place.

  • Enables SSO: Single Sign-On across multiple applications.

  • Enhanced Security: Offloads password storage and complex flows (e.g., MFA) to a specialized, hardened service.

  • Standardized: Interoperable across different providers and clients.

  • Complexity: Introduces an external dependency (the IdP) and requires understanding OAuth 2.0 flows.

  • Network Latency: Involves redirects and communication with the IdP, which can add minor latency.

  • Enterprise Applications: When integrating with corporate directories (LDAP/AD).

  • Modern Web/Mobile Apps: When you need robust user management, social logins, or SSO.

  • Any scenario where you want to avoid managing user credentials yourself.

Mutual TLS (mTLS)

Client-certificate authentication. Both the client and server present and validate X.509 certificates during the TLS handshake to authenticate each other.

  • Very High Security: Strong, cryptographically-verified identity for both parties.

  • No Transmitted Credentials: Authentication occurs at the transport layer, not the application layer.

  • Non-interactive: Excellent for automated M2M communication.

  • PKI Management Overhead: Managing a Public Key Infrastructure (issuing, distributing, and revoking client certificates) is complex and operationally intensive.

  • High-Security B2B Integrations: Where strong, provable client identity is required.

  • Regulated Environments: Common in finance (Fintech) and government.

  • Service Mesh: Securing inter-service communication within a cluster (e.g., Istio, Linkerd).

WebAuthn (Web Authentication)

A W3C standard for passwordless authentication using public-key cryptography, typically via biometrics (Touch/Face ID) or hardware security keys (YubiKey).

  • Phishing Resistant: Eliminates the primary vector for credential theft.

  • Strong Security: Uses public-key crypto, which is much stronger than passwords.

  • Improved UX: Offers a fast, passwordless login experience.

  • Modern Browser/Platform Dependent: Requires support from the user's browser and OS.

  • Implementation Complexity: The registration and authentication ceremonies are more complex than simple forms.

  • Account Recovery: Requires well-designed recovery mechanisms if a user loses their authenticator.

  • High-Security Consumer Applications: Protecting sensitive user data (e.g., banking, health).

  • Any application aiming to provide a modern, passwordless, and highly secure login experience for end-users.

Potential candidates from the above to be considered for FSSP

  • Mutual TLS authentication

  • JWT authentication

  • OpenID Connect authentication

HashiCorp Vault

Authentication in Vault is the process by which user or machine supplied information is verified against an internal or external system. In most cases, Vault will delegate the authentication administration and decision to the relevant configured external auth method (e.g., Amazon Web Services, GitHub, Google Cloud Platform, Kubernetes, Microsoft Azure, Okta etc.). Upon authentication, a token is generated. This token is conceptually similar to a session ID on a website. The token may have attached policy, which is mapped at authentication time.

Authentication is performed by Auth methods. These methods are the components in Vault that perform authentication and are responsible for assigning identity and a set of policies to a user. Auth methods can be enabled/disabled using the CLI or the API. When enabled, auth methods are mounted within the Vault mount table and can be accessed and configured using the standard read/write API. By default, auth methods are mounted to auth/<type> (example for GitHub - auth/github).

Authentication mechanisms overview

Authentication Mechanism

Short Description

Benefits

Drawbacks

When to Use

Authentication Mechanism

Short Description

Benefits

Drawbacks

When to Use

AWS

Allows AWS entities (EC2, Lambda, IAM users/roles) to authenticate with Vault. The client provides cryptographically signed AWS identity information (GetCallerIdentity request), which Vault verifies with AWS.

  • No Secret Zero: Applications running on AWS don't need a pre-configured secret to start.

  • Strong Identity: Leverages the robust and mature AWS IAM system.

  • Highly Automated: Ideal for ephemeral infrastructure and CI/CD pipelines within AWS.

  • Platform Lock-in: Tightly coupled to the AWS ecosystem; not usable outside of it.

  • Complex Configuration: Requires careful setup of IAM policies on the AWS side to grant necessary permissions.

  • Primary choice for applications running within AWS. This includes services deployed on EC2 instances, in ECS/EKS containers, or as Lambda functions.

JWT/OIDC

Authenticates clients that provide a JSON Web Token (JWT) signed by a trusted OIDC provider. Vault validates the JWT signature against the provider's public keys (JWKS) and checks claims within the token.

  • Standard-Based: Interoperable with any OIDC-compliant Identity Provider (IdP) like Keycloak, Okta, Auth0, etc.

  • Centralized Identity: Leverages an existing IdP for user and application identity, avoiding credential silos.

  • Flexible: Policies can be mapped to JWT claims, enabling fine-grained access control.

  • Dependency: Requires a well-managed and highly available OIDC/JWT provider.

  • Configuration Complexity: Setting up roles, claim mappings, and bound audiences can be intricate.

  • Integrating with an existing IdP (like Keycloak) for human or application authentication.

  • Kubernetes: Authenticating applications using their Kubernetes Service Account tokens.

  • CI/CD Pipelines: Authenticating jobs from systems like GitHub Actions or GitLab CI.

TLS Certificates

Authenticates clients via a mutual TLS (mTLS) handshake. The client presents a TLS certificate, and Vault verifies it was signed by a trusted Certificate Authority (CA). Identity is typically mapped from the certificate's Common Name (CN) or Subject Alternative Name (SAN).

  • High Security: Based on strong public-key cryptography

  • Automated M2M: Excellent for non-interactive machine-to-machine authentication

  • Leverages Existing PKI: Integrates well into environments that already have a Public Key Infrastructure.

  • Operational Overhead: Requires a robust PKI to manage certificate issuance, rotation, and revocation, which can be a significant undertaking if not already in place.

  • Securing service-to-service communication in a service mesh (e.g., Consul Connect).

  • Environments where a strong, managed PKI is already a core part of the security infrastructure.

  • Any scenario requiring strong, non-repudiable machine identity.

Tokens

The core authentication mechanism. Every other auth method, upon success, ultimately returns a token. A token is a string used in the X-Vault-Token header for API requests. They have TTLs, policies, and can be revoked.

  • Fundamental & Flexible: The core of Vault's security model.

  • Scoped & Ephemeral: Can be created with short lifespans and limited privileges, minimizing the impact of a leak.

  • Hierarchical: Parent tokens can create child tokens with a subset of permissions.

  • The "Secret Zero" Problem: Manually managing tokens for initial application access is an anti-pattern. The key challenge is securely delivering the first token.

  • Risk of Leakage: Long-lived or root tokens are extremely sensitive and represent a major security risk if mishandled.

  • Not a primary bootstrap method. It is the result of authentication from other methods.

  • Humans: For direct CLI or UI access during development or for break-glass operational tasks (always with short TTLs).

  • Initial Setup: The root token is required to configure Vault initially.

Username & Password

A straightforward method where username/password pairs are stored directly within Vault. Vault hashes and salts the passwords for storage.

  • Simple & Familiar: Easy to understand and quick to set up.

  • Self-Contained: No external dependencies are required.

  • Static Credentials: Long-lived passwords are a significant security liability.

  • No Federation: Creates a silo of identities separate from a corporate IdP (like LDAP/AD or OIDC).

  • Generally Discouraged: Considered a legacy approach and not recommended for production applications.

  • Human Operators: For simple Vault UI login scenarios where an IdP is not integrated.

  • Development & Testing: Quick bring-up for local development environments.

  • Legacy Systems: For applications that cannot be updated to use a more modern, dynamic authentication method.

Potential candidates from the above to be considered for pass-through authentication (delegating from FSSP to Vault and then to underlying authentication engine)

  • AWS authentication

  • JWT/OIDC authentication

  • TLS Certificates authentication (although it can be tricky to pass the certificates)

AWS Systems Manager

Authentication for accessing secrets stored in AWS Systems Manager (SSM) Parameter Store is fundamentally different from HashiCorp Vault's pluggable model. Access control is not handled by SSM itself but is exclusively delegated to and governed by AWS Identity and Access Management (IAM).

Any client (a user, an application running on EC2, a Lambda function, etc.) attempting to perform an action on a parameter (e.g., ssm:GetParameter) must present valid AWS credentials. IAM then evaluates the policies attached to the client's identity (the IAM Role or User) to determine if the action is permitted on the requested resource (the specific parameter ARN).

The AWS SDK and CLI abstract away the complexity of this process through the Default Credential Provider Chain. This chain automatically searches for credentials in a sequence of standard locations (e.g., environment variables, instance metadata), making the developer experience seamless. The "authentication mechanisms" below, therefore, refer to the different ways these IAM credentials can be supplied to a client.

Authentication mechanisms overview

Authentication Mechanism

Short Description

Benefits

Drawbacks

When to Use

Authentication Mechanism

Short Description

Benefits

Drawbacks

When to Use

IAM Roles for AWS Compute

An application running on an AWS compute service (e.g., EC2, ECS, Lambda, EKS) assumes an IAM Role. The AWS SDK automatically retrieves temporary, auto-rotated credentials from the local AWS metadata service. No credentials need to be hard-coded in the application.

  • Best Practice Security: Avoids static, long-lived credentials, eliminating the "Secret Zero" problem for applications running on AWS.

  • Automatic Rotation: Credentials are short-lived and managed by AWS.

  • Tightly Integrated & Auditable: Leverages native AWS identity and logging capabilities.

  • Platform Specific: This mechanism only works for code running within the AWS ecosystem.

  • Configuration: Requires careful setup of the IAM Role and its trust policy to grant the correct permissions.

  • The primary and recommended method for any application deployed within AWS. It is the most secure and manageable approach for service-to-service authentication.

IAM User Access Keys

A developer or system is configured with a long-lived IAM User Access Key ID and Secret Access Key. These are typically provided to the client application via environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) or a shared credentials file (~/.aws/credentials).

  • Simple Setup: Easy to generate and configure for initial access.

  • Location Independent: Works from any machine with an internet connection, including local developer laptops and on-premise servers.

  • High Security Risk: These are static, long-lived credentials. If leaked, they provide a persistent entry point for an attacker.

  • Manual Management: Rotation is a manual process and is often neglected, leading to security vulnerabilities.

  • Discouraged for Applications: Considered an anti-pattern for production services.

  • Local Development: For individual developers running code on their local machines.

  • Legacy Systems: For applications running outside of AWS that cannot use a more modern federation method.

  • Initial CLI/SDK access from a trusted machine.

IAM Roles via Web Identity Federation (OIDC)

An external system (e.g., a CI/CD pipeline like GitHub Actions, a Kubernetes cluster, or an IdP like Keycloak) authenticates the client and provides it with a standard OIDC JSON Web Token (JWT). The client then exchanges this JWT for temporary, short-lived AWS credentials by calling the AWS STS AssumeRoleWithWebIdentity API.

  • Eliminates Static Keys: Allows external systems to access AWS resources without storing long-lived IAM user keys.

  • Standards-Based: Leverages the OIDC standard for trusted, federated identity.

  • Fine-Grained & Auditable: The assumed role can be narrowly scoped, and access is tied to the federated identity.

  • Configuration Complexity: Requires setting up a trust relationship in IAM between AWS and the external OIDC identity provider.

  • Dependency: Relies on the availability and security of the external OIDC provider.

  • CI/CD Pipelines: Granting systems like GitHub Actions or GitLab CI secure, keyless access to deploy resources or fetch secrets.

  • Kubernetes Workloads: Allowing pods to assume an IAM role using their Kubernetes service account token.

  • User Federation: Allowing users from a corporate IdP to assume a role to access AWS resources.

Potential candidates from the above to be considered for FSSP

  • IAM Roles for AWS Compute

Proposed Solutions

Option 1: Mutual TLS between client and FSSP

Overview

Standard TLS secures communication by having the server prove its identity to the client using a certificate. Mutual TLS (mTLS) extends this by requiring the client to also prove its identity to the server with a certificate. This creates a two-way, cryptographically-verified authentication channel.

Authentication is achieved as follows:

  1. When a client connects to FSSP service, the service presents its server certificate (as in normal TLS).

  2. Crucially, the service then demands that the client present its own certificate.

  3. The client provides its certificate.

  4. FSSP validates the client's certificate by checking if it was signed by a trusted Certificate Authority (CA) whose certificate is present in the server's truststore.

If the client certificate is valid and trusted, the client is considered authenticated. Its identity is derived directly from the certificate's attributes (e.g., the Common Name), which can then be used for authorization (i.e., mapping the identity to a role). This eliminates the need for passwords, API keys, or JWTs for authentication. The approach also minimize code changes on both sides since authentication is done on transport layer.

Sequence Diagram

The following diagram illustrates the mTLS handshake and subsequent authorization process between a client and FSSP.

Configuration Changes

To enable mTLS, configuration is required on both the client and FSSP side.

On FSSP side

FSSP needs a keystore to hold its own identity certificate and a truststore to hold the public certificates of trusted clients or the CAs that sign them.

It’s also crucial to disable the plain HTTP protocol, requiring all requests to use HTTPS.

Sample FSSP configuration:

  • application.properties

# enable SSL on port 8443 quarkus.http.ssl.port=8443 ## configure certificates quarkus.http.ssl.certificate.key-store-file=server-keystore.p12 quarkus.http.ssl.certificate.key-store-password=the_key_store_secret quarkus.http.ssl.certificate.trust-store-file=server-truststore.p12 quarkus.http.ssl.certificate.trust-store-password=the_trust_store_secret ## makes the server demand client certificates quarkus.http.ssl.client-auth=required ## disables the plain HTTP protocol, requiring all requests to use HTTPS quarkus.http.insecure-requests=disabled ## defines a policy where only authenticated users can access resources quarkus.http.auth.permission.certauthenticated.paths=/* quarkus.http.auth.permission.certauthenticated.policy=role-policy-cert ## only users with "user", "admin" role can have access quarkus.http.auth.policy.role-policy-cert.roles-allowed=user,admin ## file with a map of certificate’s CN values to roles quarkus.http.auth.certificate-role-properties=cert-role-mappings.properties

  • cert-role-mappings.properties

## CN name fssp-admin maps to "user","admin" roles fssp-admin=user,admin ## CN name fssp-user maps to "user" role only fssp-user=user

On Client side

The client must be configured with its own keystore containing its private key and certificate.

If a client is arbitrary Quarkus application the configuration will look like the following

# enable SSL on port 8443 quarkus.http.ssl.port=8443 ## configure certificates quarkus.http.ssl.certificate.key-store-file=client-keystore.p12 quarkus.http.ssl.certificate.key-store-password=the_key_store_secret quarkus.http.ssl.certificate.trust-store-file=client-truststore.p12 quarkus.http.ssl.certificate.trust-store-password=the_trust_store_secret

If a client is FsspStore then its configuration has to be extended with keystore parameters and will look like the following:

application: secret-store: type: FSSP fssp: address: ${SECRET_STORE_FSSP_ADDRESS:} secret-path: ${SECRET_STORE_FSSP_SECRET_PATH:secure-store/entries} enable-ssl: ${SECRET_STORE_FSSP_ENABLE_SSL:false} trust-store-path: ${SECRET_STORE_FSSP_TRUSTSTORE_PATH:} trust-store-file-type: ${SECRET_STORE_FSSP_TRUSTSTORE_FILE_TYPE:} trust-store-password: ${SECRET_STORE_FSSP_TRUSTSTORE_PASSWORD:} ## new configuration key-store-path: ${SECRET_STORE_FSSP_KEYSTORE_PATH:} key-store-file-type: ${SECRET_STORE_FSSP_KEYSTORE_FILE_TYPE:} key-store-password: ${SECRET_STORE_FSSP_KEYSTORE_PASSWORD:}

Code Changes

On FSSP side

FSSP should enforce roles within its application code. This is the crucial final step that actually protects the business logic.

Standard Jakarta Security annotation @RolesAllowed can be used to implement Role-Based Access Control (RBAC) directly on resource endpoints.

Sample code with resources protected by roles:

@Path("/api/data") public class DataResource { // Injects the security context to access information about the authenticated client. @Context SecurityContext securityContext; /** * An endpoint to read data. * Access is granted to any client whose certificate maps to either the "user" OR "admin" role. */ @GET @RolesAllowed({"user", "admin"}) @Produces(MediaType.TEXT_PLAIN) public String readData() { String principalName = securityContext.getUserPrincipal().getName(); // The principal name will be the full Subject DN from the client certificate. // In a real application, this would call the SecureStore. return "Data read successfully by: " + principalName; } /** * An endpoint to write data. * Access is restricted ONLY to clients whose certificate maps to the "admin" role. */ @POST @RolesAllowed("admin") @Consumes(MediaType.APPLICATION_JSON) public String writeData(String data) { String principalName = securityContext.getUserPrincipal().getName(); // Logic to write data to the SecureStore... return "Data written successfully by admin: " + principalName; } }

On Client side

FsspStore should take into account new configuration parameters for keystore. These parameters should be reflected in FsspConfigProperties. During FsspStore initialization configured keystore should be loaded and passed to SSL Context:

image-20250709-153956.png

Generating a Client Certificate (PKCS12)

openssl tool can be used to generate a key and a self-signed certificate, then package them into a PKCS12 (.p12) file, which is a common format for Java keystores.

Step 1: Generate a private key and a self-signed certificate.
This command creates client.key and client.crt. The -subj flag sets the Distinguished Name, which is critical for role mapping.

# For an admin certificate openssl req -x509 -newkey rsa:4096 -nodes \ -keyout admin.key -out admin.crt -sha256 -days 365 \ -subj "/CN=fssp-admin/OU=services/O=MyOrg/C=US"

Step 2: Package the key and certificate into a PKCS12 file.
This bundles admin.key and admin.crt into admin-keystore.p12, which the client application can use.

openssl pkcs12 -export -in admin.crt -inkey admin.key \ -name admin-alias \ -out admin-keystore.p12

Pros and Cons

Pros

Cons

Pros

Cons

High Security: Authentication is based on strong, asymmetric cryptography. Certificate private keys never leave the client.

Operational Overhead: Requires a robust Public Key Infrastructure (PKI) to issue, distribute, rotate, and revoke certificates. This is a significant undertaking.