...
because mod-scheduler’s sidecar gets x-okapi-token per timer request
...
Keycloak Resource and Sidecar Issues
Keycloak struggles to store all resources required for authorization when we have more than 10 realms. Each realm contains approximately 1,500 resources, and the default cache size of 10,000 was insufficient. To address this, we increased the cache size to 80,000 items.
Additionally, we observed excessive overhead caused by requests to Keycloak using incorrect credentials. This occurs because sidecars fail to retrieve passwords from the AWS Parameter Store.
...
Fixes Implemented:
SSM Issues:
Problem: Sidecars attempt to retrieve passwords for system users even when it's unnecessary. This happens because the
x-okapi-user-idheader does not include a user ID.Root Cause: Modules fail to copy incoming request headers into outgoing requests.
Solution: Updated system user validation logic in the sidecar to avoid unnecessary password retrievals.
Potential Impact: This fix may affect other flows, requiring additional testing.
Unnecessary Password Retrievals for Modules:
Problem: Sidecars attempt to retrieve system user passwords for modules like
mod-scheduler, where such requests are unnecessary.
AWS Parameter Store Limits:
Problem: Sidecars are unable to retrieve passwords from SSM due to the rate limit of 40 requests per second. Each sidecar makes password requests every 300 seconds for all system users across all tenants, often at the same time.
Example Calculation:
Number of sidecars: ~70
Number of system users per sidecar: ~16
Number of tenants: ~15
Total requests: ~16,800 requests every 300 seconds
Conclusion
Refining the logic for system user password retrieval and ensuring proper validation will significantly reduce unnecessary calls to the AWS Parameter Store, improving overall efficiency.
...