...
Currently, we have seen that sidecars are sometimes unable to retrieve the parameters from the AWS parameter store to retrieve necessary information, like passwords for system users, which causes the problem that it can’t work properly.
Keycloak Resource and Sidecar Issues
Keycloak struggles to store all resources required for authorization when we have more than 10 realms. Each realm contains approximately 1,500 resources, and the default cache size of 10,000 was insufficient. To address this, we increased the cache size to 80,000 items.
Additionally, we observed excessive overhead caused by requests to Keycloak using incorrect credentials. This occurs because sidecars fail to retrieve passwords from the AWS Parameter Store.
Problem Statement
During our investigation into the sidecar performance issue related to authorization in Keycloak, we observed that some sidecars cannot authorize with Keycloak to retrieve the system user token. Upon further analysis, we discovered that these sidecars fail to retrieve passwords for system users from the AWS Parameter Store because they exceed the allowed rate limit, resulting in a "Rate Exceeded" exception.
...
because mod-scheduler’s sidecar gets x-okapi-token per timer request
...
Keycloak Resource and Sidecar Issues
Keycloak struggles to store all resources required for authorization when we have more than 10 realms. Each realm contains approximately 1,500 resources, and the default cache size of 10,000 was insufficient. To address this, we increased the cache size to 80,000 items.
...
Additionally, we observed excessive overhead caused by requests to Keycloak using incorrect credentials. This occurs because sidecars fail to retrieve passwords from the AWS Parameter Store.Fixes Implemented:
SSM Issues:
...