Page Comparison

...

• Egress request routing from mod-scheduler

• Requests to both public and system interfaces

• Changes/enhancements to Eureka core components (i.e module sidecars, and/or mod-scheduler, and/or Kong)

Out of Scope

• …

Research Questions

1. How should mod-scheduler calls be routed to system interfaces?

2. What is the relative effort and complexity for each of the solutions?

...

...

2. scheduled job is triggered for /timer-url-A endpoint. mod-scheduler prepares a request and put impersonated token for the system user into x-okapi-token request header. Then the request goes from mod-scheduler to its sidecar, which in turns forwards the request to Kong, since the request URL is not registered inside sidecar’s egress routing table

   1. Kong receives the request and searches for known routes associated with the given URL and method.

      • Once the route is found, the request will be forwarded to target Module A sidecar

   2. Module A sidecar receives the request and as usual performs several steps to authorize request:

      • get token from request header

      • parse token

      • call Keycloak to evaluate permissions. Since the system user has access to all resources, authorization will be successful

   3. Finally Module A sidecar calls /timer-url-A interface of the module and the chain of calls succeeds

3. External actor (UI or another system) attempts to request /timer-url-A system interface.

   1. Kong performs the same steps to handle the request as it does in case of a call from mod-scheduler:

      • the route is searched in the list of registered routes

      • once the route is found, the request will be forwarded to target Module A sidecar

   2. Module A sidecar receives the request and tries to authorize request in Keycloak

      • external actor has no permission to access /timer-url-A system interface, thus Keycloak will reject the call with "403 Forbidden" error.

Implementation details

The system already supports the above flow except one piece: request authorization for _timer interfaces in sidecar. Currently timer interfaces are treated as system ones, along with _tenant interfaces. For system interfaces in general sidecar skips token analyses and authorization procedures, but from now on it shouldn’t do this for timer interfaces.

The logic to verify user token and access grants resides inside classes called filters. They have shouldSkip(RoutingContext rc) method to understand whether or not a filter should be applied to particular request. Method example from KeycloakAuthorizationFilter:

...

So to enable authorization for timer interfaces shouldSkip(RoutingContext rc) method should be modified in several filter:

• KeycloakJwtFilter

• KeycloakTenantFilter

• KeycloakAuthorizationFilter (the main one responsible for sending authorization request to Keycloak)

Pros

• the implementation is simple. it will affect only sidecar code

• protects timer interfaces in a standard way via Keycloak authorization, like all other resources

• can be extended later to support more accurate resource-based access as opposite to “all resources“ access

Cons

• system user has unrestricted access to all system resources. Ideally it would be more correct to have some dedicated user and role (like “Run scheduled jobs”) with access to timer interfaces only. Right now it’s not that easy to implement because timers usually do not require any permissions and it’s tricky to build the right capabilities for them and automatically assign to some role

...

Option 2

Have mod-scheduler send egress requests to it’s sidecar like every other module, and add a switch to the module-sidecar which indicates it should retrieve ALL bootstrap info at startup, and consume all discovery events.process all scheduled job events to build and maintain routing information. These information should be stored per tenant basis. It also has to be saved to DB so that after sidecar restarting jobs requests can be handled properly.

Pros

• No special handling required in Kong

• No security concerns

Cons

• the most complicated solution

• the sidecar needs to retrieve and manage discovery and interface/endpoint information for all scheduled job APIs (system / public interfaces) in the system.

• scheduler specific logic introduced in sidecar code which is of general purpose

• potential grow of memory consumption due to increased volume of routing information

• sidecar would need to manage a permanent storage

• partial intersection of responsibilities with Kong, it already manages routes for each tenant. Sidecar would need to do the same but for a smaller group of routes

Option 3

Sidecars dynamically retrieve discovery information on an as-needed basis. If this fails, fallback to routing the request to Kong. Maybe this requires a new endpoint in mgr-tenant-entitlements, or maybe not.

Pros

• No special handling required in Kong

• No security concerns

• No need for the sidecars to get ALL discovery information, only what’s required, when it’s required.

Cons

• Multiple ways for sidecars to get discovery information

  • Ask during startup, then get updates via kafka

  • Retrieve on an as-needed basis

• Possibly larger memory footprint required for the mod-scheduler sidecar

  • Could be mitigated by adding a trait to endpoints indicating whether or not they’re eligible for being scheduled

Risks & Assumptions

1. Risk 1

2. Risk 2 ...

3. Assumption 1

4. Assumption 2 ...

Conclusion

...

Future Considerations

The use of an all-powerful system user is a potential risk which we’d like to address at some point. The high level idea is to instead provision and use system users with distinct capabilities. In order for this to happen changes would need to be made to module descriptors since many of these timers are defined w/o requiredPermissions. One potential gotcha here is with backward compatibility. We need to ensure that specifying required permissions on these system timer interfaces will not break things when using OKAPI.

Conclusion

Let’s go with 1.2, and in parallel look into how we can possibly adopt option 3 as well. We’d like to consolidate the mechanisms used by the sidecars for obtaining discovery information. We don’t want 2 or 3 ways this is done.

Spike Status:

Attachments

Include any relevant attachments, such as documents, diagrams, or presentations that support the spike