[MGRENTITLE-1] CO - mgr-tenant-entitlements - Failed to create routes Created: 31/Dec/23  Updated: 05/Feb/24

Status: In Review
Project: mgr-tenant-entitlements
Components: None
Affects versions: None
Fix versions: None

Type: Bug Priority: P3
Reporter: Craig McNally Assignee: Yauhen Viazau
Resolution: Unresolved Votes: 0
Labels: back-end, epam-eureka, eureka-phase4, need-review
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File MGRENTITLE-1_app_minimal.mp4    
Issue links:
Defines
defines UXPROD-4605 Component Ownership In Progress
Sprint: Eureka Sprint 45
Story Points: 0
Development Team: Eureka

 Description   

mgr-tenant-entitlements periodically fails to create a routes

{{"errors"": [ { ""message"": ""Application flow '88f6c07d-6b19-4983-ba78-789417891e1d' executed with status: FAILED"", ""type"": ""FlowExecutionException"", ""code"": ""service_error"", ""parameters"": [ ...SKIPPED.... { ""key"": ""KongGatewayRouteRegistrar"", ""value"": ""FAILED: [IntegrationException] Failed to create routes, parameters: [{key: RoutingEntry(methods=[GET], pathPattern=/identifier-types/

{id}

, path=null), value: [502 Bad Gateway] during [PUT] to http://kong-admin.evrk.folio-eis.us-west-2:8001/services/f1b2323c-9565-41d7-996d-5048782bfec7/routes/e2a92cf21171873411c8ad67cab1f1505c8766a3 KongAdminClient#upsertRoute(String,String,KongRoute): [<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n</body>\r\n</html>\r\n]}]"" }, { ""key"": ""KongGatewayRouteRegistrar"", ""value"": ""FAILED: [IntegrationException] Failed to remove routes, parameters: [}}

{{

{key: 8df188b1-ae7b-4d50-a9c1-fe2293d67143, value: [502 Bad Gateway] during [DELETE] to http://kong-admin.evrk.folio-eis.us-west-2:8001/services/13f65407-bf15-43e9-a250-89be33f2900c/routes/8df188b1-ae7b-4d50-a9c1-fe2293d67143 [KongAdminClient#deleteRoute(String,String)]: [<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n</body>\r\n</html>\r\n]}

}}

]"" },1694509224627,com.ebsco.entitlement.integration.IntegrationException: Failed to create routes1694509224627, at com.ebsco.entitlement.integration.kong.KongGatewayService.addRoutes(KongGatewayService.java:55) ~[classes!/:1.0.0-SNAPSHOT]1694509224627, at com.ebsco.entitlement.integration.kong.KongGatewayRouteRegistrar.execute(KongGatewayRouteRegistrar.java:30) ~[classes!/:1.0.0-SNAPSHOT]1694509224627, at jdk.internal.reflect.GeneratedMethodAccessor328.invoke(Unknown Source) ~[?:?]1694509224627, at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]1694509224627, at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]1694509224627, at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-6.0.6.jar!/:6.0.6]1694509224627, at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698) ~[spring-aop-6.0.6.jar!/:6.0.6]1694509224627, at com.ebsco.entitlement.integration.kong.KongGatewayRouteRegistrar$$SpringCGLIB$$0.execute(<generated>) ~[classes!/:1.0.0-SNAPSHOT]1694509224627, at com.ebsco.flow.api.ExecutableStage.executeStage(ExecutableStage.java:66) ~[folio-flow-engine-1.0.2-SNAPSHOT.jar!/:1.0.2-SNAPSHOT]1694509224627, at com.ebsco.flow.api.ExecutableStage.execute(ExecutableStage.java:32) ~[folio-flow-engine-1.0.2-SNAPSHOT.jar!/:1.0.2-SNAPSHOT]1694509224627, at com.ebsco.flow.api.ParallelStage.lambda$composeExecutionFuture$7(ParallelStage.java:93) ~[folio-flow-engine-1.0.2-SNAPSHOT.jar!/:1.0.2-SNAPSHOT]1694509224627, at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.CompletableFuture$AsyncSupply.exec(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.ForkJoinPool$WorkQueue.helpAsyncBlocker(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.ForkJoinPool.helpAsyncBlocker(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.CompletableFuture.waitingGet(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.CompletableFuture.join(Unknown Source) ~[?:?]1694509224627, at com.ebsco.flow.api.ParallelStage.execute(ParallelStage.java:57) ~[folio-flow-engine-1.0.2-SNAPSHOT.jar!/:1.0.2-SNAPSHOT]1694509224627, at com.ebsco.flow.api.ExecutableStage.executeStage(ExecutableStage.java:66) ~[folio-flow-engine-1.0.2-SNAPSHOT.jar!/:1.0.2-SNAPSHOT]1694509224627, at com.ebsco.flow.api.ExecutableStage.execute(ExecutableStage.java:32) ~[folio-flow-engine-1.0.2-SNAPSHOT.jar!/:1.0.2-SNAPSHOT]1694509224627, at com.ebsco.flow.api.FlowImpl.lambda$executeStageAsync$6(FlowImpl.java:270) ~[folio-flow-engine-1.0.2-SNAPSHOT.jar!/:1.0.2-SNAPSHOT]1694509224627, at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.CompletableFuture$AsyncSupply.exec(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.ForkJoinPool$WorkQueue.helpAsyncBlocker(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.ForkJoinPool.helpAsyncBlocker(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.CompletableFuture.waitingGet(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.CompletableFuture.join(Unknown Source) ~[?:?]1694509224627, at com.ebsco.flow.api.FlowImpl.execute(FlowImpl.java:151) ~[folio-flow-engine-1.0.2-SNAPSHOT.jar!/:1.0.2-SNAPSHOT]1694509224627, at com.ebsco.flow.api.ExecutableStage.executeStage(ExecutableStage.java:66) ~[folio-flow-engine-1.0.2-SNAPSHOT.jar!/:1.0.2-SNAPSHOT]1694509224627, at com.ebsco.flow.api.ExecutableStage.execute(ExecutableStage.java:32) ~[folio-flow-engine-1.0.2-SNAPSHOT.jar!/:1.0.2-SNAPSHOT]1694509224627, at com.ebsco.flow.api.FlowImpl.lambda$executeStageAsync$6(FlowImpl.java:270) ~[folio-flow-engine-1.0.2-SNAPSHOT.jar!/:1.0.2-SNAPSHOT]1694509224627, at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.CompletableFuture$AsyncSupply.exec(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.ForkJoinPool.scan(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) ~[?:?]1694509224627, at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) ~[?:?]

Tenant entitlement result must be stable and return 200 OK after installation 

Scope:

Investigate and find the root cause and document findings on the wiki.

Either fix the issue if it's simple, or create a follow-on story to be triaged/prioritized

NOTE:  Upgrading to the latest version of Kong may resolve this.  It might be worth doing that task first to see if this is still an issue afterwards. 

TODO:  @Craig McNally for find and link the Kong upgrade story as a dependency.



 Comments   
Comment by Yauhen Viazau [ 12/Jan/24 ]

Checked on Demo env

When running POST /entitlements for "app-platform-complete-0.0.2", sometimes entitlement gets cancelled due to timeout error, for example:
 

{                     
   "key": "mod-inventory-20.1.3-moduleInstaller",                     
   "value": "FAILED: [IntegrationException] [HttpTimeoutException] Failed to perform request [method: POST, uri: http://mod-inventory-b.oed.folio-eis.us-west-2:8051/mod-inventory/_/tenant], cause: request timed out"  
},

 

Comment by Yauhen Viazau [ 05/Feb/24 ]

Tested on “evrk“ env - works as expected

Verified by entitling and removing “app-platform-minimal“ to 5 different tenants. See example:

As per discussion with developers, this issue stopped being reproducible after switching to newer Keycloak version

Generated at Thu Feb 08 22:18:05 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.