Sidecar fix performance testing

11-14 13:28
Test 1
QECP1 we deployed Sidecat 1.0.0.343
RTAC 10 VU-25inst, throughput 6.4-7, RT ~3 sec

image-20241122-102700.png

 

Test 2
RTAC 20 VU-25 , throughput 6.4-7, RT ~4.6 sec

image-20241122-102912.png


Data Import

1.1.0-Snapshot 349

2.1.0-Snapshot 349

Add 100mb of memory on sidecars

 

November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,482 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-3) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,474 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-4) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,473 ERROR [io.net.uti.con.Def.rejectedExecution] (vert.x-eventloop-thread-2) Failed to submit a listener notification task. Event loop shut down? [Error Occurred After Shutdown]: java.util.concurrent.RejectedExecutionException: event executor terminated
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:931)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:350)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:343)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:833)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:824)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:814)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:862)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:500)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:97)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.group.DefaultChannelGroupFuture.setSuccess0(DefaultChannelGroupFuture.java:200)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.group.DefaultChannelGroupFuture.access$400(DefaultChannelGroupFuture.java:41)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.group.DefaultChannelGroupFuture$1.operationComplete(DefaultChannelGroupFuture.java:75)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.group.DefaultChannelGroupFuture$1.operationComplete(DefaultChannelGroupFuture.java:48)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:97)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.DefaultChannelPromise.setSuccess(DefaultChannelPromise.java:78)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.DefaultChannelPromise.setSuccess(DefaultChannelPromise.java:73)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.vertx.core.net.impl.ConnectionBase.lambda$close$0(ConnectionBase.java:130)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:97)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.DefaultChannelPromise.setSuccess(DefaultChannelPromise.java:78)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.DefaultChannelPromise.setSuccess(DefaultChannelPromise.java:73)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.vertx.core.net.impl.ConnectionBase.handleClosed(ConnectionBase.java:381)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.vertx.core.http.impl.Http1xClientConnection.handleClosed(Http1xClientConnection.java:1180)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.vertx.core.net.impl.VertxHandler.channelInactive(VertxHandler.java:143)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:412)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:377)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:410)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1402)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:900)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:811)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at java.base/java.lang.Thread.run(Unknown Source)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,474 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-2) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,472 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-4) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,472 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-6) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,472 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-2) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,473 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-2) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,468 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-1) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,469 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-6) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,469 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-2) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,469 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-4) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,431 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-2) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,431 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-4) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,430 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-6) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,426 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-1) Sending error response: type = InternalServerErrorException, message = Failed to proxy request
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
2024-11-21 11:42:52,401 ERROR [io.net.uti.con.Def.rejectedExecution] (vert.x-eventloop-thread-4) Failed to submit a listener notification task. Event loop shut down? [Error Occurred After Shutdown]: java.util.concurrent.RejectedExecutionException: event executor terminated
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:931)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:350)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:343)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:833)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:824)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:814)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:862)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:500)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.vertx.core.net.impl.TCPServerBase.lambda$actualClose$8(TCPServerBase.java:366)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.vertx.core.impl.future.FutureImpl$4.onSuccess(FutureImpl.java:176)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:60)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
sidecar-mod-rtac
November 21, 2024 at 13:42 (UTC+2:00)
at java.base/java.lang.Thread.run(Unknown Source)
sidecar-mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
11:39:23 [351518/mod-rtac] [fs09000000] [422272c0-07c9-4539-a7a1-01ffd2feaf8a] [mod_rtac] INFO RestRouting invoking postRtacBatch
mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
11:39:23 [351518/mod-rtac] [fs09000000] [422272c0-07c9-4539-a7a1-01ffd2feaf8a] [mod_rtac] INFO InventoryClient Getting item and holding information from inventory
mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
2024-11-21 11:39:23,837 INFO [org.fol.sid.int.key.fil.KeycloakAuthorizationFilter] (vert.x-eventloop-thread-4) Authorizing request to: /rtac-batch#POST for tenant: fs09000000
sidecar-mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
2024-11-21 11:39:23,837 INFO [org.fol.sid.ser.rou.IngressRequestHandler] (vert.x-eventloop-thread-4) Forwarding ingress request to underlying module: [method: POST, path: /mod-rtac/rtac-batch]
sidecar-mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
2024-11-21 11:39:23,833 INFO [org.fol.sid.ser.rou.IngressRequestHandler] (vert.x-eventloop-thread-4) Handling ingress request [method: POST, path: /mod-rtac/rtac-batch]
sidecar-mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
172.17.0.6:51840 - sidecar:8081 - [21/11/2024:11:39:18 UTC] "GET /circulation/requests HTTP_1_1" 200 44 rt=0.031 uct="" uht="0.0" urt="0.0" "Vert.x-WebClient/4.5.4" "fs09000000" "" "832232/circulation"
sidecar-mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
172.17.0.6:51938 - sidecar:8081 - [21/11/2024:11:39:18 UTC] "GET /circulation/requests HTTP_1_1" 200 44 rt=1.944 uct="" uht="0.0" urt="0.0" "Vert.x-WebClient/4.5.4" "fs09000000" "" "759304/circulation"
sidecar-mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
2024-11-21 11:39:18,966 INFO [org.fol.sid.ser.rou.EgressRequestHandler] (vert.x-eventloop-thread-2) Handling egress request [method: GET, path: /circulation/requests]
sidecar-mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
2024-11-21 11:39:18,966 INFO [org.fol.sid.ser.rou.EgressRequestHandler] (vert.x-eventloop-thread-2) Authenticating and forwarding egress request [method: GET, path: /circulation/requests, requestId: null]
sidecar-mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
2024-11-21 11:39:18,966 INFO [org.fol.sid.ser.ServiceTokenProvider] (vert.x-eventloop-thread-2) Getting service token [method: GET, path: /circulation/requests, requestId: null]
sidecar-mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
2024-11-21 11:39:18,966 INFO [org.fol.sid.ser.ServiceTokenProvider] (vert.x-eventloop-thread-2) Token found in cache [requestId: null]
sidecar-mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
2024-11-21 11:39:18,966 INFO [org.fol.sid.ser.rou.EgressRequestHandler] (vert.x-eventloop-thread-2) Forwarding egress request to module: [method: GET, path: /circulation/requests, moduleId: mod-circulation-24.2.5, url: http://mod-circulation-b.qecp1.folio-eis.us-east-1:8051/mod-circulation]
sidecar-mod-rtac
November 21, 2024 at 13:39 (UTC+2:00)
2024-11-21 11:39:18,963 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-2) Uncaught exception received by Vert.x: java.lang.OutOfMemoryError: Java heap space
sidecar-mod-rtac

Data-import

@timestamp,@message,@logStream,@log
2024-11-21 10:44:17.890,"2024-11-21 10:44:17,889 ERROR [io.net.uti.con.Def.rejectedExecution] (vert.x-eventloop-thread-3) Failed to submit a listener notification task. Event loop shut down? [Error Occurred After Shutdown]: java.util.concurrent.RejectedExecutionException: event executor terminated",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:44:17.889,"2024-11-21 10:44:17,889 ERROR [io.net.uti.con.Def.rejectedExecution] (vert.x-eventloop-thread-2) Failed to submit a listener notification task. Event loop shut down? [Error Occurred After Shutdown]: java.util.concurrent.RejectedExecutionException: event executor terminated",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:44:12.566,"2024-11-21 10:44:12,566 WARN [io.net.uti.con.AbstractEventExecutor] (vert.x-eventloop-thread-2) A task raised an exception. Task: io.vertx.core.impl.future.FutureBase$$Lambda$1494/0x00000098017b34e0@3edc4f62: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:43:56.698,"2024-11-21 10:43:56,697 WARN [io.net.uti.con.AbstractEventExecutor] (vert.x-eventloop-thread-4) A task raised an exception. Task: io.vertx.core.impl.future.FutureBase$$Lambda$1494/0x00000098017b34e0@469449b: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:43:54.854,"Exception in thread ""Keep-Alive-Timer"" java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:41:15.345,"2024-11-21 10:41:15,344 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-0) Uncaught exception received by Vert.x: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:19.842,"2024-11-21 10:40:19,841 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-1) Uncaught exception received by Vert.x: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:19.842,"2024-11-21 10:40:19,841 WARN [io.net.uti.con.AbstractEventExecutor] (vert.x-eventloop-thread-6) A task raised an exception. Task: io.vertx.core.impl.future.FutureBase$$Lambda$1494/0x00000098017b34e0@329a18df: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:17.330,"Exception in thread ""vertx-blocked-thread-checker"" java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:17.028,"2024-11-21 10:40:17,026 WARN [io.net.uti.con.AbstractEventExecutor] (vert.x-eventloop-thread-0) A task raised an exception. Task: io.vertx.core.http.impl.Http1xClientConnection$$Lambda$1497/0x00000098017b3b78@4c1d0a8e: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:17.028,"2024-11-21 10:40:17,026 WARN [io.net.cha.AbstractChannelHandlerContext] (vert.x-eventloop-thread-6) An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:17.027,"2024-11-21 10:40:17,027 WARN [io.net.cha.AbstractChannelHandlerContext] (vert.x-eventloop-thread-3) An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:17.026,"2024-11-21 10:40:17,025 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-7) Uncaught exception received by Vert.x: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:17.026,"2024-11-21 10:40:17,026 WARN [io.net.cha.AbstractChannelHandlerContext] (vert.x-eventloop-thread-5) An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.104,10:40:10 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] INFO KafkaEventPublisher publish:: Event with type: 'DI_ERROR' by jobExecutionId: '256c3c04-12da-4b62-86b7-b08eb56956f9' and recordId: 'eeb54291-6b74-4110-92f7-6abca18e5223' with chunkId: '86721def-6e1b-4418-9131-4e52c0f8f754' was sent to the topic 'qecp1.Default.ALL.DI_ERROR' ,qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.104,10:40:10 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] WARN taImportKafkaHandler Failed to process data import event payload: DI_ERROR,qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.104,"10:40:10 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] ERROR KafkaConsumerWrapper businessHandlerCompletionHandler:: Error while processing a record - id: 57 subscriptionPattern: SubscriptionDefinition(eventType=DI_INCOMING_MARC_BIB_RECORD_PARSED, subscriptionPattern=qecp1\.Default\.\w{1,}\.DI_INCOMING_MARC_BIB_RECORD_PARSED) offset: 205963",qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.104,"10:40:10 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] WARN KafkaConsumerWrapper businessHandlerCompletionHandler:: Error handler has not been implemented for subscriptionPattern: SubscriptionDefinition(eventType=DI_INCOMING_MARC_BIB_RECORD_PARSED, subscriptionPattern=qecp1\.Default\.\w{1,}\.DI_INCOMING_MARC_BIB_RECORD_PARSED) failures",qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.091,"10:40:10 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] ERROR InstanceEventHandler Error posting Instance by instanceId:'8d41f871-57c1-463d-828d-4499eaf3db6c' cause {""errors"":[{""type"":""InternalServerErrorException"",""code"":""service_error"",""message"":""Failed to proxy request"",""parameters"":[{""key"":""cause"",""value"":""Java heap space""}]}],""total_records"":1}, status code 500",qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.091,10:40:10 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] ERROR InstanceEventHandler Error creating inventory Instance by jobExecutionId: '256c3c04-12da-4b62-86b7-b08eb56956f9' and recordId: 'eeb54291-6b74-4110-92f7-6abca18e5223' and chunkId: '86721def-6e1b-4418-9131-4e52c0f8f754' ,qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.091,"io.vertx.core.impl.NoStackTraceThrowable: {""errors"":[{""type"":""InternalServerErrorException"",""code"":""service_error"",""message"":""Failed to proxy request"",""parameters"":[{""key"":""cause"",""value"":""Java heap space""}]}],""total_records"":1}",qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.089,"2024-11-21 10:40:10,088 ERROR [io.ver.cor.net.imp.ConnectionBase] (vert.x-eventloop-thread-6) invalid version format: DATE:",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080,"2024-11-21 10:40:10,079 WARN [io.ver.cor.imp.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-eventloop-thread-6,5,main] has been blocked for 2229 ms, time limit is 2000 ms: io.vertx.core.VertxException: Thread blocked",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at org.folio.sidecar.service.ErrorHandler.sendErrorResponse(ErrorHandler.java:83),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at org.folio.sidecar.service.ErrorHandler.sendErrorResponse(ErrorHandler.java:62),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at org.folio.sidecar.service.ErrorHandler_ClientProxy.sendErrorResponse(Unknown Source),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.vertx.core.http.impl.HttpClientRequestBase.handleException(HttpClientRequestBase.java:177),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.vertx.core.http.impl.HttpClientRequestImpl.handleException(HttpClientRequestImpl.java:90),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleException(Http1xClientConnection.java:714),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.vertx.core.http.impl.Http1xClientConnection.handleException(Http1xClientConnection.java:1243),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.vertx.core.net.impl.VertxHandler.exceptionCaught(VertxHandler.java:136),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:145),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:231),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:10.080, at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:08.801,"2024-11-21 10:40:08,801 WARN [org.fol.sid.ser.ErrorHandler] (vert.x-eventloop-thread-6) Sending error response: type = InternalServerErrorException, message = Failed to proxy request",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:07.846,"2024-11-21 10:40:07,846 ERROR [io.ver.cor.net.imp.ConnectionBase] (vert.x-eventloop-thread-6) Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:40:07.845,"2024-11-21 10:40:07,844 WARN [io.net.uti.con.SingleThreadEventExecutor] (vert.x-acceptor-thread-0) Unexpected exception from an event executor: : java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:57.229,"2024-11-21 10:39:57,228 ERROR [org.apa.kaf.cli.con.int.ConsumerCoordinator] (kafka-coordinator-heartbeat-thread | sc-mod-inventory_b6991194-64b8-4a71-b6cc-5d9ba9766587) [Consumer clientId=kafka-consumer-discovery, groupId=sc-mod-inventory_b6991194-64b8-4a71-b6cc-5d9ba9766587] Heartbeat thread failed due to unexpected error: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:56.597,"2024-11-21 10:39:56,597 ERROR [org.apa.kaf.cli.con.int.ConsumerCoordinator] (kafka-coordinator-heartbeat-thread | sc-mod-inventory_b6991194-64b8-4a71-b6cc-5d9ba9766587) [Consumer clientId=kafka-consumer-logout, groupId=sc-mod-inventory_b6991194-64b8-4a71-b6cc-5d9ba9766587] Heartbeat thread failed due to unexpected error: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:49.725,"2024-11-21 10:39:49,725 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-3) Uncaught exception received by Vert.x: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:31.086,"10:39:31 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] ERROR KafkaConsumerWrapper businessHandlerCompletionHandler:: Error while processing a record - id: 57 subscriptionPattern: SubscriptionDefinition(eventType=DI_INCOMING_MARC_BIB_RECORD_PARSED, subscriptionPattern=qecp1\.Default\.\w{1,}\.DI_INCOMING_MARC_BIB_RECORD_PARSED) offset: 205887",qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:31.086,"10:39:31 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] WARN KafkaConsumerWrapper businessHandlerCompletionHandler:: Error handler has not been implemented for subscriptionPattern: SubscriptionDefinition(eventType=DI_INCOMING_MARC_BIB_RECORD_PARSED, subscriptionPattern=qecp1\.Default\.\w{1,}\.DI_INCOMING_MARC_BIB_RECORD_PARSED) failures",qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:31.085,10:39:31 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] INFO KafkaEventPublisher publish:: Event with type: 'DI_ERROR' by jobExecutionId: '256c3c04-12da-4b62-86b7-b08eb56956f9' and recordId: 'f8317eb6-6623-4ab1-a08f-5d65ede30340' with chunkId: '85e45412-0f3c-4d90-a698-c51737b06ded' was sent to the topic 'qecp1.Default.ALL.DI_ERROR' ,qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:31.085,10:39:31 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] WARN taImportKafkaHandler Failed to process data import event payload: DI_ERROR,qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:31.073,10:39:31 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] ERROR InstanceEventHandler Error creating inventory Instance by jobExecutionId: '256c3c04-12da-4b62-86b7-b08eb56956f9' and recordId: 'f8317eb6-6623-4ab1-a08f-5d65ede30340' and chunkId: '85e45412-0f3c-4d90-a698-c51737b06ded' ,qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:31.073,"io.vertx.core.impl.NoStackTraceThrowable: {""details"":""Error id 246edb0a-c469-4e0e-b003-78ca9096fa9c-1"",""stack"":""""}",qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:31.070,"10:39:31 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] ERROR InstanceEventHandler Error posting Instance by instanceId:'2ce0588a-b7eb-4e5f-9009-5bd4bd72b8bb' cause {""details"":""Error id 246edb0a-c469-4e0e-b003-78ca9096fa9c-1"",""stack"":""""}, status code 500",qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:31.069,"2024-11-21 10:39:31,069 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (vert.x-eventloop-thread-6) HTTP Request to /instance-storage/instances failed, error id: 246edb0a-c469-4e0e-b003-78ca9096fa9c-1: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:30.729,"2024-11-21 10:39:30,413 ERROR [org.apa.kaf.cli.con.int.ConsumerCoordinator] (kafka-coordinator-heartbeat-thread | sc-mod-inventory_b6991194-64b8-4a71-b6cc-5d9ba9766587) [Consumer clientId=kafka-consumer-entitlement, groupId=sc-mod-inventory_b6991194-64b8-4a71-b6cc-5d9ba9766587] Heartbeat thread failed due to unexpected error: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:30.726,"2024-11-21 10:39:30,413 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-4) Uncaught exception received by Vert.x: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:23.100,"2024-11-21 10:39:23,099 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-6) Uncaught exception received by Vert.x: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:39:22.500,"2024-11-21 10:39:22,499 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-5) Uncaught exception received by Vert.x: java.lang.OutOfMemoryError: Java heap space",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:38:09.409,"2024-11-21 10:38:09,408 WARN [org.fol.sid.ser.rou.RequestMatchingService] (vert.x-eventloop-thread-4) Egress routing entry was not found for the request's path. Forwarding request to the Gateway: moduleId = sc-mod-inventory, path = /user-tenants, destination = https://kong-qecp1.int.aws.folio.org, x-okapi-module-id = null",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:35:36.071,"10:35:36 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] INFO cceedingTitlesHelper createPrecedingSucceedingTitles:: parameters instance: Instance ID: 24e7c916-3aa7-4dfb-ae65-3ea153eecbb0, HRID: null, Title: With reference to writs of error. , context: org.folio.inventory.dataimport.handlers.matching.util.EventHandlingUtil$1@5dc6238f ",qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:33:09.036,"2024-11-21 10:33:09,035 WARN [org.fol.sid.ser.rou.RequestMatchingService] (vert.x-eventloop-thread-1) Egress routing entry was not found for the request's path. Forwarding request to the Gateway: moduleId = sc-mod-inventory, path = /user-tenants, destination = https://kong-qecp1.int.aws.folio.org, x-okapi-module-id = null",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:32:08.019,"10:32:08 [] [fs09000000] [43687cb7-5307-47e9-a668-a7737657010e] [] INFO cceedingTitlesHelper createPrecedingSucceedingTitles:: parameters instance: Instance ID: 46265f31-f237-4c36-9948-b0cad0a57c6f, HRID: null, Title: With reference to writs of error. , context: org.folio.inventory.dataimport.handlers.matching.util.EventHandlingUtil$1@4de8c552 ",qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 10:28:08.814,"2024-11-21 10:28:08,813 WARN [org.fol.sid.ser.rou.RequestMatchingService] (vert.x-eventloop-thread-0) Egress routing entry was not found for the request's path. Forwarding request to the Gateway: moduleId = sc-mod-inventory, path = /user-tenants, destination = https://kong-qecp1.int.aws.folio.org, x-okapi-module-id = null",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:31:34.003,"2024-11-21 08:31:34,000 ERROR [io.sma.hea.checks] (executor-thread-3) SRHCK02002: Reporting URL health check error : java.net.SocketTimeoutException: Read timed out",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:31:34.003,"2024-11-21 08:31:34,001 INFO [io.sma.health] (vert.x-eventloop-thread-7) SRHCK01001: Reporting health down status: {""status"":""DOWN"",""checks"":[{""name"":""SmallRye Reactive Messaging - liveness check"",""status"":""UP"",""data"":{""logout"":""[OK]"",""discovery"":""[OK]"",""entitlement"":""[OK]""}},{""name"":""Kafka connection health check"",""status"":""UP"",""data"":{""nodes"":""b-1.fsetenant.c0t9fd.c3.kafka.us-east-1.amazonaws.com:9092,b-2.fsetenant.c0t9fd.c3.kafka.us-east-1.amazonaws.com:9092,b-3.fsetenant.c0t9fd.c3.kafka.us-east-1.amazonaws.com:9092,b-4.fsetenant.c0t9fd.c3.kafka.us-east-1.amazonaws.com:9092""}},{""name"":""SmallRye Reactive Messaging - readiness check"",""status"":""UP"",""data"":{""logout"":""[OK]"",""discovery"":""[OK]"",""entitlement"":""[OK]""}},{""name"":""Module health check"",""status"":""DOWN"",""data"":{""host"":""GET http://mod-inventory:8081/mod-inventory/admin/health"",""error"":""java.net.SocketTimeoutException: Read timed out""}},{""name"":""SmallRye Reactive Messaging - startup check"",""status"":""UP"",""data"":{""logout"":""[OK]"",""discovery"":""[OK]"",""entitlement"":""[OK]""}}]}",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:31:32.589,"2024-11-21 08:31:32,589 INFO [io.sma.health] (vert.x-eventloop-thread-3) SRHCK01001: Reporting health down status: {""status"":""DOWN"",""checks"":[{""name"":""SmallRye Reactive Messaging - liveness check"",""status"":""UP"",""data"":{""logout"":""[OK]"",""discovery"":""[OK]"",""entitlement"":""[OK]""}},{""name"":""Kafka connection health check"",""status"":""UP"",""data"":{""nodes"":""b-1.fsetenant.c0t9fd.c3.kafka.us-east-1.amazonaws.com:9092,b-2.fsetenant.c0t9fd.c3.kafka.us-east-1.amazonaws.com:9092,b-3.fsetenant.c0t9fd.c3.kafka.us-east-1.amazonaws.com:9092,b-4.fsetenant.c0t9fd.c3.kafka.us-east-1.amazonaws.com:9092""}},{""name"":""SmallRye Reactive Messaging - readiness check"",""status"":""UP"",""data"":{""logout"":""[OK]"",""discovery"":""[OK]"",""entitlement"":""[OK]""}},{""name"":""Module health check"",""status"":""DOWN"",""data"":{""host"":""GET http://mod-inventory:8081/mod-inventory/admin/health"",""error"":""java.net.SocketTimeoutException: Read timed out""}},{""name"":""SmallRye Reactive Messaging - startup check"",""status"":""UP"",""data"":{""logout"":""[OK]"",""discovery"":""[OK]"",""entitlement"":""[OK]""}}]}",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:31:32.567,"2024-11-21 08:31:32,565 ERROR [io.sma.hea.checks] (executor-thread-1) SRHCK02002: Reporting URL health check error : java.net.SocketTimeoutException: Read timed out",qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:52.494,JAVA_OPTS=-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-inventory.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -Dport=8082 -Dorg.folio.metadata.inventory.storage.type=okapi -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=512m -Xmx1814m,qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:51.174," ""file_path"": ""/var/log/nginx/error.log"",",qecp1/mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:48.087,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:46.930,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:45.820,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:44.685,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:43.448,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:42.324,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:41.234,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:40.075,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:38.957,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:37.779,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:36.637,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:35.514,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:34.253,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:33.132,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:32.003,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:30.835,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:29.720,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis
2024-11-21 08:30:28.665,jq: error (at <stdin>:0): Cannot iterate over null (null),qecp1/sidecar-mod-inventory/5f7c819edf1d49398391b5f0d37de151,054267740449:qecp1-folio-eis

X2 memory on sidecar-rtac

mod - rtac didn`t failed during last test (30 VU, 50 instances)

X2 memory on sidecar-rtac
100K failed