Infrastructure
PTF -environment
- 10 m6i.2xlarge EC2 instances
- 2 instances of db.r6.xlarge database instances, one reader and one writer
- MSK
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=120
- Kafka topics
- .data-export.job.command - 50 partitions
- data-export.job.update - 50 partitions
# | Modules | Version | Task Definition Revision | Running Tasks | CPU | Memory (Hard Limit) | MemoryReservation (Soft Limit) | maxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|---|
| 1 | mod-inventory-storage | 24.1.0 | 10 | 2 | 1024 | 2208 | 1952 | 512m | 1440m |
| 2 | okapi | 4.14.2 | 3 | 3 | 1024 | 1684 | 1440 | 512m | 922m |
| 3 | mod-invoice-storage | 3 | 2 | 1024 | 1872 | 1536 | 512m | 1024m | |
| 4 | mod-orders | 12.4.3 | 5 | 2 | 1024 | 2048 | 1440 | 512m | 896m |
| 5 | mod-agreements | 5.2.2 | 5 | 2 | 128 | 1592 | 1488 | 512m | 968m |
| 6 | mod-data-export-worker | 1.4.10 | 22 | 2 | 1024 | 3072 | 2800 | 612m | 2128m |
| 7 | mod-data-export-spring | 1.4.5 | 6 | 1 | 256 | 2048 | 1844 | 512m | 1292m |
| 8 | mod-notes | 3.1.2 | 3 | 2 | 128 | 1024 | 896 | 128m | 768m |
| 9 | mod-orders-storage | 13.3.3 | 4 | 2 | 128 | 1024 | 896 | 128m | 768m |
| 10 | mod-inventory | 18.2.2 | 4 | 2 | 1024 | 2880 | 2592 | 512m | 1814m |
| 11 | mod-users | 18.3.0 | 7 | 2 | 258 | 1024 | 896 | 128m | 768m |
Summary
- The safe limit is 8-10 jobs with 1000 orders per organization
- A few jobs failed with FTP error: Connection refused. However it doesn’t looks like it's the matter of FTP server. We did observe same behavior previously with 3-4 parallel job, and it was fixed by giving mod-data-export-worker more memory. At the same moment we can't see significant resource usage on FTP server side either.
- The percentage of success is highly depended on the number of orders. The more orders there are the higher chance of completing all jobs successfully.
- The upper safe limit for EDIFACT is 10 jobs with 1000 - 4000 orders per job.
- Most of the tests were run one after another and no memory leak was found.
Test Results
| Test # | Concurrent Jobs | Orders Per Job | Success rate | Time | |
|---|---|---|---|---|---|
| 1 | 10 | 100 | 6/10 | ±10 sec | Failed login to FTP (FtpException) Socket closed (SocketException) |
| 2 | 10 | 500 | 8/10 | 55-57 sec | |
| 3 | 10 | 1000 | 7/10 | 3 min | Failed login to FTP (FtpException) Socket closed (SocketException) |
| 4 | 10 | 1000 | 9/10 | 2 min 17 s | Connection refused (Connection refused) (ConnectException) |
| 5 | 10 | 1000 | 8/10 | 2 min 48 s | Connection refused (Connection refused) (ConnectException) File upload failed. 450 PASV: data transfer in progress (FtpException) |
| 6 | 10 | 2000 | 10/10 | ±5 min | |
| 7 | 10 | 4000 | 10/10 | 9-10 min | |
| 8 | 10 | 6000 | 8/10 | 16 min | Connection closed without indication. (FTPConnectionClosedException) Connection is not open (IOException) |
Resources usage
test 1
test 2
test 3
test 4
test 5
test 6
test 7
test 8
ERRORS
12:08:00 [] [] [] [] ERROR AbstractStep Encountered an error executing step saveToFTPStep in job EDIFACT_ORDERS_EXPORTjava.lang.reflect.UndeclaredThrowableException: nullat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:780) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.folio.dew.batch.acquisitions.edifact.jobs.SaveToFileStorageTasklet$$EnhancerBySpringCGLIB$$b0a3b5d8.execute(<generated>) ~[classes!/:1.4.10]at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.3.20.jar!/:5.3.20]at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.3.6.jar!/:4.3.6]at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.3.6.jar!/:4.3.6]at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:152) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:413) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:136) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:320) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:149) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) ~[spring-core-5.3.20.jar!/:5.3.20]at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:140) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at jdk.internal.reflect.GeneratedMethodAccessor603.invoke(Unknown Source) ~[?:?]at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:128) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.20.jar!/:5.3.20]at com.sun.proxy.$Proxy210.run(Unknown Source) ~[?:?]at org.springframework.batch.integration.launch.JobLaunchingMessageHandler.launch(JobLaunchingMessageHandler.java:50) ~[spring-batch-integration-4.3.6.jar!/:4.3.6]at org.folio.dew.batch.ExportJobManager.launchJob(ExportJobManager.java:29) ~[classes!/:1.4.10]at org.folio.dew.service.JobCommandsReceiverService.receiveStartJobCommand(JobCommandsReceiverService.java:127) ~[classes!/:1.4.10]at jdk.internal.reflect.GeneratedMethodAccessor594.invoke(Unknown Source) ~[?:?]at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:169) ~[spring-messaging-5.3.20.jar!/:5.3.20]at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119) ~[spring-messaging-5.3.20.jar!/:5.3.20]at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:56) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:347) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:53) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2645) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2625) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2552) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2433) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2311) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1982) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1366) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1357) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1252) ~[spring-kafka-2.8.6.jar!/:2.8.6]at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]at java.lang.Thread.run(Thread.java:829) ~[?:?]Caused by: java.net.ConnectException: Connection refused (Connection refused)at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?]at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?]at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?]at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]at java.net.Socket.connect(Socket.java:609) ~[?:?]at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:920) ~[commons-net-3.6.jar!/:3.6]at org.apache.commons.net.ftp.FTPClient._storeFile(FTPClient.java:653) ~[commons-net-3.6.jar!/:3.6]at org.apache.commons.net.ftp.FTPClient.__storeFile(FTPClient.java:639) ~[commons-net-3.6.jar!/:3.6]at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:2030) ~[commons-net-3.6.jar!/:3.6]at org.folio.dew.repository.FTPObjectStorageRepository.upload(FTPObjectStorageRepository.java:83) ~[classes!/:1.4.10]at org.folio.dew.repository.FTPObjectStorageRepository$$FastClassBySpringCGLIB$$866336c1.invoke(<generated>) ~[classes!/:1.4.10]at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.folio.dew.repository.FTPObjectStorageRepository$$EnhancerBySpringCGLIB$$e6581fb6.upload(<generated>) ~[classes!/:1.4.10]at org.folio.dew.batch.acquisitions.edifact.jobs.SaveToFileStorageTasklet.execute(SaveToFileStorageTasklet.java:66) ~[classes!/:1.4.10]at org.folio.dew.batch.acquisitions.edifact.jobs.SaveToFileStorageTasklet$$FastClassBySpringCGLIB$$9088e182.invoke(<generated>) ~[classes!/:1.4.10]at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.20.jar!/:5.3.20]... 57 more
12:08:00 [] [] [] [] ERROR AbstractStep Encountered an error executing step saveToFTPStep in job EDIFACT_ORDERS_EXPORTjava.lang.reflect.UndeclaredThrowableException: nullat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:780) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.folio.dew.batch.acquisitions.edifact.jobs.SaveToFileStorageTasklet$$EnhancerBySpringCGLIB$$b0a3b5d8.execute(<generated>) ~[classes!/:1.4.10]at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.3.20.jar!/:5.3.20]at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.3.6.jar!/:4.3.6]at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.3.6.jar!/:4.3.6]at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:152) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:413) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:136) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:320) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:149) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) ~[spring-core-5.3.20.jar!/:5.3.20]at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:140) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at jdk.internal.reflect.GeneratedMethodAccessor603.invoke(Unknown Source) ~[?:?]at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:128) ~[spring-batch-core-4.3.6.jar!/:4.3.6]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.20.jar!/:5.3.20]at com.sun.proxy.$Proxy210.run(Unknown Source) ~[?:?]at org.springframework.batch.integration.launch.JobLaunchingMessageHandler.launch(JobLaunchingMessageHandler.java:50) ~[spring-batch-integration-4.3.6.jar!/:4.3.6]at org.folio.dew.batch.ExportJobManager.launchJob(ExportJobManager.java:29) ~[classes!/:1.4.10]at org.folio.dew.service.JobCommandsReceiverService.receiveStartJobCommand(JobCommandsReceiverService.java:127) ~[classes!/:1.4.10]at jdk.internal.reflect.GeneratedMethodAccessor594.invoke(Unknown Source) ~[?:?]at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:169) ~[spring-messaging-5.3.20.jar!/:5.3.20]at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119) ~[spring-messaging-5.3.20.jar!/:5.3.20]at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:56) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:347) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:92) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:53) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2645) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2625) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2552) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2433) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2311) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1982) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1366) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1357) ~[spring-kafka-2.8.6.jar!/:2.8.6]at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1252) ~[spring-kafka-2.8.6.jar!/:2.8.6]at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]at java.lang.Thread.run(Thread.java:829) ~[?:?]Caused by: java.net.ConnectException: Connection refused (Connection refused)at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?]at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?]at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?]at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]at java.net.Socket.connect(Socket.java:609) ~[?:?]at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:920) ~[commons-net-3.6.jar!/:3.6]at org.apache.commons.net.ftp.FTPClient._storeFile(FTPClient.java:653) ~[commons-net-3.6.jar!/:3.6]at org.apache.commons.net.ftp.FTPClient.__storeFile(FTPClient.java:639) ~[commons-net-3.6.jar!/:3.6]at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:2030) ~[commons-net-3.6.jar!/:3.6]at org.folio.dew.repository.FTPObjectStorageRepository.upload(FTPObjectStorageRepository.java:83) ~[classes!/:1.4.10]at org.folio.dew.repository.FTPObjectStorageRepository$$FastClassBySpringCGLIB$$866336c1.invoke(<generated>) ~[classes!/:1.4.10]at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.folio.dew.repository.FTPObjectStorageRepository$$EnhancerBySpringCGLIB$$e6581fb6.upload(<generated>) ~[classes!/:1.4.10]at org.folio.dew.batch.acquisitions.edifact.jobs.SaveToFileStorageTasklet.execute(SaveToFileStorageTasklet.java:66) ~[classes!/:1.4.10]at org.folio.dew.batch.acquisitions.edifact.jobs.SaveToFileStorageTasklet$$FastClassBySpringCGLIB$$9088e182.invoke(<generated>) ~[classes!/:1.4.10]at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.20.jar!/:5.3.20]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.20.jar!/:5.3.20]... 57 more