...
Remote partitioning using Spring Batch Integration : https://docs.spring.io/spring-batch/reference/spring-batch-integration/sub-elements.html#remote-partitioning with Kafka .
With such approach we will have batch job “manager“, which will construct chunks when job is submitted, then send chunk metadata to kafka so consumers (batch job “worker“) can read, process chunks, write/upload the file and return processing result metadata to kafka to later be consumed by “manager“ to complete the job.
...