PoC: two variants of Export and Input managers implementation and compare them

Problem statement 

Spring migration process for mod-data-export module assumes removing Vert.x completely. In this case, most of the existing logic that utilizes asynchronous operations needs to be refactored as well. Moreover, current implementation of Input and Export managers uses Vert.x event bus messaging system to communicate to each other, so that alternative must be found. 

Spring allows us to use SpringEvents as a messaging system in two modes: synchronous with one single thread, and asynchronous with multiple threads running in parallel. In this PoC it needs to prove that keeping asynchronous logic is the better approach in terms of efficiency and scalability. 

Approach description 

For this PoC Input and Export managers were simulated using two approaches: single thread and multiple threads. Both utilize Spring Events.  

To simulate communications between Input and Export managers, a specific CSV file with UUIDs was used. InputDataManager reads this file by chunks (chunk size = 50) and invokes ExportManager to export data. In turn, ExportManager creates and initializes Event with the obtained data and publishes Event using ApplicationEventPublisher. Next, InputDataManager listens to events and accepts them writing the event data to the temporary file. 

Experiments 

The number of records to export was up to 8 million. Multithreading approach assumes a specific pool size that can be set to TaskExecutor in SpringEvents. In the experiment sizes 2 through 14 were used. Tests were run for single thread and multiple threads with different pool size. The final result was obtained as the average value after running test 10 times.

Results

According to the results above, the most efficient way (in terms of elapsed time for export) is to use multithreading approach with pool size > 2.

Appendix

Project that was used for testing: project