MODBULKOPS-13 - Pagination mechanism for csv-file reading
To investigate pagination approach 100M rows file was created with identifiers like data.
The reading was done by 1M rows at same time. The time of reading 1M rows increases while offset increases:
Also it was created 100K rows file. The reading was done by 100 rows at same time. The time increases as well while offset increases:
Assuming that the reading time of the rows batch is the same reading time increases as it is necessary to find right place (correct offset) to start reading, the bigger offset is the more time is needed.