MODBULKOPS-13 - Pagination mechanism for csv-file reading
MODBULKOPS-13 - Pagination mechanism for csv-file reading
To investigate pagination approach 100M rows file was created with identifiers like data.
The reading was done by 1M rows at same time. The time of reading 1M rows increases while offset increases:
Also it was created 100K rows file. The reading was done by 100 rows at same time. The time increases as well while offset increases:
Assuming that the reading time of the rows batch is the same reading time increases as it is necessary to find right place (correct offset) to start reading, the bigger offset is the more time is needed.
, multiple selections available,
Related content
SPIKE - improve parsing of .csv files with identifiers
SPIKE - improve parsing of .csv files with identifiers
More like this
Overview: Reliably process large MARC files (Data-Slicing)
Overview: Reliably process large MARC files (Data-Slicing)
More like this
PTF - Performance testing of moving Parsed Records Flattening From Database To Module Space (Quesnelia) [non-ECS]
PTF - Performance testing of moving Parsed Records Flattening From Database To Module Space (Quesnelia) [non-ECS]
More like this
Data Import test report Ramsons [non-ECS]
Data Import test report Ramsons [non-ECS]
More like this
Data Import Create MARC Holdings Records [Poppy]
Data Import Create MARC Holdings Records [Poppy]
More like this
Data Import Creates + Updates multi tenant with file split enabled
Data Import Creates + Updates multi tenant with file split enabled
More like this