|
Current situation or problem:
The number of records that can be successfully exported depends on the type of mappings and how the export has been triggered. Exporting a couple of millions with the default mapping profile normally completes successfully but when the same list of records is exported with additional data from holdings or items records defined in a custom profile, the performance plummets. Also, for exports that are triggered by a CQL query the performance further deteriorates. In the Kiwi release the recommended max number of records is 300K but this limit is not enforced. During the Data Export Working Group meeting on Dec 9, 2021 it was proposed to let the system automatically splits the requested records into batches that are exported in the supported number of records.
In scope
- Add a tenant level configuration value that would set the limit for data export, the default value will be 100K.
- During the export, the system will split the files that are generated so that each file contains maximum of 100K records.
- User can trigger export by submitting list of identifiers or cql and system will automatically split larger exports.
- All files generated during the export will be stored in one directory and will be compressed.
- UI will display one link to the directory containing the files.
- If errors occur, they will be listed in one error log associated with the given export job.
Questions
- Should the number of records per file be configurable? If yes should it be configurable on the tenant level (it will affect all exports) or on the job profile level (it will affect all exports of the same type) - Initial implementation will support configuration on the tenant level.
|