|
Current situation or problem:
The existing implementation of users can export up to 22 M records with the default mapping profile but significantly less when exporting with a custom mapping profile that includes data coming from holdings and item records. The performance deteriorates further when triggered by a CQL query and recommended number of records is 300K. However, the limits not enforced programmatically and are causing additional work for librarians who must manually create a files with the specified number of records UUIDs to trigger the export.
In scope
- Performance improvements for exporting:
- instances and SRS records
- holdings
- authority records
- Export of 22M SRS records in 24 hours without significantly impacting the performance of the other FOLIO modules (cataloging, check in, check out, data import)
- Support multiple concurrent exports:
- daily (thousands) :
- monthly:
- annually:
- Establish performance baseline to determine areas that will require improvements
- Determine an easy way for triggering the export of all instances/SRS records without the need to first list UUIDs of matching records.
- Set to 100k the max number of records to be saved in .mrc file and split the entire export into supported size files
- Compress all files generated in one export job into one directory and provide the link to it similarly as it is done currently for single files
Additional information:
|