The goal of this tool is to make mapping updates that require updating all instance or authority records easier and more performant. We hope to continue improvements so that this tool is used by hosting providers.
Reported challenges
Migration takes a long time. We have tested in on a tenant with a large amount of data (8 million instances). It takes around 3.5-4 hours for remapping and around 8 hours for data saving
Migration cannot be run in the background.
Errors during the migration process (remapping). If the dataset has some issues (at least 1 incorrect instance or authority), you are not able to finish the migration process, and you need to fix data first and start from scratch.
Errors during the migration process (data_saving).
During data saving, I encountered several different errors, which I’ve collected in this ticket. As far as I know, there is one more ticket from the PTF team related to marc-migration as well (url).
General feeling—I have used chunk size equal to 500; that was recommended by the PTF team (recommendations) and it’s really hard to analyze and work with errors. During the data saving process, I had 100 chunks with 1 or 2 failed records; that means I need to download 100 files from S3 bucket, analyze them separately, and start the data saving process one more time after fixing all of the issues. Sometimes it’s not clear what should be fixed.
Troubleshooting errors that cannot be resolved by restarting the process.
Authority updates are more successful than Bib updates
ASA is unable to use this tool to update authority records.
Ways to address these issues
Investigate ways to improve MARC bibs updates especially in the area of performance (its a performance beast) and decrease downtime.
Is there a way for this tool to be run in the background so that other actions/workflows can continue.
Resolve issues/defects that prevent successful MARC bib updates
Update documentation for hosting providers
Retry mechanism > If a chunk fail then restart and continue.
If it continues to fail, then skip and go onto the next chunk.
Report the failed chunk as an error.
Error handling
Provide a way to query for errors
Generate an export that at minimum include the UUID, HRID, Title/Heading?, error message that can be used to troubleshoot by support or provided to the library.
Investigate how to mitigate the risk of data corruption
Work with FSE hosting on 2-3 institutions to help address issues.
The goal of this tool is to make mapping updates that require updating all instance or authority records easier and more performant. We hope to continue improvements so that this tool is used by hosting providers.
Reported challenges
Migration takes a long time. We have tested in on a tenant with a large amount of data (8 million instances). It takes around 3.5-4 hours for remapping and around 8 hours for data saving
Migration cannot be run in the background.
Errors during the migration process (remapping). If the dataset has some issues (at least 1 incorrect instance or authority), you are not able to finish the migration process, and you need to fix data first and start from scratch.
I’ve used instructions that @Abdenour Drif prepared (https://eis.atlassian.net/wiki/spaces/GSE/pages/1367670833/MARC+Migration+Protocol) and still remapping failed for some instances. We need to have proper scripts prepared to make sure that we can validate data until we start the migration process.
Errors during the migration process (data_saving).
During data saving, I encountered several different errors, which I’ve collected in this ticket. As far as I know, there is one more ticket from the PTF team related to marc-migration as well (url).
General feeling—I have used chunk size equal to 500; that was recommended by the PTF team (recommendations) and it’s really hard to analyze and work with errors. During the data saving process, I had 100 chunks with 1 or 2 failed records; that means I need to download 100 files from S3 bucket, analyze them separately, and start the data saving process one more time after fixing all of the issues. Sometimes it’s not clear what should be fixed.
Troubleshooting errors that cannot be resolved by restarting the process.
Authority updates are more successful than Bib updates
ASA is unable to use this tool to update authority records.
Ways to address these issues
Investigate ways to improve MARC bibs updates especially in the area of performance (its a performance beast) and decrease downtime.
Is there a way for this tool to be run in the background so that other actions/workflows can continue.
Resolve issues/defects that prevent successful MARC bib updates
Update documentation for hosting providers
Retry mechanism > If a chunk fail then restart and continue.
If it continues to fail, then skip and go onto the next chunk.
Report the failed chunk as an error.
Error handling
Provide a way to query for errors
Generate an export that at minimum include the UUID, HRID, Title/Heading?, error message that can be used to troubleshoot by support or provided to the library.
Investigate how to mitigate the risk of data corruption
Work with FSE hosting on 2-3 institutions to help address issues.
Performance testing
ECS
Non-ECS
single cluster
multi-cluster