2020-01-06 - Data Migration Subgroup Agenda and Notes

• Find a note taker. (Jason Root volunteered - he is solely responsible for the shabby state of these notes. To be fair, it IS the Monday after a much needed break...)

• We will take the hour to see how people are proceeding with their migrations. In addition to individual issues people care to raise, we will discuss Edelweiss, the status of the new bulk APIs, and Wolfcon.

• Talking about Folio Q4 builds from the group:

Chris Creswell last attempted, and got Stripes to build - but his system is not showing much in the UI. Permissions are there in the DB he says.

Dale Arntson pulled in the Q4 branch from Dec 30th, and his system is up and running. He noticed there is schema changes with Inventory, but stated the data should load fine using Q3.2 scripts.

Jason Root hasn't tried yet, states there were a lot of bug fixes still in the pipeline.

Q4 official release still on track for a Friday (Jan 10th) release.

• Jon Miller has updated his general purpose file loader application:

https://github.com/jemiller0/Folio

• The times he is seeing when loading data with the Q4 release are as follows using bulk web API:

Added 844923 instances
00:29:15.9154289 elapsed

Added 922039 holdings
00:05:12.7609204 elapsed

Added 999575 items
00:07:03.0633404 elapsed

Load using SQL COPY FORMAT BINARY

Added 844923 instances
00:21:51.0854300 elapsed

Added 922039 holdings
00:01:26.4027687 elapsed

Added 999575 items
00:02:24.3550047 elapsed

• There is a need for updating the bulk APIs for mod-user-import. It is insufficient for the daily loading and updating of users from an Ops stand-point. Patty Wanninger will follow up with that need.
• Dale Arntson stated that Chicago has made available redacted instances/holdings/items data along with reference data for load testing:

http://lib.uchicago.edu/public/ucrecords/folio/chicago_inventory_dataset.2019-12-19.tar.gz

• For Wolfcon, there is talk about an information-only session about reference data, and the work-flow needed for reference data needed in advance before a data load/migration. Anne Highsmith was going to help in this regard.

• Documentation from the group as a whole:

Documentation position - status still unknown?

Data-migration "tips and tricks" for the future? Step-by-step, or a workflow to follow for future implementers.