2022-06-13 - Data Migration Subgroup Agenda and Notes

Jiras

Load vendor files with Data Import app, even if doing under the hood with apis, use "Data Import app endpoint"

Brooks: Should be a way to bypass the version check for batch updates/batch loads
Tod: Discussion on what the solution should be needs to happen.  Bypassing optimistic locking is one.  Partial load and an error report is another (so all other records load except the ones with version conflicts)
Jeremy: If batch fails, we post each record on its own so we can see which one failed.
Tod: if the apis could do that, it would be nice
Brooks: in PO group presentation, they had provided a way to bypass optimistic locking, but then didn’t implement it. The “solution” of looking up version number and updating your incoming record to match it is not feasible for 10K+ records

Tod: Would some small number of people be willing to write up the operational problem, and the obstacles we’re running into.
Ingolf: Is this a pain point that TC should review?
Tod: yes, but that’s more long-term.  Needs escalation now and a short-term solution

Jeffrey: Does the same as Jeremy, trying to post each record in a failed batch/doing the checks.  Does some of each depending on process.
Dale: Every institution creating their own workaround is waste of resources, solution should be in the apis.

Brooks: the apis seem to have been overlooked when considering optimistic locking. Works for Data Import because if a record hits the problem, code pulls it again, does its update, and reposts.

Tod: We don’t need to provide a solution, we need to say we have an operational problem. Is a problem for anyone migrating. Batch updates (dump, update externally, reload). 

Probably this group does not have the power to actually change this. But we could raise awareness.

ACTION FOR THIS GROUP: Small subset of us (Ian, Jeremy, Ingolf, Theo): Create a document that describes our troubles with the current implementation of optimistic locking.  Doing this outside of the ticket so it can be seen by all and hopefully agreement that it is a problem that needs solving.  Document: Optimistic locking makes inventory-batch APIs upsert loads fail

Work on this document and report back to this group in two weeks.

Advice about amount of resources to allocate for loading orders, number of pods?  We only have one mod-orders/mod-orders-storage composite-orders.  Jeff gets ~400-500/hour.  Stanford getting 120/hour.

Question about finance data/budgets.  Working from spreadsheet to form json. For budgets, required field allocated is supposed to be used when posting the real budget allocation? Or need to use finance transaction endpoint when allocating budgets (Erin’s advice?). Duke allocates the $$ after the budget is created (a secondary step), with a financial transaction. Jeff sets allocation to 0 when creating the budgets. Julie then allocates money to them. Maybe ask on Acquisitions SIG channel #folio-acquisitions-sig