2023-05-01 - Data Migration Subgroup Agenda and Notes

Working out ordering, now have inventory/SRS loading in 2 days. Still have to do fines/fees.

Thinks it was a simpler data model when they migrated. They had option to do migration direct to db, even though “officially frowned on.” Almost forced to do it because APIs were not very good. Worked out well. Dale did inventory and circulation, John Miller did orders and finance and users. They ended up migrating mid-year, they had to figure out how to do financial data mid fiscal year. Went live January 2022.  They migrated without Elasticsearch.

Some questions

• I think IndexData hosts you?  Did you do test runs to your production FOLIO instance in advance, then wipe out the data, then do the production load?
  • • yes, ID, but had their own test systems. Did ETL to various test systems, had probably 6 systems going.  Could do a full ETL overnight, so could do data refreshes easily.
    • used python, C#
    • easier to understand the data model cuz of doing db migration, didn’t have to go through the api models. Could do something in the UI, see “where it landed” and then could write their db load from that
    • started loading to the ID test system
    • but ID did APIs for the production migration
      • Dale pulled data into python dictionaries and transformed it, then wrote it out in the db level model required for FOLIO (some jsonb, etc). For ID, wrote something to pull the jsonb out of the tables that they needed. 
      • Created synthetic json object for the SRS because no “direct to db"
• I assume you “turned off” your legacy ILS, did a data extraction, and handed off the data to IndexData to load into FOLIO.  How long did your extraction take?  Did you extract more than:
  • bibs/holdings/items
  • circ data (loans, requests, fines/fees)
  • order data
  • user data
    • • get from University feed. do a fresh load every day

• Did you load user data at cutover, or in advance?
  • • they could do in advance, they prepended email in user record with something that would be caught by spam filter.  They probably did a db update to strip that when went live. 
• What sort of communication methods did you set up with IndexData during migration?  Slack?  Did you exchange cell phone numbers, and do any communication via phone?
  • • slack (both with ID and internal)
    • had exchanged phone numbers but didn’t use
    • Bugzilla for issue tracking
      • QA roles were distributed amongst SMEs
      • bugs in data loading got worked out pretty early on, and could iterate
        
        
• How long were you down for production cutover?
  
  • • A couple of days to be partially up. Took a while to load the inventory.  By ~day4 fully up