2018-07-09 - Data Migration Subgroup Agenda and Notes

• Christie's update on progress in the MM group on the gap analysis:
  • For this review they formed 2 subgroups: Instances and Holdings & Items
  • They'll finish this analysis during their last meeting today (7/9/18)
  • They created two new columns on the 'gaps' tab:
    • MM Sig Review (for comments)
    • Priority (Most ended up being 'go-live', some 'not needed', and some that need further discussion with other SIGs)
  • They will be recommending a new element that was heavily discussed during this meeting:
    • 'Statistical Search Code' – a repeatable element applicable to any instance, holding or item record
    • They found many different uses for this across institutions
    • We wondered about the impact to development
    • How is this element different than tags? → tags won't be viable for high confidence reporting. The understanding is that tags will be more organic and not something for high confidence reporting.
      • If there could be be 'reserved' tags that have an additional layer of control...that might work.
    • The data structure doesn't support things we need to know about our data. The Statistical Search Code remedies this.
    • We recognize that data loaders will have to incrementally change as the data model evolves. Once the basic framework of how the data loader will work is established, the changing data model will probably just require changes to the data loader config. (rules).
    • The MM SIG is already having conversations with circ & acqu. SIGs about elements that impact them. (e.g. what does circ need in an item record).
    • Role of Data Migration SIG in all of this? We identified gaps and pass to SIGs. We don't have a hands on role in ongoing conversations as long as SIGs are aware of gaps and are making decisions about them...making sure conversations happen where needed.

Patty and Tod have put together a set of user stories for the data loader: (feedback on this document is welcome)

• They've compared data migration loader specs to batch loader specs for inventory because much of it is the same.
• After discussions it was decided (by Yakov ultimately) that the data migration loader & batch loader for inventory should be treated as the same thing. The only requirement that might have been different is performance. If the batch loader can meet that, the overlap is 100%.
• We need a set of rules to transform MARC into inventory - need a common way of specifying rules for batch and migration.
• Should be forming best practices and common strategies meaning...make the loading process uniform across data modules.
• Institutions migrating data will be required to produce a json document per record for data that needs to be migrated for each module other than inventory. This expectation can allow for commonalities across modules if they can always expect JSON.
• We should specify that when we load records, Folio will take care of getting the data into any module that needs it (MarcCat, Storage). For now this is just inventory.
• Action items for the data loader user stories:
  • Possibly have Ann Marie attend our meeting?
  • Add to the document: the idea that for loading data, institutions will be responsible for getting their data into json format, except for inventory, which will include an option to convert marc to json.
  • Add to the document: Include a transformation for MARC into json for inventory
  • Add to the document: Folio will be responsible for getting data into all required modules via the data loader.
• Transformation is already a work in progress. See this spreadsheet. A POC has already been implemented. Ann successfully tested it.

• Action Item: Cheryl added a 'gap tab' to the loan spreadsheet and added some possible U Chicago gaps. Some of these might reside elsewhere. Please take a look and provide feedback.

• The plan is to present this to the RA SIG. Action Item: Next week we'll talk about when.