2018-11-26 - Data Migration Subgroup Agenda and Notes

Suggestion for next meeting: Some weeks ago we were busy with the user data. In the meantime a little bit of the data model has changed. Therefore I adapted the spreadsheet composite_to_folio_user.xlsx accordingly. https://docs.google.com/spreadsheets/d/1jiMJz-wTBhSkpVrJOFQh2tGmm9-AXATphu2dMcACkws/edit#gid=1406669146

As a basis I used the current JSON files: https://github.com/folio-org/mod-users/tree/master/ramls

The file https://github.com/folio-org/mod-user-import/blob/master/ramls/schemas/userdataimport.json is 10 month old, so I assume that we should take the files in mod-users into account.

NOTES

• Data Migration subgroup would like to know where to upload data on user Affiliation (e.g., college, department, faculty, staff, student)
• Charlotte suggested adding this to the Gaps page on the User Management data elements mapping sheet
• Tod will contact User Management SIG to find out more about Affiliation data plans
• Anne: Would be useful to have confirmation that Data Import tool can be used for batch loading data from the registrar
• some questions as to whether some metadata fields will be cleaned up in the Data Import tool
• JSON schema inconsistency in mod-user versus user-data-import; patron group id should be patron group UUID, for instance
• Patty: Data Import tool is MARC data
• User-Import Endpoint tool - we have several questions for PO Khalilah
• Would be useful to have both Khalilah and Ann-Marie attend a Data Migration subgroup meeting
• We will review the ORDERS data elements in the Data Migration subgroup meeting in 2 weeks (Dec 10)
• Patty: we need to continue data import and export testing and discuss results
• Data Migration subgroup would like to have a Product Owner; perhaps getting a SysOps PO and asking that person to also represent Data Migration would help
• Dale: regardless of PO issues, we may need a closer relationship with the developers of data migration tools (data import development team)
• Institutions will need to do an extraction and transformation of source data prior to using tools delivered by Data Import group, but we may be able to add JIRA tickets to request additional features
• Wayne: we will handle MARC records natively with a batch data loader, but institutions are individually responsible for preparing the rest of the data to be loaded
• Wayne: none of the modules offer an interface for loading data
• Patty: data loaders for each data domain (e.g., circ, requests, vendors) are a priority
• We need a conversation with Cate, Jakub, and Harry on the need for data loaders
• Wayne: we might be able to accomplish a lot of what we need to do by extending the RAML module builder to support streaming
• Dale: institutions need the support of data loader(s) for their implementations
• Dale: let's review the existing APIs to see what data loading capabilities we have already