2018-03-05 - Data Migration Subgroup Notes

5 min

Welcome, identification of note taker and next week's convener

@Chris Manly Arntson

@Chris Manly will take notes. @Ingolf Kuss will host next time.

Report on inventory discussion

@Charlotte Whitt, @Theodor Tolstoy (One-Group.se), @Anne L. Highsmith

• Theodor could not make the meeting due to illness

• Anne reported on her notes from the discussion:

  • The data definitions are described by the json schemas in the module definitions. At the moment, they are very fluid

  • What we have now was designed for the alpha release, and was focused on what was needed to get alpha up and runnning. We should expect changes before we get to beta as requirements develop.

  • The spreadsheet from the MM SIG illustrates many pending changes (link needed)

  • Some of the things listed in that spreadsheet map to things we've listed as needs, but there may be things we need for migration that aren't yet in that spreadsheet.

  • We need to talk more about IDs and linkages between records: can we use UUIDs for linkages, or do we need to support legacy IDs? There may be a need to support accession-style call numbers, but we'll discuss that when Theodor is available.

  • Location handling is very much in development, so we can't do anything with that yet.

• Discussion

  • Clarification: the high-level data structure (what lives in which module) is not in flux; just the details of what data elements are there and how they are represented.

  • We discussed the related issues of the general need for human-readable IDs for records (things that could be shared over the phone or written down on paper) and the need to make sure that linkages between records (bibs/holdings/items, for example) are intact across the data migration when the primary key will be changing from a legacy integer ID to UUID.

  • The human-readable ID is a requirement

  • We should handle the two issues separately, since the legacy integer ID may not be the best way to meet the human-readable requirement in FOLIO. It would be difficult/awkwards to generate "new" legacy IDs for records that originate in FOLIO.

  • The MM SIG inventory data elements spreadsheet includes a "resource identifier" field that would hold the legacy ID for inventory

  • The data migrator can create the UUID outside of FOLIO and assign it on record creation, which makes maintaining linkage on data migration more viable. It would likely need to be tracked in an external/intermediary database as part of the migration process

  • We might be able to build some common tools for data migration, including UUID creation/mapping/tracking. How common remains to be seen; it will depend on variations between source systems and variations in local practice between institutions.

  • The need for human-readable IDs has been discussed in MM-SIG. @charlotte will review what @jakub and devs have come up with for this issue.

  • For consortia that have a union catalog that would be separate from FOLIO, there will be an ongoing need to map legacy IDs to UUIDs because that union catalog is keyed off the legacy ID

Discuss approached to gap analysis

Lynn Whittenberger

• This also came up at the Metadata Management SIG. The SIGs are interested to hear where the data model falls short and where system needs aren't being addressed.

• It would be overkill to have a cross-SIG group specifically for this. Rather, we can take issues to specific SIGs as needed.

• We will need to pull in members from SIGs that are not already represented here, when we come to subjects that are relevant.

• There is no common view across the spreadsheets. Can we consolidate it, and does that happen within the SIG or this group? Probably within this group? Not a task to be done withing the subgroup meeting, but by a smaller set of folks. @Anne L. Highsmith will take a first pass at creating the composite view.

• When we have specific requests to add data elements, we work with the product owner for that module

• We should keep the Product Council informed with high-level documents, but not get them involved in details unless there are political considerations to work out on an issue.

• When this group makes a recommendation to a SIG for changes, SIG would pick it up, invite folks from this to the discussions. We'll probably want to discuss things like outliers and come to consensus about what things we recommend get added. There are elements that are calculated fields, or are currently there to aid performance that may not be needed anymore. The Logical mapping should take precedence over technical history. (If a technical piece in the old system is there for techncial reasons, we should articulate the functional need, rather than the techncial requirement, in case a different technical approach makes more sense in FOLIO.)

Discuss how to approach missing data elements in modules.

Everyone

(Not discussed this week)

Useful spreadsheets from Highsmith, Klute, Tolstoy:
https://drive.google.com/open?id=1aUMIqc4SwRzOGR4yPzrRSRElwM2CQ3uY

MM Sig spreadsheet - column I: Comments/Questions from the Data Migration Subgroup https://docs.google.com/spreadsheets/d/1LkVIglt3n3y2bckDoamxWQ73TCsmvw43qrAbLfCQ_O0/edit#gid=952741439

Data migration subgroup - gap analysis cross SIG working group

@patty.wanninger, @Lynn Whittenberger (convener of MM-SIG)

(Not discussed this week)

Initial questions:

1. What is the data migration subgroup timeline for getting mappings completed?

 - is the intent to just come up with lists of data elements in existing systems that don't have a corresponding element in FOLIO, or will there be a more formal analysis?

2. Is a 'grass-roots' cross-SIG group the best way to approach resolving those gaps, or is that something that may need to be looked at from a higher level?