2018-04-23 - Data Migration Subgroup Notes

Begin discussion of loan data migration.

Anne Highsmith volunteered to go to to the UM SIG with Chris Manly to present the User spreadsheet.

A discussion began of the Composite_to_FOLIO_Loan table: https://docs.google.com/spreadsheets/d/1DbcGt93trebGlS1MegVeyGggDzomWphoKwqMGs-34F8/edit#gid=1944740608

Anne brought up a number of questions:

-What is the flow of migrating the loans? Anne remarked that she does not want to do a get command to get every FOLIO UUID for patron group, item type, etc.

Wayne replied that he doesn't think migrators will use the API. There's a question of whether all that data needs to be in the loan record - the data is in the user interface but could use GraphQL to generate the object.Maybe some of these fields won't survive V.1.

Tod said UChicago is thinking about setting up a staging database; he described it as E - L - T - Data would be loaded into FOLIO and the FOLIO UUIDs passed back to the staging database. There would be a set of conversion rules that reference a lookup table.

Chris M thought there is a need for a separate API for bulk loading

Anne explained Column A of the table is hyperlinked to the github for the loan json data element. There are some differences compared to the JSON in the dev.folio.org Github. Wayne said he would take as an action item to look into it.

In the current scheme, most of the data in the loan record is from FOLIO. Very little is coming from the old database.

Staff people in some migrating institutions are interested in keeping/migrating location information..

Status of loan from Voyager - list of possible item statuses will be IDed by UUID. Loan transaction record will contain UUID for "checked out" item status record.

Dale said they would like to preserve metadat like LAD etc.

Wayne pointed out the Shared repo of RAML that includes the schema for metadata. https://github.com/folio-org/raml/blob/master/schemas/metadata.schema

Discussion followed about each institution deciding what loan history details to bring over, and whether users includes operators' details. UChi confirmed that their data is combined; Voyager data has separate user and patron files. FOLIO has a single repository of actors with UUID - associated with permissions as well as circulations.

Questions were raised about the completeness of metadata - ie. some things say "bulk load" or details have been lost in previous migrations. Anne said TAMU uses groups of operators that attach to groups of records, to process YPB invoices, for example. They need to re-create this functionality.

If metadata recording has gaps - i.e., the operator is now no longer working at the library - do we care?

Wayne summarized - Generally, libraries prefer to migrate legacy record metadata when it's appropriate

Tod is interested in looking at Mod_circ_storage.

The metadata is more complete in the mod-circ-blob https://github.com/folio-org/mod-circulation/blob/master/ramls/loan.json

Chris M suggests creating a migration user, then if wanted to preserve legacy data, it could be put in username as a string, or create migration users.

Anne committed to doing some clean up on the spreadsheet https://docs.google.com/spreadsheets/d/1DbcGt93trebGlS1MegVeyGggDzomWphoKwqMGs-34F8/edit#gid=1944740608, which she completed after the meeting. The spreadsheet is now ready for entry from other vendor systems.

Some clarity is needed in the terms used in the circ_storage JSON - usage of Due date, return date, system return date need clarification from RA SIG.

The schema can be updated, sometimes with no code implications.