2019-06-17 - Folio F2F: Data Migration Planning & Implementation

1. Develop requirements for migration tools - number of high level tickets around this (broad jira’s) - throughout the project these have not been considered mission critical….stories are at high level and not ready for dev
2. Experimenting - Chalmers, TAMU - populating data - how? Bespoke scripts - some load directly in Postgres tables, some use the APIs.  We’ve had some demo’s of this: John Miller, Texas AM - Circ. We maintain a spreadsheet of ‘tries’. ...but it’s been pretty sketchy and iterative up until now.

Now...coming into the 3rd Quarter...seeing more interest.  

In the most recent prioritization ability to have tools for data migration came up pretty high.

1. 1 million records within four hours - acceptable/benchmark  (MARC/holdings/items)
2. Match and de-dup (initial data load and refinements.  Doesn’t want to have to wipe and reload each time. Prefers to match on certain fields
3. Bulk export is performant - Data needs to come out of the system and go back in - in a performant manner.

  They do usually do at least 2 migrations during one migration.  (ByWater)

  The de-dup without having load fully load over and over would be a good tool for any implementer.

Q: existing data import tool - what is the name?  Source record storage surfaces into inventory...what is this called?  Has an updated map (updated last week).

There was a problem w/dragging down the system?

Bogged down currently two places:

• Getting record into source record storage….
• Turning the MARC into inventory

No matching, just creation.

Requirement of migration loader - ability to have full mapping of items and holdings in inventory.

...and need to match with Bibs that have been loaded.  “Addressable Mapping” - able to tell this process to pick up a map.

Chicago has been working on this.  They’ve found a simple map is not robust enough is not enough.  They’ve found they need business logic around the map. They need to make choices.  Not all data meets current standards...and they have to address this during transformation.

There is more flexibility in FOLIO than in current systems...so they are creating logic for how to populate the new fields that are available.

E.g. if you have x, y, z...we want your material type to be A

At Chicago they are loading directly into inventory.

Legacy ID mappings -

Needs to be part of the migration process.  Part of load? Part of transformation?

Chalmers -  it’s part of transform...T. uses a map of old IDs and new IDs.  Loading items and holding accesses the old id/new id map.

However - they are not loading MARC.  

Question: How much transformation/cleaning/mapping has to be done BEFORE loading? where is the dividing line.  ...migration tool set vs work you have to do ahead of time?

E.g.  location and service points...item types...initially have to be figured out - locations, service desks and what is the map from old system to new system.  The library has to do that. All of that has to be done by implementing libraries.

Does that mapping (above) get applied outside of migration tooling...or will the migration type use that and make the transformation.

Data migration subgroup is trying to collect reference data….26 different tables that have reference tables.  Tool community can use.

Marc and holdings to instance inventory - is a community tool….but should it be?  Don't people want to build their own maps.

Question: vision for data migration load MARC and have folio generate the instances?  (The default map is not going to get them inventory instances they need - Chicago)

Requirement - they want to be able to use a customized map.

Thought: Data goes to tool with map...process ingests it’s without having to parse it outside of Folio.

Chicago - generate MARC, generating instances (using the tools they developed)  outside of folio w/intention of load them…

Data migration vs import...we’ll always be importing.  That is different than migration….it has loan and patron data which has to be connected.  Two different tasks.

Now - can load user data….ticket to make it bulk. (instead of one at a time)

People are loading open orders experimentally.

People have loaded open loans….vendors.

MARC is getting lots of discussion because of volume and nested data.

Devs need to look at performance.

Want to see path to improving APIs.

Being able to De-dup….matching and mapping.

E.g. list of patrons...does it have to be formatted outside of FOLIO?

Comment - default mapping, default loaders - imply there is something other than DEFAULT.  If we can’t apply our own standards to our own data...that is a mistake.

Comment - Regular ongoing feeds...e.g updating patrons.

Comment - putting this into a timeframe...what do we need now or very soon...and other sort of refining elements we can wait for.

The smaller group will discuss these thoughts.

We need to do a better job of documenting...so people know where to find information ….

We need to be able to specify succinctly where we need development to help the early implementers.

Source record storage vs MARCat - nobody will be loading/migrating to MARCat.  Source record storage will make a copy into MARCat.

Keeping a record of experiments...sharing

-go live for early implementers

-FOLIO  as a product that can be adopted.

During PC meeting/discussion

Success of early adopters...these tools can help.

Long term - working toolkit is an investment into after 2020