2019-03-25 - Data Migration Subgroup Agenda and Notes

Dale will have a brief report on a discussion with Wayne and Sebastian about what we can do to initiate work on the FOLIO APIs, while we wait for a data loader to be developed.

Wayne reported on the discussion. He said that FOLIO currently has very few bulk apis; he had talked to some of the developers about what needs to happen to make this possible. He said that they assume that the "singleton" APIs were not sufficient to do migration loading, but that they needed to get metrics on how they were operating to document what migration needs would be. One possible approach is to target the collection endpoints that currently exist to allow get calls and update them to allow put/post procedures. There isn't any time to work on this project during this quarter, but hopefully there would be next quarter. The question was asked – if the changes were made through RMB with they automatically apply to all modules? Wayne replied that the change would still have to be implemented in each module that needed to support it but that the heavy lifting would have been done by changing RMB.

 Open discussion on data extraction, mapping, and loading.

Sharon Beltaine asked how does the migration effort move forward and who does what? Wayne Schneider reminded the group of the WOLFCon decision, which was that with the exception of MARC data, it was the responsibility of the individual institutions to create json to load to the endpoints. The libraries would need to do the extract and transform steps of ETL, whereas FOLIO would be responsible only for the load step. Dale contributed that the loader specs the group has been working on were written based on that assumption; Wayne then asked if there was consensus about what was needed going forward? In other words, should we be developing a single loader or pieces. The group agreed that we needed to move forward on developing performant apis that a loader would use, regardless of whether it's a single loader or multiple loaders. Sharon also asked for an update with what was currently going on with MARC data, saying that she understood that the data import tool will take care of MARC, but wondering whether it would be developed in a timeframe that will work for those who are planning on going live in 2020.

There was some discussion of whether the ability to do bulk loading should be available on every endpoint.Jon Miller (Chicago) felt that it should; Wayne Schneider (IndexData) said that while it could be developed for every endpoint he wasn't sure that he agreed that it needed to be. In the course of the discussion, Jon Miller mentioned his all-purpose file loader and gave a detailed description of its functionality, in response to questions. Jon volunteered to demo the loader at a meeting and will try to be ready to do that next week.

Sharon Beltaine pointed out another need for data migration, the need to have a central point have current information on the state of data elements in FOLIO.