2019-07-01 - Data Migration Subgroup Agenda and Notes

Issues include:

• APIs in migration to support updates and overlays in order to update record post-migration, to help load the delta if running old production while doing the migration.
  • Might be able to get away without it for bibliographic, but maybe not for circulation or acquisitions.
  • May be simpler to be offline during migration.
  • Need to come to some conclusion about the extent to which we need to support overlays or gap loads.
    • May treat new records and updates differently
    • PostgreSQL offers support of for UPSERT concept, as one implementation possibility, with a query parameter.
    • A more RESTful option might by to have separate support for POST, PUT, etc.
  • Wayne Schneider does not think we have stories for a batch overlay, may need one.
  • Jon Miller would like to see some support for projections, where you state only which parts of the data you wish returned, simplifies client code.
  • Jeremy Huff points out that these would be useful, but may need to prioritize. Probably best to prioritize batch POST and PUT so that provides a path forward, even if it does not take complexity out of the client.
  • One complication is the need to keep track of the relationship between the old IDs and the FOLIO UUIDs, so that you can ensure the correct records get updated and relationships between entities are preserved.
  • There's an outstanding question about where we are in batch POST and rolling out to modules. Please makes comments on  FOLIO-2050 - Getting issue details... STATUS .
  • Need to have good error feedback on what records cause failures, and whether failure is for entire batch or for single records. There may be a tradeoff between performance and granular error reporting, likely more of a problem when data is less controlled.