[FOLIO-2604] SPIKE: investigate upgrade/migrate strategy for reference data Created: 18/May/20  Updated: 22/Jun/20  Resolved: 22/Jun/20

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P3
Reporter: Jakub Skoczen Assignee: Wayne Schneider
Resolution: Done Votes: 0
Labels: devops-backlog
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Relates
relates to FOLIO-2655 Upgrade fails on failure to update re... Open
Sprint: DevOps: sprint 90
Development Team: FOLIO DevOps

 Description   

Module tenant initialization for both first-time init and for upgrades calls the _tenant interface, and tenant parameters included in the Okapi install API call are passed to the module, including loadSample and loadReference.

As currently implemented in most modules, this will cause the module to attempt to load all reference data on module upgrade (not just new data). New records will be created if needed, and existing records (matched by UUID) will be overlaid.

Due to this, issues arise if the tenant has altered or deleted any of the reference data loaded by the module when it was first enabled. Any changes will of course be overwritten with the system default, and deleted records will be re-created.

More subtle problems arise if the record type in question has data constraints (for example, the requirement that a particular property be unique), and the tenant has created a new record of that type which causes a conflict with incoming reference data. As currently implemented, this kind of conflict causes the module upgrade to fail, potentially leaving the tenant data in an inconsistent state.

These kinds of issues would very likely also arise if an operator specified loadSample=true in an upgrade, but that is currently untested, and seems like an unlikely use case, at least for production.



 Comments   
Comment by Wayne Schneider [ 18/May/20 ]

Discussion paper for SysOps SIG has been posted: https://discuss.folio.org/t/reference-data-and-upgrades/2858. Will collect feedback from SIG and discuss at the meeting on 22 May.

Comment by Wayne Schneider [ 27/May/20 ]

Proposal on Wiki: https://folio-org.atlassian.net/wiki/display/SYSOPS/Upgrades+with+Reference+data

Comment by Wayne Schneider [ 02/Jun/20 ]

Proposal will be discussed in TC on 3 June.

Comment by Wayne Schneider [ 09/Jun/20 ]

Discussion in the TC was wide-ranging, and the topic will be discussed again, to be sure. Jakub Skoczen asked for a description of the basic operational issue that needs to be solved. In a nutshell:

  • By using the UI or the APIs (that is, in sanctioned ways) it is possible to create user data that conflicts with incoming reference data on module upgrade
  • This conflict causes loading reference data to fail
  • This failure aborts the entire upgrade. Okapi continues to route to the original (un-upgraded) module set for the tenant.
  • The storage module schemas, however, may have been fully or partially upgraded, so system storage may be left in an inconsistent state. It is likely impossible to know the state of the system storage without Okapi logfile analysis and module upgrade code inspection.
Comment by Wayne Schneider [ 22/Jun/20 ]

Created issue FOLIO-2655 Open and put into Core: Platform backlog.

Generated at Thu Feb 08 23:21:53 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.