[FOLIO-406] Design module life cycle Created: 07/Dec/16  Updated: 11/Oct/19  Resolved: 19/Apr/17

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Umbrella Priority: P3
Reporter: Heikki Levanto Assignee: Heikki Levanto
Resolution: Done Votes: 0
Labels: sprint6, sprint7
Remaining Estimate: Not Specified
Time Spent: 3 days
Original estimate: Not Specified

Issue links:
Relates
relates to UXPROD-1815 support for upgrading schemas without... Closed
relates to UXPROD-2120 support for upgrading schemas without... Closed
relates to FOLIO-464 Design and implement module upgrade p... Closed
relates to OKAPI-232 Initialize modules Closed
Sprint:

 Description   

Modules need to be installed on the system, enabled for tenants, upgraded, disabled, and deprecated and removed. At many of these points default configurations, permission sets, and other things may need to be created, modified, or deleted. How will all this be organized?



 Comments   
Comment by Heikki Levanto [ 07/Dec/16 ]

My most pressing worry is related to the permissions and permission sets that must come from ModuleDescriptors and find their way into mod-permissions. But there is also the question about creating the necessary database tables, and probably a lot of other things that needs to be set up. We need to have a systematic way to handle all this. And this all is related to the app store concept too...

Comment by Heikki Levanto [ 14/Dec/16 ]

I think we can at least identify some stages a module must go through. Later we can try to define all the things that need to happen at each of the stages. Here is a preliminary list:

  • Someone writes a nice new module
  • Module gets installed on our cluster
  • Module gets deployed on some node(s). Some admin or algorithm needs to decide which node(s)
  • Module gets enabled for the first tenant, for the first time
  • Module is used
  • Updated version of the module gets installed
  • Updated module gets enabled for the first tenant, data upgraded
  • The old version of the module gets deprecated, new tenants can only get to the new one
  • Module gets disabled for a tenant, data left behind, or it gets deleted
  • Module gets removed from the system
Comment by Heikki Levanto [ 16/Dec/16 ]

Some loose points about installing a module on a cluster:

  • It must be possible to install multiple different versions of the software on the same cluster.
  • The names of the executables or docker images must be different. This can be done by including the version number in the name, like mod-something-v-3.1.4
  • We can probably get away without any special intialization code at this point
Comment by Heikki Levanto [ 16/Dec/16 ]

Some points about enabling a module for a tenant, for the first time:

  • This is where a lot of init stuff needs to happen:
    + It may need to create tables in the database for its data
    + It may need to create a default configuration
    + It may need some manual tweaking in the configuration before it can be used. Since the module can not directly invoke an adminstrator to do this config, it needs to set a flag that causes an error condition until the admin clears this flag.
    + The module may need to insert data into othe modules.
  • The module needs to do something about setting up its part of the UI
    + It probably needs to trigger generating the UI bundle
  • The permissions and permission sets defined by the ModuleDescriptor need to be installed in the permission module for that tenant.
Comment by Heikki Levanto [ 16/Dec/16 ]

Upgrading a module may need to do many of the same things as a first-time enabling. But with the caveat that we may already have important data in the system that must be preserved.

I hope we can get away without supporting downgrading, that could quickly get messy.

Comment by Heikki Levanto [ 16/Dec/16 ]

Since the module itself is the only one who can know what configuration, tables, settings, and data it needs to install, I think it must be responsible of doing it. So I propose that we add a well-defined service endpoint that does all that. The same endpoint may also take on the responsibility of upgrading the data, settings, etc to a newer version. It should get two parameters: The currently enabled version (if any), and the version about to be enabled.

I propose we use something analoguous to our .../health, maybe .../upgrade that every module should implement if they wish to do any initialization or upgrading at all.

I have proposed above that we tell the module what version was previously enabled. This is not strictly necessary, each module could store that data in its own configuration, and take it from there. But that means that every module should do the same kind of thing, I think it might be better to do it once, in Okapi. Okapi will have to remember what modules are enabled for what tenants, so it can understand the versions too, and pass them along to the .../upgrade endpoint.

Comment by Heikki Levanto [ 04/Jan/17 ]

Some loose thoughts after today's discussions:

  • We certainly need a API call to be invoked at the time a module is enabled for a tenant
  • Probably also one when a module is disabled. This one could take a parameter to tell if it is a "remove" or a "purge". That is, if it should delete all data it has created.
  • We also need one for upgrading
  • These functions may need to know some global configuration settings. But we can not know what they would like to know, so the modules will have to ask for what they need, most likely by making API calls to a (global) config module. We already may have something like that in the RAML builder stuff.
  • Not every module needs to do any initialization. It is perfectly OK for a module to return 404-NOTFOUND on these requests. But if they return any other error response, Okapi must consider that to be a serious error.
Comment by Heikki Levanto [ 04/Jan/17 ]

Some loose thoughts about dependecies and permissions:

  • The init routine can only be called when the dependencies of the module are satisfied. That is how it is already, Okapi will not allow enabling a module that has missing dependencies.
  • That also means that the module is free to make other API calls to any module that is listed as its dependency.
  • In order to make such calls, the module needs to pass a suitable auth token, since the call will be passing through the auth module. This token must contain the necessary permissions needed to initialize stuff, for example "module-foo-initialize".
  • The only place this token can come from is with the original request to Okapi to enable the module for this tenant. Who ever makes that request (the admin console?) must have received such a token - with the correct tenant in the token!
  • The design of the auth system allows for a function to change tenant - that is what happens when a user logs in as a user of one tenant. It should be possible to write a small helper method in the auth system, something like make_this_request_as_tenant_X. This helper should be heavily guarded, probably with its own permission bit "may-behave-as-any-tenant". This bit could be set to the admin console.
  • All this auth token operations must be done outside of Okapi, since Okapi itself should be fully functional even if no auth modules are loaded.

All in all, this looks a bit messy, but perfectly doable. It would be cleaner if we had implemented permission system for Okapi's own operations too, like I proposed in OKAPI-194 Closed

Comment by Heikki Levanto [ 04/Jan/17 ]

How to get there from here?

This design requires some changes to the auth module, and the admin console, but surprisingly little for Okapi itself:

  • Before enabling a module for a tenant, check that we have a running instance
  • Make the init call to the module, passing any X-Okapi headers along
  • If it returns 404, log a warning that the module does not support init
  • If it returns any other error, fail with that error
  • If no error, enable the module.

The disabling call behaves about the same way, and has the same permission complications, as does the upgrade thing. That needs more thinking.

Even without the changes in auth and admin console, the init calls can be used for developing, testing, and demoing. If we have an Okapi setup with no auth module, Okapi can still make the init call, and it can intialize some internal state, like setting up a database. The init call can even make calls to other modules, but without an auth module, no permissions will be checked.

If we have an Okapi setup with the auth module in place, we can still fake things a bit. If the module has a module-level permission to run its own init, things ought to work. It can even have module-level permissions for any operations the init call needs to make. That way, we might be able to use the current admin console, without any log in or token manipulation, since the auth module will make a "not-yet-logged-in" token when it sees a request without auth tokens. This token grants no user-level permissions, but allows use of moduile-level permissions.

Comment by Heikki Levanto [ 05/Jan/17 ]

Some more thoughts:

  • We should quickly come up with the canonical names for these services
  • If we need permissions in the ModuleDescriptor for the init call, where exactly should those be? Not in a routing entry, since these will not be called via normal routing. A new section for system level calls?
  • If we have something in the ModuleDescriptor about the init service, we can conclude that the module promised to support it, and therefore fail also with a 404.
  • We just decided that the special services that are called from Okapi itself (as opposed to Okapi routing external request to them), must be prefixed with _/, as in _/health or _/tenant. So, to enable a module for a tenant, Okapi POSTs to the _/tenant in that module.
Comment by Heikki Levanto [ 05/Jan/17 ]

I created OKAPI-232 Closed for the actual changes in Okapi.

Some points that came up in slack discussions:

  • The _ / ` services are per module, and not per (abstract) interface. They can still depend on other interfaces.
  • The / _ / tenant should probably POST the TenantDescriptor, so that we have something to post.
  • the GET on / _ / tenant should tell if that tenant has been initialized. If a module does not need any persistent data, it will not keep track of whom it has been initialized for, and can not return that bit. But in that case it does not need any init service, and will (probably?) get away without implementing the whole / _ / tenant interface.
Comment by Heikki Levanto [ 05/Jan/17 ]

The permission stuff may not work as easy as I had hoped. The reason is that the admin console makes this enabling request directly to Okapi, and Okapi makes the init request directly to the module, so the auth module is never involved in the process. That means that the service in the module can not depend on permissions, and can not have module-level permissions it may need for further calls.

The proper way would be to implement OKAPI-194 Closed , so that the auth module can be part of Okapi's own requests too.

Time being a workaround can be to give the admin user all necessary permissions, somehow.

Comment by Heikki Levanto [ 06/Jan/17 ]

I came to think that there is another workaround, at least for the first few packages that may need module-specific permissions. The module can make a regular Okapi call to itself, and when it serves that request, it will have the permissions it needs. This is a messy thing to ask the module developers to do, but we can possibly get away with it in the first module or two.

Comment by Heikki Levanto [ 06/Jan/17 ]

I am sure we can work around the permission issues. The next big question is upgrading modules. The first step is to install the upgraded version on the cluster we are running on, and to deploy it. No problem with that, Okapi can happily support two versions of the same module running at the same time. The problems come when we wish to switch a tenant from the old version to the new one. That poses two problems:

1) The new version may have different configuration settings, different database layouts, etc. Only the module can know what data needs to be upgraded, but someone must tell it when to do it.

2) I think we need a special upgrade service in Okapi, and in the admin console. We can not simply disable the old version and enable the new one. If we try to enable first, we run into conflicts. If we disable first, we need to disable everything that depends on this module, and re-enable. That is bound to go wrong.

Comment by Jakub Skoczen [ 06/Jan/17 ]

1) the versioning will tell us if there are breaking changes, and if so we may need to perform migration, question is whether our system should provide ways to migrate

2) potentially, I think whatever we do we must always protect the data and be able to revert to the previous version, so for major version upgrades where compatibility is broken it would seem that cloning the data is the way forward

Comment by Heikki Levanto [ 12/Jan/17 ]

Some more thoughts about upgrading:

  • We clearly need to upgrade modules at some point.
  • Okapi already supports running multiple versions of a module at the same time, and enabling different versions for different tenants. This is good, for we can not expect all tenants to schedule upgrades at the same time.
  • I had earlier assumed that upgrading for a tenant would simply mean disabling the old version and enabling the new one. This is no longer true. Disabling a module that other modules depend on will not work, and enabling the new one will call its tenant interface to initialize its database, which is not good.
  • So, I believe we need a new Okapi service to upgrade a module selection for a tenant. And a new system service so the module will be able to upgrade its database, and what ever else it may have.
  • I don't necessarily like the idea of cloning data at every upgrade, we may have gigabytes of catalog data, cloning that can be expensive.
  • Experience shows that downgrading is often tricky
  • I would hope that we support backups per tenant, so if an upgrade fails, we can at least revert to a backup.
Comment by Heikki Levanto [ 23/Jan/17 ]

Ok, we have some for of tenant interface that allows the module to initialize its tables. I have been speculating about upgrading above. Now I start to get worried about what else a module needs to do when it gets enabled for a tenant and/or installed for the first time. Some examples:

  • Create permission bits and roles in the auth subsystem
  • Create some initial data in its own tables
  • Create some intial data in some other module's tables

Most of these can be done by regular requets through Okapi. The problem is that when the tenant interface is invoked, we come in through the admin interface, without any auth token or user login. This makes it very hard for the modules to make further requests. I have a solution for that, but it requires using the auth module for admin stuff too, internal modules (Okapi-194), some extra facility for the admin to make requests on behalf of a real tenant, etc. All possible, but will take some work.

Comment by Heikki Levanto [ 19/Apr/17 ]

I think we have moved on from this. We have a tenant interface, and a mechanism to pass permissions, etc. let's open new ones when we need to.

Generated at Thu Feb 08 23:05:32 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.