[FOLIO-1953] SPIKE: propose an approach for scheduling tasks in Okapi Created: 09/Apr/19  Updated: 09/Jul/20  Resolved: 06/May/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Jakub Skoczen Assignee: Adam Dickmeiss
Resolution: Done Votes: 0
Labels: platform-backlog
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File okapi_cluster.png    
Issue links:
Blocks
blocks UXPROD-1586 Send patron notices for time based lo... Closed
blocks UXPROD-1587 Send patron notices for time based re... Closed
blocks UXPROD-1636 Make Hold Shelf Expiration Date Respe... Closed
blocks FOLIO-1786 SPIKE: evaluate "init" tokens as a w... Draft
Gantt End to Start
has to be done before UXPROD-82 Automated Patron Blocks Closed
has to be done before UXPROD-87 Loan: Aged to Lost using SET COST Closed
has to be done before CIRC-292 Set Item Status to "Pickup expired" w... Closed
has to be done before UXPROD-1085 Automatically Anonymizing/Scrubbing L... Closed
has to be done before CIRCSTORE-133 Request Expiration Job Takes Too Long... Closed
has to be done before UIREQ-246 Hold Shelf Expiration Date Should Res... Closed
has to be done before UXPROD-105 Future Fees/Fines: Update current aut... Closed
has to be done before UXPROD-499 Future Fees/Fines: Collections agency... Draft
has to be done before UXPROD-527 Loans: Recently returned Draft
has to be done before UXPROD-1858 Future Fees/Fines: Collections agency... Draft
Relates
relates to CIRC-272 [SPIKE] Investigate scope of work for... Closed
relates to FOLIO-2010 Alternative Scheduling Approaches Closed
relates to MODSENDER-8 [SPIKE] Investigate sending multiple ... Closed
relates to OKAPI-730 system call timer Closed
relates to FOLIO-889 Develop a strategy to implement sched... Closed
Requires
requires CIRC-272 [SPIKE] Investigate scope of work for... Closed
Sprint: CP: sprint 62
Story Points: 5
Development Team: Core: Platform

 Description   

We currently only have the scheduling capability in the storage modules as provided by RMB. There is a need to have a scheduling capability also in the BL modules (e.g mod-circulation, please link the appropriate stories).

Since BL modules have no persistence and may run in multiple instance we need a solution that will ensure the scheduling is coordinated (the scheduled event happens only once).

Also, any scheduled activities needed to be granted permissions to interact with other module endpoints.

DRAFT PROPOSAL

One option is to do scheduling in Okapi;

  • extend the module descriptor with a a new "handler" type that would also include a property that would indicate to Okapi that the handler should be called periodically (e.g "tick" = minute,quarter,hour) (should this really be part of the tenant API?)
  • have Okapi elect a designated scheduling instance that will call the "handler" and ensure the handler is called only once
  • have the scheduling Okapi instance call the "handler" with a token that include permission reported though modulePermissions.


 Comments   
Comment by Marc Johnson [ 10/Apr/19 ]

Jakub Skoczen Kostyantyn Khodarev Cate Boerema Below is a quick write up of what I think are the initial requirements 1 and possible future extensions. Do these seem reasonable and fit with what we talked about? Have I missed anything?

Initial Requirements

  • All tasks 2 are defined when a module is enabled for a tenant
  • A task can be granted permissions which allow it to make requests to other modules
  • Each execution of a task happens on only one module instance
  • A task can be scheduled to execute at an interval chosen by the module
  • Smallest execution interval can be 1 minute (no need for seconds precision, in practice this is likely to be 5 minutes or more)

These closely mimic the current scheduling implementation in RAML Module Builder. The main differences are that it is tenant aware (rather than running as soon as the module is deployed) and limited to running on only a single instance.

Possible Future Extensions

  • Schedule can be more complicated than only an interval between executions
  • A module can register new scheduled tasks (e.g. could register a specific task for expiry of a single request if we wanted)
  • Scheduled task can be run once (or a defined number of times)
  • A module can remove scheduled tasks (may need to consider ownership of tasks)
  • A module can change the schedule for a task

1) I've tried to defer any description of how this could be accomplished to a later comment
2) I've used the term task to be consistent with other parts of this issue

Comment by Kostyantyn Khodarev [ 12/Apr/19 ]

Marc Johnson
Proposed solution fits our needs
It's possible to implement patron notices on top of described scheduling functionality.

Comment by Cate Boerema (Inactive) [ 17/Apr/19 ]

Marc Johnson maybe you can take a look at the requirements outlined in the issues I've linked up just to double check it's all covered. For example, do we want to have the Okapi scheduler handle the closed library due date management stuff? This is desired for hold shelf expiration period calculation. What about short term hold shelf expiration periods? Will this scheduler be able to be run frequently enough to expire those relatively quickly?

Here are the issues that seem relevant to look at: UIREQ-246 Closed , CIRC-292 Closed and CIRCSTORE-133 Closed

Comment by Marc Johnson [ 23/Apr/19 ]

Cate Boerema That will depend upon the approach.

If my suggestion of initial requirements is accepted, then it should be possible to schedule task execution up to every minute (in another issue, there is a conversation about making loan due date only accurate to a minute precision).

Where this gets more complicated is that if a task is executed every minute, we need to be careful about how much work that picks up (e.g. what question it asks about records to expire) and how long it takes to run.

We need to either design the tasks to avoid overlapping the work they pick up, handle the overlap, or make sure the execute fast enough to not execute for longer than the interval between executions.

These considerations will need to be taken into account for any story involving scheduled tasks.

Does that make sense?

Comment by Cate Boerema (Inactive) [ 24/Apr/19 ]

Yes - thanks!

Comment by Adam Dickmeiss [ 24/Apr/19 ]

Similar to _tenant and _tenantPermissions WRT path and permissions a new interface _timer could be provided:

  "provides": [
   {
      "id": "_timer",
      "version": "1.0",
      "interfaceType": "system",
      "handlers": [
        {
          "methods": [ "POST" ],
          "pathPattern": "/mypath",
          "modulePermissions" : [ "someperm" ],
          "wait" : "5",
          "unit" : "minute"
        }
      ]
    }

In this example once very 5 minutes.

Comment by Marc Johnson [ 24/Apr/19 ]

Adam Dickmeiss

Will this design allow the module to define multiple tasks (with different endpoints and waits), either with multiple provisions of the interface or multiple handlers? I think that’s the aspect I care most about, as I think it’s important that a module can schedule more than one task.

As to expressing this as an interface provision, I’ll defer the decision to you. If that fits better with Okapi’s model or it’s expedient to do so, then fine by me.

Personally, tasks are different to interfaces. Whilst trying to follow that thought process, I tried to turn it into a sentence: To schedule a task, a module provides the timer system interface for a specified endpoint and wait period. Does that reflect the model you describe?

It would mean specifying an interface provision with a variable endpoint(s), which is unlike any other interface I’m aware of. Do we have other circumstances where we do that?

(A less important aside, we’ve defined a period of time in other places in the system, where the duration is an integer and the interval/unit is plural, as i think it reads easier, for any duration other than one)

Anyway, just my thoughts, feel free to ignore.

Comment by Adam Dickmeiss [ 25/Apr/19 ]

Since handlers is an array you can define zero or more of _timer(s).

This is an interface that the module provides. Don't see to invent a new notion of task. This is Web service call like any and very similar to _tenant.

path.. Use the pathPattern should be honored by Okapi. So even for that one it's possible to use another one.. Although why?

Suppose the path / pathPattern is fixed, then the module can not distinguish between different handler calls. And that's a problem that I'd like to avoid, by simply giving the freedom to choose any path.

The BODY to to be sent is "empty" in version 1.0. I think a module should use POST, but if the timer call has no side effects, then method GET could be used instead and Okapi would send a GET instead.

For the time/period specs.. I'll look at the link and update my proposal example accordingly.

Comment by Adam Dickmeiss [ 25/Apr/19 ]

intervalId .. ? come on.. unit is better.. But that's too late to change now.

Comment by Marc Johnson [ 26/Apr/19 ]

Adam Dickmeiss Thanks for your thoughts.

intervalId .. ? come on.. unit is better.. But that's too late to change now.

Yeah, intervalId isn't great, and unit is likely better. That structure is only used in a few limited places, it might be better for Okapi to not follow that lead.

My main point about that was the plurality of the units.

Since handlers is an array you can define zero or more of _timer(s).

Ok. Modules would define a handler for each scheduled task they want triggering?

I think a module should use POST

I agree, I think tasks should likely be POST. Maybe we can agree on that as a convention?

Suppose the path / pathPattern is fixed, then the module can not distinguish between different handler calls. And that's a problem that I'd like to avoid, by simply giving the freedom to choose any path.

I wasn't suggesting fixing the path / pathPattern.

This is an interface that the module provides. Don't see to invent a new notion of task.

My comments about this being an interface come from my understanding of what a FOLIO interface commonly is. That this is being modelled as a special Okapi interface means the constraints are likely different (e.g. being able to be defined in RAML), which is why I defer to you about this modelling.

As usual, feel free to ignore / disregard as you see fit

Comment by Adam Dickmeiss [ 26/Apr/19 ]

Appreciate the feedback. Hopefully that means this can be implemented soon(ish)

Comment by Marc Johnson [ 26/Apr/19 ]

Adam Dickmeiss My understanding from Cate Boerema is that there is a hope this is implemented soon.

I imagine we wait until Jakub Skoczen is back for a story to get planned in. Unless others have any thoughts, I tried to add some folks as watchers.

Comment by Cate Boerema (Inactive) [ 26/Apr/19 ]

Thanks Adam Dickmeiss and Marc Johnson. Yes, there is great urgency to get this done, as it blocks many features including two or three that are needed for Chalmers go-live and must be done in Q2.

Comment by Craig McNally [ 26/Apr/19 ]

have the scheduling Okapi instance call the "handler" with a token that include permission reported though modulePermissions.

So which user would the call be made as? I ask with record metadata in mind - e.g. if a record is created or modified

Comment by Marc Johnson [ 26/Apr/19 ]

Craig McNally

So which user would the call be made as? I ask with record metadata in mind - e.g. if a record is created or modified

Ah, good question.

My focus has been on permissions, and I hope that is covered by granting the handler module permissions. Is that the case?

That would mean there wasn't a user as such.

Comment by Craig McNally [ 26/Apr/19 ]

I'm fairly positive there is a JIRA issue with a lengthy discussion of system users and situations similar to this, though I can't seem to find it.

**Update**: Found it: https://folio-org.atlassian.net/browse/FOLIO-1781

Comment by Craig McNally [ 26/Apr/19 ]

Ah, I think https://folio-org.atlassian.net/browse/RMB-353 is relevant here

Comment by Jakub Skoczen [ 29/Apr/19 ]

Craig McNally Marc Johnson The _timer would work the same as the tenant init call here – it would be made from Okapi without a user but within the context of the particular tenant and the call will include a token that includes whatever modulePermissions have been specified in the ModuleDescriptor (so the module can make further calls). RMB-353 Closed is only relevant to the bug that we had in RMB when dealing with change metadata for calls without an explicit user.

Comment by Jakub Skoczen [ 29/Apr/19 ]

Marc Johnson Kostyantyn Khodarev Adam Dickmeiss I recommend we change wait into duration and keep unit as the property to designate the unit for the timeout/duration. We should not use intervalId – the usage of this term in mod-circulation-storage to denote the enum for the unit seems very wrong to me, almost feels like a bug. The widely used convention in FOLIO is to use xxxId properties to denote foreign keys of type UUID and intervalId breaks that convention.

Comment by Marc Johnson [ 29/Apr/19 ]

Jakub Skoczen

it would be made from Okapi without a user but within the context of the particular tenant and the call will include a token that includes whatever modulePermissions have been specified in the ModuleDescriptor (so the module can make further calls). RMB-353 Closed is only relevant to the bug that we had in RMB when dealing with change metadata for calls without an explicit user.

Am I following this correctly, any endpoints invoked as a scheduled task will not have a user header, and therefore any downstream requests to other modules or database changes won't either?

And hence, RMB-353 Closed will need to be fixed and used in all modules that are involved in scheduled tasks?

Comment by Marc Johnson [ 29/Apr/19 ]

Adam Dickmeiss Jakub Skoczen

I recommend we change wait into duration

Duration in this sense would mean the duration between timer occurrences?

intervalId – the usage of this term in mod-circulation-storage to denote the enum for the unit seems very wrong to me, almost feels like a bug. The widely used convention in FOLIO is to use xxxId properties to denote foreign keys of type UUID and intervalId breaks that convention.

I agree that interval isn't good terminology, especially in a more general context. At the time I couldn't think of anything better (it was agreed in CIRCSTORE-7 Closed , and I think it is terminology used by the domain).

The `ID` postfix was me sitting on the fence over the topic of whether enumerated values were to become reference records (and so would need the postfix).

Comment by Craig McNally [ 29/Apr/19 ]

how about period or delay?

Comment by Craig McNally [ 29/Apr/19 ]

Am I following this correctly, any endpoints invoked as a scheduled task will not have a user header, and therefore any downstream requests to other modules or database changes won't either?

And hence, RMB-353 Closed will need to be fixed and used in all modules that are involved in scheduled tasks?

If I understand correctly, I think that's right Marc Johnson... From the aforementioned JIRA:

When a record is updated in anonymous request (no information about user id in request for example after an login attempt has failed) metadata is not populated. This causes error in the trigger if record has been previously saved with metadata.

So what's preventing the same problem from occurring here?

  • I have an record which has been created by a user, and has metadata associated with it.
  • A scheduled task runs and updates that record
  • the trigger fires and ...

That said, I think RMB-353 Closed will take care of this though, right? I haven't looked at the code but I'm guessing (hoping?) all that would be required is an upgrade to RMB 25.0.0+ in modules which might run into this.

Comment by Kostyantyn Khodarev [ 02/May/19 ]

Marc Johnson want to add to initial requirements, future solution should support:

1. Task scheduling
2. Clustering
3. Authentication (calls between modules)

What is cluster in Okapi world? Does attached picture reflect Okapi cluster structure?

Comment by Jakub Skoczen [ 07/May/19 ]

Kostyantyn Khodarev the initial implementation (see OKAPI-730 Closed ) shipped with 2.28 will already support this:

1. scheduling will be limited to a simple timer (you could extend this to a full scheduler by providing an external module or by implementing additional scheduling logic in your module)
2. clustering will be supported (and yes, your Okapi cluster diagram is correct). Okapi will ensure only one module instance receives the timer call.
3. authentication and authorization will be provided by Okapi exactly like it is provided for standard interfaces. When you add a handler for the "_timer" interface in your module, you will use modulePermissions to request any permissions you need to make calls to other modules and those permissions will be included in the authentication token your handler receives.

Generated at Thu Feb 08 23:17:07 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.