[FOLIO-1953] SPIKE: propose an approach for scheduling tasks in Okapi Created: 09/Apr/19 Updated: 09/Jul/20 Resolved: 06/May/19 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | P2 |
| Reporter: | Jakub Skoczen | Assignee: | Adam Dickmeiss |
| Resolution: | Done | Votes: | 0 |
| Labels: | platform-backlog | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Issue links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | CP: sprint 62 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Story Points: | 5 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Development Team: | Core: Platform | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
We currently only have the scheduling capability in the storage modules as provided by RMB. There is a need to have a scheduling capability also in the BL modules (e.g mod-circulation, please link the appropriate stories). Since BL modules have no persistence and may run in multiple instance we need a solution that will ensure the scheduling is coordinated (the scheduled event happens only once). Also, any scheduled activities needed to be granted permissions to interact with other module endpoints. DRAFT PROPOSAL One option is to do scheduling in Okapi;
|
| Comments |
| Comment by Marc Johnson [ 10/Apr/19 ] |
|
Jakub Skoczen Kostyantyn Khodarev Cate Boerema Below is a quick write up of what I think are the initial requirements 1 and possible future extensions. Do these seem reasonable and fit with what we talked about? Have I missed anything? Initial Requirements
These closely mimic the current scheduling implementation in RAML Module Builder. The main differences are that it is tenant aware (rather than running as soon as the module is deployed) and limited to running on only a single instance. Possible Future Extensions
1) I've tried to defer any description of how this could be accomplished to a later comment |
| Comment by Kostyantyn Khodarev [ 12/Apr/19 ] |
|
Marc Johnson |
| Comment by Cate Boerema (Inactive) [ 17/Apr/19 ] |
|
Marc Johnson maybe you can take a look at the requirements outlined in the issues I've linked up just to double check it's all covered. For example, do we want to have the Okapi scheduler handle the closed library due date management stuff? This is desired for hold shelf expiration period calculation. What about short term hold shelf expiration periods? Will this scheduler be able to be run frequently enough to expire those relatively quickly? Here are the issues that seem relevant to look at:
|
| Comment by Marc Johnson [ 23/Apr/19 ] |
|
Cate Boerema That will depend upon the approach. If my suggestion of initial requirements is accepted, then it should be possible to schedule task execution up to every minute (in another issue, there is a conversation about making loan due date only accurate to a minute precision). Where this gets more complicated is that if a task is executed every minute, we need to be careful about how much work that picks up (e.g. what question it asks about records to expire) and how long it takes to run. We need to either design the tasks to avoid overlapping the work they pick up, handle the overlap, or make sure the execute fast enough to not execute for longer than the interval between executions. These considerations will need to be taken into account for any story involving scheduled tasks. Does that make sense? |
| Comment by Cate Boerema (Inactive) [ 24/Apr/19 ] |
|
Yes - thanks! |
| Comment by Adam Dickmeiss [ 24/Apr/19 ] |
|
Similar to _tenant and _tenantPermissions WRT path and permissions a new interface _timer could be provided: "provides": [ { "id": "_timer", "version": "1.0", "interfaceType": "system", "handlers": [ { "methods": [ "POST" ], "pathPattern": "/mypath", "modulePermissions" : [ "someperm" ], "wait" : "5", "unit" : "minute" } ] } In this example once very 5 minutes. |
| Comment by Marc Johnson [ 24/Apr/19 ] |
|
Will this design allow the module to define multiple tasks (with different endpoints and waits), either with multiple provisions of the interface or multiple handlers? I think that’s the aspect I care most about, as I think it’s important that a module can schedule more than one task. As to expressing this as an interface provision, I’ll defer the decision to you. If that fits better with Okapi’s model or it’s expedient to do so, then fine by me. Personally, tasks are different to interfaces. Whilst trying to follow that thought process, I tried to turn it into a sentence: To schedule a task, a module provides the timer system interface for a specified endpoint and wait period. Does that reflect the model you describe? It would mean specifying an interface provision with a variable endpoint(s), which is unlike any other interface I’m aware of. Do we have other circumstances where we do that? (A less important aside, we’ve defined a period of time in other places in the system, where the duration is an integer and the interval/unit is plural, as i think it reads easier, for any duration other than one) Anyway, just my thoughts, feel free to ignore. |
| Comment by Adam Dickmeiss [ 25/Apr/19 ] |
|
Since handlers is an array you can define zero or more of _timer(s). This is an interface that the module provides. Don't see to invent a new notion of task. This is Web service call like any and very similar to _tenant. path.. Use the pathPattern should be honored by Okapi. So even for that one it's possible to use another one.. Although why? Suppose the path / pathPattern is fixed, then the module can not distinguish between different handler calls. And that's a problem that I'd like to avoid, by simply giving the freedom to choose any path. The BODY to to be sent is "empty" in version 1.0. I think a module should use POST, but if the timer call has no side effects, then method GET could be used instead and Okapi would send a GET instead. For the time/period specs.. I'll look at the link and update my proposal example accordingly. |
| Comment by Adam Dickmeiss [ 25/Apr/19 ] |
|
intervalId .. ? come on.. unit is better.. But that's too late to change now. |
| Comment by Marc Johnson [ 26/Apr/19 ] |
|
Adam Dickmeiss Thanks for your thoughts.
Yeah, intervalId isn't great, and unit is likely better. That structure is only used in a few limited places, it might be better for Okapi to not follow that lead. My main point about that was the plurality of the units.
Ok. Modules would define a handler for each scheduled task they want triggering?
I agree, I think tasks should likely be POST. Maybe we can agree on that as a convention?
I wasn't suggesting fixing the path / pathPattern. This is an interface that the module provides. Don't see to invent a new notion of task. My comments about this being an interface come from my understanding of what a FOLIO interface commonly is. That this is being modelled as a special Okapi interface means the constraints are likely different (e.g. being able to be defined in RAML), which is why I defer to you about this modelling. As usual, feel free to ignore / disregard as you see fit |
| Comment by Adam Dickmeiss [ 26/Apr/19 ] |
|
Appreciate the feedback. Hopefully that means this can be implemented soon(ish) |
| Comment by Marc Johnson [ 26/Apr/19 ] |
|
Adam Dickmeiss My understanding from Cate Boerema is that there is a hope this is implemented soon. I imagine we wait until Jakub Skoczen is back for a story to get planned in. Unless others have any thoughts, I tried to add some folks as watchers. |
| Comment by Cate Boerema (Inactive) [ 26/Apr/19 ] |
|
Thanks Adam Dickmeiss and Marc Johnson. Yes, there is great urgency to get this done, as it blocks many features including two or three that are needed for Chalmers go-live and must be done in Q2. |
| Comment by Craig McNally [ 26/Apr/19 ] |
So which user would the call be made as? I ask with record metadata in mind - e.g. if a record is created or modified |
| Comment by Marc Johnson [ 26/Apr/19 ] |
Ah, good question. My focus has been on permissions, and I hope that is covered by granting the handler module permissions. Is that the case? That would mean there wasn't a user as such. |
| Comment by Craig McNally [ 26/Apr/19 ] |
|
I'm fairly positive there is a JIRA issue with a lengthy discussion of system users and situations similar to this, though I can't seem to find it. **Update**: Found it: https://folio-org.atlassian.net/browse/FOLIO-1781 |
| Comment by Craig McNally [ 26/Apr/19 ] |
|
Ah, I think https://folio-org.atlassian.net/browse/RMB-353 is relevant here |
| Comment by Jakub Skoczen [ 29/Apr/19 ] |
|
Craig McNally Marc Johnson The _timer would work the same as the tenant init call here – it would be made from Okapi without a user but within the context of the particular tenant and the call will include a token that includes whatever modulePermissions have been specified in the ModuleDescriptor (so the module can make further calls).
|
| Comment by Jakub Skoczen [ 29/Apr/19 ] |
|
Marc Johnson Kostyantyn Khodarev Adam Dickmeiss I recommend we change wait into duration and keep unit as the property to designate the unit for the timeout/duration. We should not use intervalId – the usage of this term in mod-circulation-storage to denote the enum for the unit seems very wrong to me, almost feels like a bug. The widely used convention in FOLIO is to use xxxId properties to denote foreign keys of type UUID and intervalId breaks that convention. |
| Comment by Marc Johnson [ 29/Apr/19 ] |
Am I following this correctly, any endpoints invoked as a scheduled task will not have a user header, and therefore any downstream requests to other modules or database changes won't either? And hence,
|
| Comment by Marc Johnson [ 29/Apr/19 ] |
Duration in this sense would mean the duration between timer occurrences?
I agree that interval isn't good terminology, especially in a more general context. At the time I couldn't think of anything better (it was agreed in
The `ID` postfix was me sitting on the fence over the topic of whether enumerated values were to become reference records (and so would need the postfix). |
| Comment by Craig McNally [ 29/Apr/19 ] |
|
how about period or delay? |
| Comment by Craig McNally [ 29/Apr/19 ] |
If I understand correctly, I think that's right Marc Johnson... From the aforementioned JIRA:
So what's preventing the same problem from occurring here?
That said, I think
|
| Comment by Kostyantyn Khodarev [ 02/May/19 ] |
|
Marc Johnson want to add to initial requirements, future solution should support: 1. Task scheduling What is cluster in Okapi world? Does attached picture reflect Okapi cluster structure? |
| Comment by Jakub Skoczen [ 07/May/19 ] |
|
Kostyantyn Khodarev the initial implementation (see
1. scheduling will be limited to a simple timer (you could extend this to a full scheduler by providing an external module or by implementing additional scheduling logic in your module) |