[FOLIO-889] Develop a strategy to implement scheduled (cron-like) events Created: 09/Oct/17  Updated: 02/May/19  Resolved: 24/May/18

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Story Priority: P3
Reporter: Kurt Nordstrom Assignee: Kurt Nordstrom
Resolution: Done Votes: 0
Labels: core, sprint24, sprint25
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Relates
relates to FOLIO-1953 SPIKE: propose an approach for schedu... Closed
relates to UIU-200 Proxy: Set Relationship Status (Spons... Closed
relates to UIU-201 Proxy: Set Relationship Status (Proxy... Closed
relates to UIU-235 User Status Should be Inactive When E... Closed
Sprint:

 Description   

We need a way to have certain events happen at regular and scheduled intervals. Examples include:

  • Scheduling batch processing
  • Checking for expired records and pruning appropriately
  • Clean up processes
  • Periodic audits

Any solution will need to answer the following questions:

  • How is security/ownership assigned to a task?
  • How exactly is the task specified? Do we require a DSL to specify task actions?
  • How does this tie-in with projected implementation of workflows (if at all)?


 Comments   
Comment by Cate Boerema (Inactive) [ 10/Oct/17 ]

Sending overdue notices is another feature that may benefit from this. Calculating accumulated overdue fines also maybe...

Should we label this as sprint24?

Comment by Kurt Nordstrom [ 18/Oct/17 ]

There are many "events" within FOLIO that need to happen at regularly scheduled intervals. Examples of events such as these would be checking for overdue loans, compounding fines, cleaning up unneeded data and the like.

The actions of a FOLIO cron job are relatively simple. A script will be designated to run at a particular interval. When it runs, it will make a number of calls to one or more other modules within FOLIO and make internal decisions based on the responses it gets.

A reasonable "mod-cron" module needs to deal with the following considerations:

  • Any mod-cron job will need to be associated with an authtoken of some kind. There needs to be a way to track who "owns" a given job, and what set of permissions it runs with. Ideally, no job should run with more permissions than it actually needs to accomplish what it needs to do.
  • There needs to be a way to allow mod-cron job designers to assign reasonably complex logic to a job.
  • There should ideally be a way to pause and resume jobs.
  • There needs to be tracking of which jobs have run and when.
  • Ideally, there should be a central log file to be able to check the output of past jobs.

It is worth noting that it would be possible to obtain a cron-like functionality by building external scripts that authenticate into FOLIO and execute commands as needed. Such a script could be scheduled with something like Unix cron and potentially live anywhere that could talk to the FOLIO modules.

There are a few problems with the "external cron jobs", though:

  • No way to centrally manage scheduling/starting/stopping of jobs.
  • No way to modify a job from within FOLIO.
  • No central record of executions or job outputs.

I envision a mod-cron module to operate around the following data model:

{
	"id" : "",
	"ownerId" : "",
	"name" : "",
	"description" : "",
	"active" : true,
	"schedule" : {
		"dayOfWeek" : "",
		"month" : "",
		"dayOfMonth" : "",
		"hour" : "",
		"minute"
	},
	"cronscript" : ""
	
}

The fields are defined as follows:

  • id : This is the unique UUID that identifies the particular job.
  • ownerId: The owner of the job, for which the job will take its permissions.
  • name: A human-readable short name for the job (e.g. "update fines").
  • description: A longer, human-readable explanation for the job.
  • active: A boolean which indicates whether or not a given job is currently running.
  • schedule: A structure which determines when the job should actually run.
  • cronscript: A textual blob containing the actual script that gets executed. The allowable scripting is a topic of importance.
Comment by Heikki Levanto [ 24/Oct/17 ]

I seem to recall that we had some kind of harvester project at ID long time ago, with a similar kind of cron-like specification. I don't think we were really happy with that. Most users needed to specify that something should happen daily or weekly, but not at what time of the night things should happen, etc.

Also, if we ever make that workflow/automation thing, it seems like the ideal way to control what should be happening.

Comment by Kurt Nordstrom [ 24/Oct/17 ]

My initial thoughts on the "cronscript" field was that this would be a blob that would contain the text of a script that would actually execute as per the schedule and carry out the instructions of the task. Something like Groovy would work as a sandboxed DSM, for which we'd provide execution constraints, as well as a limited number of FOLIO-affecting calls that the script could actually make.

I think this offers some powerful advantages, namely that expressive and useful automations can be created and managed in userspace. It gives a lot of flexibility and utility to individual systems.

There are downsides to this approach, though:

  • Potential risks to permitting arbitrary scripting, in spite of sandboxing
  • Need to maintain scripts when system updates

An alternative approach might be to define the useful behaviors in the modules themselves. For example, mod-users-bl might have a method to send overdue notices defined, since this is a logical place that is very close to the involved modules. What we'd need to do then would be a way to expose these "tasks" to the API, and mod-cron's job would be to call these task endpoints in accordance with the defined schedules.

Comment by Cate Boerema (Inactive) [ 17/May/18 ]

Kurt Nordstrom and Jakub Skoczen is this ticket still needed? If so, let's finish it up. If not, let's close it.

Comment by Kurt Nordstrom [ 23/May/18 ]

Cate Boerema I would feel that this would be subsumed by the workflow work being done. I have no objection to closing it.

Generated at Thu Feb 08 23:09:04 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.