When enabled, mod-data-import will split large jobs into smaller “chunks,” adding each chunk into a queue and dynamically ordering them to ensure a fair distribution of jobs are run at the same time, considering metrics such as job size, tenant usage (for multi-tenant environments), and how long a job has been waiting.  The algorithm for selecting which chunk will be run next is highly configurable, allowing experimentation and "dialing in" of parameters for specific tenants and deployments.  Details on how this algorithm works, as well as how to customize it, may be found below:

Approach

When a worker becomes available, it will calculate and assign every waiting chunk a single numerical “score.” This score will combine many factors according to the parameters, and is designed to represent a holistic view of the chunk, including the job size, waiting time, and more.  Higher scores are better.

Factors considered

Job size

This metric considers the total size of the job, in records.  This allows control over prioritizing smaller jobs over larger ones; for instance, if a large job has been running for many hours (or would otherwise have priority), it may be desired for a job with only a handful of records to be able to "skip the line" and get processed next.

This is computed on a logarithmic scale, meaning that every doubling of the value only increases the score by one.  For example, if the score ranges from 5 (smallest) to 0 (largest), and the reference value is 32, a job with size 1 would get score 5, size 2 gets score 4, size 4 gets score 3, size 8 gets score 2, size 16 gets score 1, and size 32 gets score 0.  For more details on the calculation, see “Implementation notes” below.

Age

This metric considers how long this chunk has been waiting, based on when the user selected "Run" in the interface.  This control is useful since it allows jobs that have been waiting longer to be prioritized.

Additionally, this metric considers an "extreme value," allowing a sort of failsafe to prevent other factors from de-prioritizing very old jobs.  For example, if we set the normal newest — oldest scores to range from 0 - 100, and job size 0 - 1000, job size could very well outweigh the age, and keep bumping an old but large job to the back.  An example usage of this failsafe could be to ensure no job waits more than 24 hours; with a threshold of 24 hours, the value could be set to something like 10000 — more than enough to jump any jobs waiting longer than a day to the top of the queue.

Tenant usage

This considers how many workers are currently being used by a tenant; jobs from a tenant which is currently saturating the queue would be deprioritized, pushing for an even distribution amongst all tenants.  Note that this will have no effect if in a single-tenant environment (or only one tenant is currently importing data), since all jobs' scores would be affected equally.

This is done on a "linear" scale, making it percentage-based; a tenant using 25% of the workers will have a score 25% of the way between the min and max values.

Part number

Lastly, this metric is useful to ensure chunks run in order (otherwise chunks from the same job could run in a non-deterministic order).  It is recommended to keep this range very low (e.g. just 0 to -1), since otherwise every chunk in the same job should have the same score.

This is logarithmic for implementation-specific reasons, but since this is intended to be used on a very small range, this does not really matter.

Environment variables

Implementation notes

Want to disregard a metric?

To disregard/ignore a metric, simply set its parameters to zero.  For example, to emulate first-come-first-serve, set all but the age-related parameters to zero (and, since there are no other metrics to compete with, simple values of newest=0, oldest=1, extreme_threshold=9999, extreme_value=2, or any other combination where newest < oldest < extreme_value, will work).

Customization tips

To make it easier to visualize this algorithm, see this playground https://codesandbox.io/s/di-scoring-playground-x4kqrw?file=/src/ChunkAdd.tsx.  This makes it easy to simulate tenant usage and jobs of different ages, sizes, and tenants.

Logarithmic factors can often be difficult to calibrate, since they can potentially be infinite, making it difficult to choose a good reference value.  When attempting to determine this, don't look for the "largest possible" value, just use something that is a "typical" or "expected" large value; any values that exceed the reference value will still have their score calculated; it will just exceed the range.  To aid in this calibration, we developed https://codesandbox.io/s/di-unbounded-logarithmic-playground-yf4yyz, which shows score ranges for a given lower and upper value:

Logging

Whenever a worker looks for a job, we log all calculated scores, to make it easier to calibrate in production.  Look for Current worker tenant usage (lists how many workers are in use by each) and Calculated scores (lists the score for each job, listed as tenant/queue chunk ID/score).

Calculations

Warning to the reader: this section gets extremely technical.

Unbounded logarithmic

These represent potentially unbounded values. As such, they do not have a limit; instead, we will define a reference/expected value for the upper bound (that would represent a typical upper bound). However, since we are using a logarithm, the effect of values past the expected bounds is minimal.

To interactively see how this calculation works, see https://codesandbox.io/s/di-unbounded-logarithmic-playground-yf4yyz

For example: the size of a chunk. If we expect chunks to typically have a size from 1 to 32, we could define scores 0 to 5. With this, we would get the following scores:

Why is this a good approach?

This may seem way overcomplicated, however, the above math can be written as one or two lines of code. Some other questions I asked myself while deciding on this were:

• Why not just use a linear approach? We want to group things into classes of different sizes; for example, jobs could be “small” (e.g. 0-100), “medium” (100-1000), and “large” (1000-10000+). Two “medium” jobs could be 900 records apart, 9x the entire range of “small”, diluting the differences within that class. By using a logarithmic scale, we can retain granularity within each group.

• Why not just have classes/ranges like you mentioned above? We could do this, however, the next immediate question is “how many ranges should there be” and “what range should each one cover.” We could attempt to answer these, but we cannot anticipate the needs of the future. For the latter, we could make configuration variables, but just sorting into five groups would introduce fifteen variables (lower, upper, score); which seems overcomplicated.

Bounded logarithmic

This acts like logarithmic, however, upon reaching the reference value/expected upper bound (EXTREME_THRESHOLD), the EXTREME_VALUE will be used instead. This is useful for something like Age, where we want to gradually increase the score over time, however, after a certain amount of time, we want that job to finish ASAP (effectively be bumped to the top). This represents the maximum we want this parameter to ever get to.

Linear

This works for percentages; if the value corresponds to 50%, then the score will be halfway between the upper and lower scores.

Adding additional metrics

The code backing this is highly extensible; to add your own factors, create a new class extending QueueItemRanker with a single method score(DataImportQueueItem queueItem, Map<String, Long> tenantUsage).  Then, to include it in the calculations, simply add your custom ranker to the score  method of QueueItemHolisticRanker.