Page Comparison

Reference JIRA issue: 

...

1. Introduce two configuration parameters (names are to be altered):
   1. di.flow.max_simultaneous_records – this parameter defines how many records can be processed by the system simultaneously. Default value is proposed to be set to 10*(kafka batch size)
   2. di.flow.records_threshold – this parameter defines how many records from previous batch must be processed before throwing new records to pipeline. Default value is proposed to be 5*(kafka batch size)
   3. since Kafka works with batches, those parameters should be set to be evenly divided by Kafka batch size. a validation could also be added for this rule
2. Implement flow control mechanism based on parameters above:
   1. mod-srm counts total number of messages it passed to DI pipeline. This number of records should not exceed di.flow.max_simultaneous_records at any time.
   2. when <<DI_COMPLETE>> or <<DI_ERROR>> is received, record counter is decreased
   3. when counter reaches di.flow.records_threshold or less, more records are pushed to pipeline (from priority queue). Count of records pushed should respect di.flow.max_simultaneous_records
   4. those counters are per-tenant (since topics are also per-tenant)

Manual operations are pushed to the pipeline directly. They do not respect the parameters above.

...

If expected latency is too high there's another solution for implementation to be considered:

Alternate Solution: Prioritized Kafka topics

Instead of creating a single topic for a command, system creates a group of similar topics each having defined priority.

...

This solution will allow to provide minimum possible latency for prioritized commands, but is also much more complex, time-consuming, and risky for all of data import, both to implement and maintain.

List of implementation stories for the Proposed Solution: