...
- some more recent modules (e.g. mod-courses, mod-data-export-spring) have both business logic and persistent storage within the same module
- some modules (e.g. mod-search, mod-inventory, mod-data-export-spring) use Kafka (rather than HTTP) for communication
- some modules (e.g. mod-search) use Elastic Search for persistent storage (though not the book of record)
- some processes e.g. data import) are working on using potentially stale inconsistent or out of date data in order to improve performance
...
Use derived data to make decisions
This option has the most potential design variations. It might involve in-memory caching or persistent storage of the copied data and it may involve using messaging infrastructure or periodic requests.Irrespective of the design variations, in involves using data that would be considered eventually consistent, meaning that at the time of use it might be out of date with respect to the source of that data (and at some point after it will become consistent again).
In order for this option to be acceptable, a variety of stakeholders within the community would need to accept some tolerance for decisions being made with stale inconsistent information. When I've talked with folks about this previously, they have been uncomfortable with doing this (see above).
This option also has the most potential design variations. It might involve in-memory caching or persistent storage of the copied data and it may involve using messaging infrastructure or periodic requests.
Characteristics
- Increases the potential for stale data inconsistent data (that is out of date) to be used for decisions
- State changes still require a downstream request (and the requisite proxying overhead)
- Is contrary to constraints that may still be present in FOLIO
- Requires less / no (depending upon variations chosen) downstream requests for fetching data during check out process
- Introduces complexity of either caching or processing messages and persistent storage into mod-circulation
- May introduces a dependency for mod-circulation on a database
- May introduces a dependency for mod-circulation on messages produced by other modules
- May introduce the need for a system user for mod-circulation (to populate / update the cache)
...
| Periodic HTTP requests | Messages consumed from Kafka | |
|---|---|---|
| Freshness | Dependent upon frequency of periodic refresh. Likely to be lead to data being stale for inconsistent for longer than with messaging | Dependent upon message processing latency |
| Access requirements | Needs a system user or module permissions granting to a timer endpoint | Needs access to Kafka topics for every record type (assuming record snapshot based messages as used with mod-search) |
| Initial population / manual state refresh | Requires requests to fetch all records for for all cached records types | Either requires reprocessing of persistent topic (not currently allowed by FOLIO standards) or custom process (similar to mod-search re-index process) |
| Load on other modules during synchronisation | Could be significant. Dependent upon number of record types and quantity of records | Potentially none with persistent topics (not currently allowed by FOLIO standards) |
...
- adoption of techniques (e.g. synchronising copied data, messaging, caching) and technologies (e.g. Kafka) unfamiliar to most developers in FOLIO
- agreement from many stakeholders (e.g. SME's, TC) that it is acceptable to use potentially stale data for inconsistent or out of date information for making decisions
Appendices
...
| Phrase | Definition |
|---|---|
| Downstream request | A request made by a module (via Okapi) in order to fulfil the original incoming request e.g. mod-circulation makes a request to mod-users to fetch patron information |
| Response time | The time taken from the client making the request to receiving a response |
| Derived data | "A dataset that is created from some other data through a repeatable process. Usually use to speed up a particular kind of read access to the data. Indexes, caches, and materialized views are examples of derived data" ([1], pg. 554) |
| Eventual consistency | Copies or derivations of data can be outdated with respect to the source. It is intended that this inconsistency is temporary, however this deliberately vague and there is no limit on on how far behind ([1] pg. 162) a copy might get or how long it will take for it to become consistent again. The term originates from database replication and is often used in event sourcing or event driven architecture . |
Requests made during a typical check out
...