[FOLIO-969] Architecture needs for a "tap" of transactions Created: 06/Dec/17  Updated: 12/Nov/18  Resolved: 27/Aug/18

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P3
Reporter: Peter Murray Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: devmtg-201801
Remaining Estimate: Not Specified
Time Spent: 1 hour
Original estimate: Not Specified

Issue links:
Relates
relates to FOLIO-925 Current reporting systems research Closed
Sprint:

 Description   

It is looking likely that the architecture will need some way to stream transactions to receivers that will transform and do other processing on transaction data. This issue is a discussion of those needs.



 Comments   
Comment by Peter Murray [ 06/Dec/17 ]

This need has come up in several contexts:

  • Reporting SIG: copies of transactions that are stored in a data refinery (for raw transaction payloads) and/or transformed into tables in a data warehouse (for reporting needs)
  • Discovery Layer: a stream of transactions that can be fed into the indexer of a discovery layer
  • External KBs: something Sebastian Hammer mentioned

The crux of the issue is how do we efficiently and effectively get data out of the transaction-oriented Okapi layer and into something that can independently transform/store the transaction data. Given that there is already more than one need for this (bullet points above), we need a general way of handling this. Which leads to these questions:

  • Is an interface to Apache Kafka and/or AWS Kinesis appropriate? (Do these two project and similar software have a common interface?)
  • Does the interface take the form of a publish/subscribe mechanism or a message bus/queue mechanism or something else?
  • Is the interface native to the Okapi Gateway or is the interface handled in an Okapi module at the end of the processing pipeline (similar to how Auth is at the start of the processing pipeline)? Or some other extension to Okapi?

Lastly, what is the best way to prototype this requirement?

Comment by Jakub Skoczen [ 08/Dec/17 ]

Note: even though we do not support a native messaging protocol, a publish/subscribe mechanism can be simulated by Okapi with so-called multiple interfaces (which would model subscribers) and regular "requires" dependency (to model publishers). What we are missing is a more flexible way to multiplex the requests and collect errors. This could be implemented in Okapi directly or as a module.

Those use cases require more in-depth analysis – e.g we can perform a lot of logging of transactions (requests/responses) transparently in Okapi and thus alleviate the module from implicit logging. But this may not be powerful enough – Okapi only sees data specific to the particular request and does not see/understand the larger context of the request and the effective change to the state of the system object/entity. E.g logging a loan action may require context for what loan policy is attached to it, if the are any outstanding request or fees/fines.

Comment by Peter Murray [ 27/Aug/18 ]

Work being covered elsewhere.

Generated at Thu Feb 08 23:09:52 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.