Table of Contents
...
The proposed solution should describe the target / long-term vision, while allowing phased implementation, as well as clearly define the scope of phases (at least the first one in Orchid).
References:
- https://issuesfolio-org.folioatlassian.orgnet/browse/UXPROD-3215
- Display a history of edits for the order record
- https://miro.com/app/board/uXjVObQrl6w=/?invite_link_id=742203032673
- Audit
- FOLIO Cross-Application Data Sync Solution
...
Therefore, the scope of this feature should implement an an API that allows getting information for the basic scenarios described above, as well as instructions for hosting providers on using the API to select the information needed for troubleshooting
UXPROD-3215 support
Data schema - need to consider https://issuesfolio-org.folioatlassian.orgnet/browse/UXPROD-3215 scope as well (see UI mock up).
The approach is to keep a snapshot of the entity for each event. This will then allow any analysis of changes, incl. identify differences, versioning, etc.
Search&filtering capabilities -
- Range of date&time
- Source (it means who initiated the event)
- Category (Cost, Item details etc... not mandatory but really valuable)
- The categories are based on the accordions found on the record in the UI. For order lines this would be Item details, Purchase order line, Vendor, Cost details, Fund distribution, Location etc.
- Keyword (similar to how inventory search works) - it's not a critical requirement here, as long as they can search by individual field
- Full-text search (FTS) is required; it's assumed that FTS in Postgres would be enough to address the expectations
Non-Functional Requirements
| Priority | Requirement | Comments, Expectations, Metrics | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Conceptual Integrity | The solution must be aligned on the technology stack, tools and approaches with the FOLIO platform. This solution is not considered cross-platform, but should address Event Log needs for all components of the Acquisition domain. However, the experience of implementing the Event Log in the Acquisition domain may be of interest in other domains. | ||||||||
| Extensibility | It should be possible in principle to add new types of events, new event sources or new log consumers to the Event Log. Note: No specific candidates are being discussed at this time (maybe inventory or agreements) It should be possible to track the source/event version and the Event Log version in each event. | ||||||||
| Performance | Generation and publication of events should not introduce noticeable delays in platform performance. It looks like there are no specific standards or metrics here. However, in some cases (like during order loads), performance issues sometimes appear. It worth testing! Need to test at least these three operations - open order, create order line, edit order line (there are links between OL and approve invoice, pay invoice) - check with TL; create a story(-ies); outcome is to get baseline metrics The architect and Product Owner have made the assumption that the delay added by posting an event can be no more than 100 ms to be considered acceptable. | ||||||||
| Reliability | The proposed solution should ensure reliable storage of all Event Log data for at least 20 years. (“Order numbers in FOLIO can live for multiple fiscal years”). The fact is that some orders can live for 10 or even 20 years; after closing an order is archived so closing can be used as a logical reason for archiving (remember about re-opening!) Immutability - stored events must not be deliberately or accidentally updated. Durability - stored events must not be deliberately or accidentally deleted (except when deleted at expiration of the lifespan). | ||||||||
| Scalability in terms of Data Volume | The solution must be efficient enough to process and store the estimated amount of data. On the example of orders:
Keeping in mind the required duration of data storage and the estimated amount of data per year, we can talk about 20 * 2.7 = 54M events for the entire period for one tenant. | ||||||||
| Availability | The solution must ensure availability at a level not lower than the SLA from the platform hosting provider. | ||||||||
| Security | The proposed solution should provide secure data storage, as well as support for security requirements for multi-tenancy. | ||||||||
| Traceability | There should be a clear way to link logged events to some common entity (e.g., order, or order line, etc.). |
...
- Every new Event Source should be able to connect Kafka and post events to specified Kafka topics.
- How to achieve a transactional approach? Outbox pattern cannot be implemented on Vert.x - !OK - there should be a way to find a workaround / solution here.
- Follow https://wikifolio-org.folioatlassian.orgnet/wiki/display/DD/Apache+Kafka+Messaging+System and https://wikifolio-org.folioatlassian.orgnet/wiki/display/~olamshin/Kafka+Practices+Proposal
- Kafka usage practices - topic per tenant and topic per event type
- Orchid phase - define some events that are to be logged in Orchid
- Event Log Storage
- OK - just a single flat table
- RDBMS
- Need to design a schema
- The intention is to avoid the approach with JSONB but how to design an extendable relational schema?
- RA: Maybe it could be reasonable to consider module -> entity -> event type hierarchy? Like, all events about mod-orders -> order go to one table, mod-orders -> something else go to another table?
- Actually, column-family or document oriented (e.g.,Mongo) NoSQL databases could fit well here..
- OK - the schema mod-audit currently has seems to be pretty extensible - it has a range of standard fields plus details in json body
- Event Log
- API for CRUD - just basic API for Orchid phase
- Event Log Showcase
- some correlation id(s) might be useful
- RA: What are the use cases? How will support/developers use the Event Log? Will full-text search be required?
- OK - Support / developers can execute SQL scripts or execute some API calls for retrieving data
...