...
For ECS environments: Shared entities' version history should be tracked only in the central tenant.
All changes in the system related to inventory entities (instances, items, holdings, bibs) generate Domain events.
Domain events related to update action have old and new versions of the entity
Baseline Architecture
In existing architecture, mod-inventory-storage is responsible for persisting such entities as instances, holdings, and items. mod-entities-links is responsible for authorities. Both modules produce domain events on create/update/delete actions from different sources.
...
Audit Consumers Sequence Diagram
| Drawio | ||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| Expand | ||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||
Audit Consumers with Outbox Sequence Diagram
The implementation can follow a Transactional outbox pattern. The approach allows enhanced guarantee for persisting the audit event but the trade-off is that this approach will negatively affect the performance of flows related to domain events. |
...
Option | Description | Pros & Cons | |
|---|---|---|---|
| 1 | RDBMS | The audit database should persist a snapshot diff of the entity. The queries are made mostly by the entity's unique identifier. Thus partitioning by UUID and subpartitioning by date ranges can be applied to audit tables | Pros:
Cons:
|
| 2 | Object Storage | AWS S3-like storage can be used to persist snapshots diffs because audit events can be stored as plain-text (JSON) documents | Pros:
Cons:
|
2. Version history display: This should be done on demand comparing each consecutive snapshot diff of the entity to the previous
...
The Kafka default delivery semantics is “AT_LEAST_ONCE”. Ensure that domain events have their unique identifiers to be able to handle consuming messages in an idempotent manner
Add new consumers in
mod-auditto inventory domain events for instances, items, and holdings.Add new consumers in
mod-auditto authority domain events for authorities.Add new consumers in
mod-auditto source record domain events for marc-bib records.(see: Source Record Domain Eventing)mod-auditshould support the following configurations on the tenant level:Retention period in years (with default value - 0 for indefinite retention)
Feature flag to enable audit capability. In case when the audit is disabled no consumers and logs should be persisted.
Anonymizing flag that indicates whether the records in the database should be anonymized before persistence to the database (To be confirmed).
mod-auditshould have the following scheduled jobs:Daily: to remove records that exceed the retention period
Monthly: to create subpartitions for audit tables
Persist audit events in an event storage. A table in DB per entity type with partitioning by UUID (hash) and subpartitioning by date range.
Create REST API
4. to provide information on a list of changes related to a particular entity
5. to provide detailed information on the particular change - this API should use the Object diff library to return a verbose description of the difference between current and previous snapshots of changes related to the entity
ERD
With data size implications, creating separate tables per each entity type is required. The default table structure is listed below:
Column | Type | required | unique | Description | |
|---|---|---|---|---|---|
| 1 | EventID | UUID | y | y | unique event identifier |
| 2 | EventDate | timestamp | y | n | date when the event appeared in the event log |
| 3 | Origin | varchar | y | n | Origin of the event: data-import, batch-update, user, etc. |
| 4 | Action | varchar | y | n | what action was performed |
| 5 | ActionDate | timestamp | y | n | when action was performed |
| 6 | EntityID | UUID | y | n | entity identifier |
| 7 | UserId | UUID | y | n | user who did the action, fixed UUID for anonymized user |
| 8 | Snapshot Diff | jsonb | y | n | Difference between “old” and “new” body of the entity |
WBS
| Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...