MARC Authority Deletion

Executive Summary

LoC requires an authority deletion feature. The implementation should not have a negative impact on active records performance.

Requirements

Functional Requirements

• Allow the library to export deleted MARC authority records via API. 
• Support providing deleted authority record UUIDs via API 
• Support authority record purge configuration to be set to Never per tenant

Non-functional Requirements

Performance:

• 10k weekly
• 50k monthly 
• Very rarely more than 100k (KG will confirm with LOC)

Data consistency:

• There should be no invalid links to deleted authorities in instances
• Authority statistics should be aligned after the authority record is deleted.

Data retention period:

• Support authority record purge configuration to be set to Never per tenant

Assumptions

• Authority record restoration is not planned in future releases

Baseline Architecture

The current implementation of authority record deletion consists of the following steps:

1. "Soft-delete" Flag is updated to true
2. Domain event regarding deleted records is sent through Kafka
3. After events are sent and processed, records are physically deleted from the authority table

Target Architecture

The target solution for authority record deletion requires the support of export of deleted records, hence they should be persisted in the database and physically deleted only after the retention period elapses. This requires support of "soft" delete and "hard" delete operations. Soft-delete should be implemented as moving deleted records into the archive table in order to provide information on deleted records. The archive table also will provide minimal impact on the performance of active records.

The approach for the archive table:

• Same tablespace as the active table. Table name pattern <original_table_name>_archive 
• Trigger on delete should be implemented to move the record to the archive table.
• Domain events should be distinct for soft- and hard-delete.
• Update lastUpdatedDate, updatedByUserId in metadata.
• Tenant-level retention policy should be provided through configuration parameters.

Process:

• Soft-delete:
  1. update the soft-delete flag in the original table, send events
  2. after events are processed, move the deleted record to the archive table
• Hard-delete:
  • if the retention policy is set then, on schedule the archive table should be cleaned by lastUpdatedDate field
  • if the retention policy for the tenant is set to never then no clean up is required

Open Questions