MARC Authority Deletion

Executive Summary

LoC requires an authority deletion feature. The implementation should not have a negative impact on active records performance.

Requirements

Functional Requirements

• Allow the library to export deleted MARC authority records via API. 
• Support providing deleted authority record UUIDs via API 
• Support authority record purge configuration to be set to Never per tenant

Non-functional Requirements

Performance:

• 10k weekly
• 50k monthly 
• Very rarely more than 100k (KG will confirm with LOC)

Data consistency:

• There should be no invalid links to deleted authorities in instances
• Authority statistics should be aligned after the authority record is deleted.

Data retention period:

• Support authority record purge configuration to be set to Never per tenant

Assumptions

• Authority record restoration is not planned in future releases

Baseline Architecture

The current implementation of authority record deletion consists of the following steps:

1. "Soft-delete" Flag is updated to true
2. Domain event regarding deleted records is sent through Kafka
3. After events are sent and processed, records are physically deleted from the authority table

Target Architecture

The target solution for authority record deletion requires the support of export of deleted records, hence they should be persisted in the database and physically deleted only after the retention period elapses. This requires support of "soft" delete and "hard" delete operations. Soft-delete should be implemented as moving deleted records into the archive table in order to provide information on deleted records. The archive table also will provide minimal impact on the performance of active records.

The approach for the archive table:

• Same tablespace as the active table. Table name pattern <original_table_name>_archive 
• Trigger on delete should be implemented to move the record to the archive table.
• Domain events should be distinct for soft- and hard-delete.
• Update lastUpdatedDate, updatedByUserId in metadata.
• Tenant-level retention policy should be provided through configuration parameters. Tenant-level configuration of the retention period for the deleted records should be kept in mod-settings.
• The projected transaction volume might cause a big amount of records. To address that the API for listing of deleted records should support export in plain text format and json. Besides that it should provide filtering by date and pagination, to be able to fetch list of deleted records in chunks.
• If the authority retention policy is set to NEVER_PURGE: should ignore the hard deletion part. This can be reached by disabling scheduled job.

Process:

Open Questions