Support export deleted MARC authority records via API

Requirements

• Support retrieving deleted authority record UUIDs via API. The work to export MARC authority records via API is already supported. This feature focus is on retrieving deleted authority record UUIDs to run a data export MARC authority records job. 
• API may support limits and pagination when retrieving UUIDs
• API must allow user to query for deleted records by authority source file, headings type, date created, and date updated facets/filters
• Support authority record purge configuration (environment variable) to never purge for a tenant. 
• Implementation and verification must account for non-enhanced consortia and enhanced consortia support environments 

Scenarios:

Scenario 1: 

As a system admin, I need to export the last 7 days worth of deleted MARC authority records so I can do some additional analysis 

Scenario 2: 

As a system admin, I need to export the last 7 days worth of deleted MARC authority records with the heading types: Personal name OR Corporate name OR Conference name so that I can do some additional analysis

Scenario 3: 

As a system admin, I need to export all deleted MARC authority records with the heading types:  Personal name OR Corporate name OR Conference name so that I can do some additional analysis

Scenario 4: 

As a system admin, I need to export all deleted MARC authority records with the authority source file: Library of Congress Name Authority File so that I can do some additional analysis

Scenario 5: 

As a system admin, I need to export all deleted MARC authority records with the authority source file: Library of Congress Name Authority File AND heading type = Personal so that I can do some additional analysis

Scenario 6: 

As a system admin, I need to export all deleted MARC authority records with the authority source file: Library of Congress Name Authority File over the last 14 days so that I can do some additional analysis

User Interface

Not applicable to the feature

Technical Design

Architecture

MARC Authority deletion (Architectural notes)

Database Schema

New table is in red on the diagram. Diagram in DBML format: mod-entities-links-dbml.txt

API Endpoints

Additional parameters will be added to GET /authority-storage/authorities endpoint.

Data Flow

Development Tasks

1. mod-eintities-links:
   1. Implement logic for archiving deleted records:
      1. Create a new table authority_archive with same structure as authority table.
      2. Create a trigger that will on delete move the record to archive table.
   2. Implement scheduled job that will get retention period configuration and based on it delete records from archive table.
   3. Migration of records that has deleted=true to archive table.
   4. Update logic of publishing domain events for deletion, change it to send 2 events: for soft- and hard-deletion.
   5. Align GET /authority-storage/authorities endpoint for retrieving deleted records:
      1. Make sure that CQL request can handle authority source file, headings type, date created, and date updated filters (update README with all possible options for CQL queries). Add required indices to the table.
      2. Add parameter deleted=true/false to the endpoint that will change destination table (authority or authority_archive).
      3. Add parameter idOnly=true/false to the endpoint that will indicate that only authority IDs have to be in the response.
      4. For deleted=true AND idOnly=true support content-type=plain/text
2. mod-source-record-storage:
   1. Remove scheduled logic for deletion of records with state = DELETED
   2. Consume authority domain events:
      1. On soft-delete mark the record with state = DELETED
      2. On hard-delete delete record from database.
      3. Ignore other domain types.
3. mod-search
   1. Ignore hard-delete events for authorities
4. UI
   1. Change delete authority logic to use DELETE  /authority-storage/authorities/<id> endpoint.

Testing

Test Cases

Test Case 1: Valid Deletion of Authority Record

Scenario: Deleting an Authority record by ID
Test Steps:

1. Given a valid Authority record ID exists in the system.
2. When the user deletes the Authority record using its ID.
3. Then the record should be deleted successfully.
4. And attempting to access the record by ID should result in a non-existent or inaccessible response.
5. And the related source record's state field should be marked as "DELETED".
6. And the 'deleted' attribute for the related source record should be set to true.
7. And the 'leaderRecordStatus' for the related source record should be 'd'.

Expected Outcome: All the specified conditions should be met after the deletion of the Authority record.

Test Case 2: Invalid Deletion Attempts

Scenario: Attempting to delete a non-existent Authority record
Test Steps:

1. Given an Authority record ID that does not exist in the system.
2. When the user attempts to delete the Authority record using this non-existent ID.
3. Then a suitable error or failure message should be returned.
4. And there should be no impact or change to any existing records.

Expected Outcome: The system should handle attempts to delete non-existent records gracefully without affecting other data.

Test Case 4: Filter by Source File

Scenario: Filtering authorities by a specific source file
Test Steps:

1. Given multiple Authority records exist with different source files.
2. When filtering authorities using a specific source file.
3. Then the API should return only the Authority records associated with that particular source file.
4. And records from other source files should not be present in the response.

Expected Outcome: The API should accurately filter and return only the Authority records associated with the specified source file.

Test Case 5: Filter by Headings Type

Scenario: Filtering authorities by a certain headings type
Test Steps:

1. Given multiple Authority records exist with different heading types.
2. When filtering authorities using a specific headings type.
3. Then the API should return only the Authority records that match the specified headings type.
4. And records with different heading types should not be included in the response.

Expected Outcome: The API should correctly filter and return only the Authority records that match the specified headings type.

Test Case 6: Filter by Date Created

Scenario: Filtering authorities by the date they were created
Test Steps:

1. Given multiple Authority records exist with various creation dates.
2. When filtering authorities based on a specific creation date range.
3. Then the API should return only the Authority records that were created within that date range.
4. And records created outside the specified date range should not be present in the response.

Expected Outcome: The API should effectively filter and return only the Authority records created within the specified date range.

Test Case 7: Filter by Date Updated

Scenario: Filtering authorities by the date they were last updated
Test Steps:

1. Given multiple Authority records exist with different update dates.
2. When filtering authorities based on a specific date range of updates.
3. Then the API should return only the Authority records that were updated within that date range.
4. And records updated outside the specified date range should not be included in the response.

Expected Outcome: The API should accurately filter and return only the Authority records updated within the specified date range.

Test Case 8: Combination of Filters

Scenario: Using multiple filters together
Test Steps:

1. Given a variety of Authority records with diverse source files, heading types, creation dates, and update dates.
2. When applying multiple filters simultaneously (e.g., source file, heading type, creation/update date ranges).
3. Then the API should return records that satisfy all specified filters simultaneously.
4. And records not meeting all filter criteria should be excluded from the response.

Expected Outcome: The API should correctly handle and apply multiple filters together, returning only the Authority records that meet all specified criteria simultaneously.

Test Case 9: Retrieving Archived Authority Records with Multiple Filters

Scenario: Fetching archived Authority records using multiple filters
Test Steps:

1. Given a mix of Authority records, some of which are archived and others that are active.
2. When retrieving archived Authority records using a combination of filters such as source file, heading type, creation date, and update date ranges.
3. Then the API should return only the archived Authority records that satisfy all specified filters simultaneously.
4. And active records or those not meeting all filter criteria should be excluded from the response.

Expected Outcome: The API should accurately filter and return only the archived Authority records that match all specified criteria (source file, heading type, creation date, update date) for archived records, leaving out any active records or those not meeting the filtering criteria.

Performance testing

Provide results of performance testing if needed.

Load testing

1. Check how much IDs could be returned in one page.

NFR Scorecard

UXPROD-4305 NFR Scorecard

Deployment

Deployment notes

Describe the deployment process.

Migration

Describe required migrations.

Documentation

Provide links to or include any documentation related to the feature, such as API documentation or user guides.

Dependencies

No dependencies

Risks and Mitigations

No risks