MARC Authority - Phase 1 Features (UXPROD-2944)

[MODELINKS-187] BE: Update MARC authority record deletion handling > Make [soft_delete_ttl_day] tenant configurable Created: 18/May/23  Updated: 26/Jan/24  Resolved: 26/Jan/24

Status: Closed
Project: mod-entities-links
Components: None
Affects versions: None
Fix versions: 3.0.0
Parent: MARC Authority - Phase 1 Features

Type: Story Priority: P3
Reporter: Khalilah Gambrell Assignee: Pavlo Smahin
Resolution: Done Votes: 0
Labels: back-end, epam-spitfire, release-notes-required, testing
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File MODSOURCE-638_Central_retention_days_is_changed_2_days_after_delete.png     PNG File MODSOURCE-638_Central_retention_days_is_set_before_delete.png     PNG File MODSOURCE-638_Central_tenant_shared_record_deleted_from_SRS_after_retentionInDays_change.png     PNG File MODSOURCE-638_Member_change_local_authority_record_SRS_after_delete.png     PNG File MODSOURCE-638_Member_change_local_authority_record_deleted_from_SRS_after_2_days.png     PNG File MODSOURCE-638_Member_change_local_authority_record_linked_before_delete.png     PNG File MODSOURCE-638_Member_change_shared_authority_record_SRS_after_2_days(not_deleted).png     PNG File MODSOURCE-638_Member_change_shared_authority_record_SRS_after_delete.png     PNG File MODSOURCE-638_Member_change_shared_authority_record_linked_before_delete.png     PNG File MODSOURCE-638_Member_retention_days_is_set_before_delete.png     PNG File MODSOURCE-638_Member_retention_days_is_set_before_delete_member_tenant.png     PNG File MODSOURCE-638_authority_record_linked_SRS_right_after_delete.png     PNG File MODSOURCE-638_authority_record_linked_before_delete.png     PNG File MODSOURCE-638_authority_record_linked_deleted_from_SRS_after_2_days.png     PNG File MODSOURCE-638_authority_record_not_linked_SRS_right_after_delete.png     PNG File MODSOURCE-638_authority_record_not_linked_deleted_from_SRS_ater_2_days.png     PNG File MODSOURCE-638_instance_links_before_delete.png     PNG File MODSOURCE-638_instance_links_right_after_delete.png     PNG File MODSOURCE-638_retention_days_is_set_before_delete.png     Text File mod-entities-links-8d5667466-kr48l_mod-entities-links.log    
Issue links:
Cloners
clones UXPROD-4304 Delete authority record updates LDR 0... Closed
Relates
relates to MODSOURCE-459 Delete authority record via user inte... Closed
relates to UXPROD-3517 Delete MARC authority records via UI Closed
relates to FAT-11207 Review of C430211 test case Closed
relates to FAT-11208 Review of C431146 test case Closed
relates to FAT-11209 Review of C431149 test case Closed
Requires
is required by UXPROD-4305 Support export deleted MARC authority... In Progress
Sprint: Spitfire Sprint 182, Spitfire Sprint 183
Story Points: 2
Development Team: Spitfire
Release: Quesnelia (R1 2024)
Epic Link: MARC Authority - Phase 1 Features
Back End Estimator: Khalilah Gambrell

 Description   

Background: There are some libraries that do not want authority records to be purged (IOW hard delete). 

Requirements:

  • Make soft_delete_ttl_days a tenant configurable setting
  • Default = 7 days
  • Maximum number of days = Never
  • Minimum number of days = 7 days
  • There will be no UI for this setting. Can be set via API with this story.
  • More details are below under Additional details section. 

Use case 1

Given my tenant is setup to Never purge

When I delete an authority record 

Then it should never return in search result

AND the record and all versions should remain in SRS

AND if connected mod-inventory storage record should remain too 

AND Both authority records (mod-inventory-storage and SRS)  should also have the state = Deleted 

 

Use case 2

Given my tenant is setup to purge after 1000 days

When I delete an authority record 

Then it should never return in search result

AND the record and all versions should remain in SRS for 1000 days then purge

AND if connected mod-inventory storage record should remain for 1000 then purge

AND Both authority records (mod-inventory-storage and SRS)  should also have the state = Deleted 

 

Additional details

Configuration/Settings:

  1. Where and how can I set the flag NEVER or a custom number of days (UI or via API)? API only 
  2. Is it common for SRS and Inventory? Yes - we should not have two different values for deletion. We have one value and it serves to handle authority record stored in mod-inventory-storage and mod-source-record-storage 
  3. Is it applied to all deleted records or selectively? Not sure what this means but it applies to authority records since we do support a hard delete. Might be a solution to use for for other record types that support hard deletes. 
  4. What should be done with those that were deleted earlier (5 days ago) when configurable soft_delete_ttl_days  changed? If the setting is changed then it would be ideal to have all records that have not been hard deleted to be updated with the new setting value. We can document this behavior and also provide details in the API response? If the record has been hard deleted then no expectation to recover. 

Dependencies:

  1. Are there any other dependencies on Authority Records (besides in the Inventory Instance and SRS MARC Authority) that we may do not know?  No

Data Management:

  1. Even with soft delete, do all links to authority records also need to be cleaned up through mod-entities-links? Yes. No change to the following requirements 1.) User can delete a linked authority record 2.) When the linked authority record is deleted then the linked bib field will be unlinked and will display bib field information that was previous linked as editable 

Data Usage:

  1. Are historical SRS data (previous versions, and deleted versions) used in any way now? I do not know of any 
  2. Do deleted records need to be obtained in a specific manner? No change from current way we store these records. I assume we store some state that indicates the record is Deleted? If so, I do not see a need for a change unless Raman states there is a need. 
  3. What happens when requesting a record by its direct ID when it was soft deleted? 1.) Depends on how we implement the export of deleted authority records - MODELINKS-133 In Review - 2.)  Assuming it is UUID that is called via mod-search then show an API message that this MARC authority has been deleted. The UI message should not change, no results returned message should display.


 Comments   
Comment by Pavlo Smahin [ 26/May/23 ]

Khalilah Gambrell, what does "if applicable" mean? In which cases it could be applicable or not?

Comment by Pavlo Smahin [ 26/May/23 ]

There is related discoverings exists for instances: https://folio-org.atlassian.net/wiki/display/DD/Mark+instance+for+deletion

Comment by Khalilah Gambrell [ 28/May/23 ]

Hey Pavlo Smahin 

Comment by Pavlo Smahin [ 15/Jun/23 ]

Kalibek Turgumbayev, I believe that this required SA work.
1. Have to be defined the best approach for soft-delete in mod-inventory-storage.
2. SRS already do a soft delete of MARC authority records and purge them in some time.
3. Have to be defined how to share soft_delete_ttl_days between SRS and inventory-storage.

Comment by Khalilah Gambrell [ 16/Jun/23 ]

Hey Kalibek Turgumbayev  - you may want to discuss approach with Raman Auramau as he is working on a platform wide approach for handling deletes and has design for handling instances that could mirror how we address soft-deletes for authority records. 

Comment by Pavlo Smahin [ 16/Jun/23 ]

Khalilah Gambrell, several questions based on discussion with Kalibek and Raman:

Configuration/Settings:

  1. Where and how can I set the flag NEVER or a custom number of days (UI or via API)?
  2. Is it common for SRS and Inventory?
  3. Is it applied to all deleted records or selectively?
  4. What should be done with those that were deleted earlier (5 days ago) when configurable soft_delete_ttl_days  changed?

Dependencies:

  1. Are there any other dependencies on Authority Records (besides in the Inventory Instance and SRS MARC Authority) that we may do not know?

Data Management:

  1. Even with soft delete, do all links to authority records also need to be cleaned up through mod-entities-links?

Data Usage:

  1. Are historical SRS data (previous versions, and deleted versions) used in any way now?
  2. Do deleted records need to be obtained in a specific manner?
  3. What happens when requesting a record by its direct ID when it was soft deleted?
Comment by Khalilah Gambrell [ 18/Jun/23 ]

Pavlo Smahin - added details to the user story. 

Comment by Viacheslav Kolesnyk [ 05/Oct/23 ]

Should be implemented after "Authority delete" feature. Clarify if this should be a tenant initialization parameter

Comment by Khalilah Gambrell [ 17/Oct/23 ]

Per October 16th discussion: May just support an environment variable as only one library is requesting the need to keep deleted authority records forever. Will be further defined in technical design. 

cc: Kalibek Turgumbayev and Pavlo Smahin and Viacheslav Kolesnyk 

Comment by Khalilah Gambrell [ 26/Oct/23 ]

Hey Pavlo Smahin  - Can I close this story?

Comment by Valery_Pilko [ 10/Jan/24 ]

During testing on https://folio-dev-spitfire-diku.ci.folio.org/, we figured out that the following error appears when module tries to get changed "retentionInDays" value from endpoint "/settings/entries".
Here is the SRS UUIDs of deleted records:
1) 876f5738-84db-4a7e-af0a-feb5d99a4f79
2) a45799a7-a878-4d37-8511-7ff0b8ec7d4a
Logs: mod-entities-links-8d5667466-kr48l_mod-entities-links.log
Time of records deletion was: 2024-01-09 11:39
cc: Pavlo Smahin 

Caused by: feign.FeignException$InternalServerError: [500 Internal Server Error] during [GET] to [http://settings/entries?query=scope%3Dauthority-storage%2520AND%2520key%3Dauthority-archives-expiration&limit=10000] [SettingsClient#getSettingsEntries(String,int)]: [Unsupported CQL index: authority-storage%20AND%20key]
Comment by Valery_Pilko [ 11/Jan/24 ]

Re-tested on https://folio-dev-spitfire-diku.ci.folio.org/ after Pavlo's fix - works as expected
See test results below:
Test 1:
1) "RetentionInDays" parameter was set for 1 day (that means that 1 day from MARC authority record deletion record should be deleted from "SRS" module).
See attached screenshot:

2) One "MARC authority" record (with "UUID" = a45799a7-a878-4d37-8511-7ff0b8ec7d4a) was linked with "MARC bib" record.
See attached screenshots:

3) One linked "MARC authority" record and one not linked record (with "UUID" = 876f5738-84db-4a7e-af0a-feb5d99a4f79) were deleted. Link was deleted automatically in the same moment.
See attached screenshots:

4) Two days after records deletion, there are no deleted record in "SRS" module.|
See attached screenshots:

 

Test 2 (ECS):
1) "RetentionInDays" parameter was set for 1 day in Member tenant (College) 
"RetentionInDays" parameter was not changed, in Central tenant (Consortium) 
See attached screenshots:

2) Two linked "MARC authority" records (Local and Shared) exist.
See attached screenshots:

3) Both records were deleted.
See attached screenshots:

4) After 2 days only local "MARC authority" record is deleted from "SRS " module
See attached screenshots:

 

Test 3 (ECS):
1) For test, used the same shared "MARC authority" as was used for Test 2 ("SRS UUID" = f0352496-2ff3-4bd1-9323-08bd23fa2a38)
"RetentionInDays" parameter was set to 10 days in Member tenant (College) 
"RetentionInDays" parameter was set to 1 day, in Central tenant (Consortium) 
See attached screenshots:

2) Right after "RetentionInDays" parameter to "1" day in Central tenant, shared record was deleted from "SRS" module, because it was deleted on UI 2 days ago. 
"RetentionInDays" parameter to "1" day in Central tenant - doesn't affect "Local" records from member tenants.
Note: deletion happened in the same moment because "trigger" which checks records deletion date and current "RetentionInDays" parameter was set to "1" min on Rancher env. By default, "trigger" value will be "1" day.

Generated at Fri Feb 09 00:38:56 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.