[Instance/Item/Holding/Bib/Authority Audit] Scheduled job

Description

Create a scheduled nightly job to clean up the records that exceed the retention period.
Acceptance Criteria:

  1. The job execution should be skipped if the configuration for the retention period is 0 (indefinite period)

  2. The job is applied to all entity types (including MARC)

  3. Use folio timer api to schedule a job(see mod-entities-links for reference). Use cron format so jobs runs at night

-- main audit table CREATE TABLE instance_audit( event_id uuid primary key, event_date timestamp not null, entity_id uuid not null, ---other fields omitted ) PARTITION BY HASH(entity_id); -- main partitions CREATE TABLE instance_audit_p0 PARTITION OF instance_audit FOR VALUES WITH (MODULUS 8, REMAINDER 0) PARTITION BY RANGE(event_date); CREATE TABLE instance_audit_p1 PARTITION OF instance_audit FOR VALUES WITH (MODULUS 8, REMAINDER 1) PARTITION BY RANGE(event_date); CREATE TABLE instance_audit_p2 PARTITION OF instance_audit FOR VALUES WITH (MODULUS 8, REMAINDER 2) PARTITION BY RANGE(event_date); CREATE TABLE instance_audit_p3 PARTITION OF instance_audit FOR VALUES WITH (MODULUS 8, REMAINDER 3) PARTITION BY RANGE(event_date); CREATE TABLE instance_audit_p4 PARTITION OF instance_audit FOR VALUES WITH (MODULUS 8, REMAINDER 4) PARTITION BY RANGE(event_date); CREATE TABLE instance_audit_p5 PARTITION OF instance_audit FOR VALUES WITH (MODULUS 8, REMAINDER 5) PARTITION BY RANGE(event_date); CREATE TABLE instance_audit_p6 PARTITION OF instance_audit FOR VALUES WITH (MODULUS 8, REMAINDER 6) PARTITION BY RANGE(event_date); CREATE TABLE instance_audit_p7 PARTITION OF instance_audit FOR VALUES WITH (MODULUS 8, REMAINDER 7) PARTITION BY RANGE(event_date); -- sub-partitions CREATE TABLE instance_audit_p0_2025_q1 PARTITION OF instance_audit_p0 FOR VALUES FROM ('2024-01-01') TO ('2025-03-31'); CREATE TABLE instance_audit_p0_2025_q2 PARTITION OF instance_audit_p0 FOR VALUES FROM ('2024-04-01') TO ('2025-06-30'); CREATE TABLE instance_audit_p0_2025_q3 PARTITION OF instance_audit_p0 FOR VALUES FROM ('2024-07-01') TO ('2025-09-30'); CREATE TABLE instance_audit_p0_2025_q4 PARTITION OF instance_audit_p0 FOR VALUES FROM ('2024-10-01') TO ('2025-12-31'); -- etc...
SELECT inhrelid::regclass::text AS child FROM pg_catalog.pg_inherits WHERE inhparent = 'diku_mod_audit.instance_audit_p0'::regclass;

Environment

None

Potential Workaround

None

is defined by

relates to

Checklist

hide

Activity

Show:

Viacheslav Kolesnyk February 28, 2025 at 9:30 AM
Edited

Retention period was set to 50 days for each settings group (inventory/authority). Partitions for previous period (Q4 2024) were created for every entity type (instance/item/holdings/marc-bib/marc-authority) and filled with 1 record for each entity type. Records were deleted from database overnight, as expected. Tested on etesting sprint env, non-consortium tenant

Viacheslav Kolesnyk February 19, 2025 at 10:18 AM

Do you know how can we specify a correct time zone to run the job at night?

Valery_Pilko February 13, 2025 at 11:40 AM

What minimal parameter for retention period could possibly be? Asking from testing perspective.
Will it be possible to set 1 day at least?

Natalia Zaitseva January 6, 2025 at 3:39 PM

the issue can be taken by any team (Spitfire / Folijet) once they have capacity

Done

Details

Assignee

Reporter

Priority

Story Points

Sprint

Development Team

Spitfire

Fix versions

Release

Sunflower (R1 2025)

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs

Created December 19, 2024 at 10:10 AM
Updated March 14, 2025 at 3:49 AM
Resolved March 5, 2025 at 4:33 PM
TestRail: Cases
TestRail: Runs

Flag notifications