SPIKE: provide data about deleted inventory instances during the harvest

Owned by Oleksandr Bozhko
Last updated: May 12, 2023
2 min read

When using metadataPrefix=marc21, instances are retrieved from Inventory with limit and offset parameters through the /instance-storage/instances endpoint on mod-inventory-storage side, but deletedRecordsSupport parameter is not supported, therefore instances are searched only in the instance table, but not in the audit_instance. The same situation when using metadataPrefix=marc21_withholdings where instance IDs are downloaded firstly, then instances are retrieved from Inventory through the /instance-storage/instances endpoint without limit and offset parameters, but with list of IDs downloaded previously.

In both cases the problem is with the /instance-storage/instances endpoint that does not support deleted instances.

Based on the investigation, it was found that there are two possible ways how to include deleted instances into the harvest:

1) Modify /instance-storage/instances endpoint to include deleted records support and search in the audit_instance table as well

2) Create view like "get_deleted_instances" mentioned in this spike on the OAI-PMH side

First way assumes to make changes on mod-inventory-storage side and if deletedRecordsSupport=true, join audit_instance table to the query where  /instance-storage/instances endpoint is handled. On the mod-oai-pmh side, it requires minimum changes, only to add deletedRecordsSupport parameter to AbstractGetRecordsHelper#requestFromInventory(...) method.

Second way involves work only on the mod-oai-pmh side and creating the following view:

create or replace view {tenant id}_mod_oai_pmh.get_deleted_instances
as
select *
from {tenant id}_mod_inventory_storage.audit_instance inst
where 
inst.instance_updated_date <= {tenant id}_mod_inventory_storage.dateOrMax(timestamptz '{date_from}') // optional
and
inst.instance_updated_date >= {tenant id}_mod_inventory_storage.dateOrMin(timestamptz '{date_until}') // optional
and
coalesce(inst.suppress_from_discovery_srs, inst.suppress_from_discovery_inventory) = false; // optional

This view can be used along with AbstractGetRecordsHelper#requestFromInventory(...) method to search for deleted instances if deletedRecordsSupport=true and append them to the final result. Depending on the parameters, date_from, date_until and discovey_suppress may or may not be used in the view. Like in this spike, for the deleted instances the QueryBuilder can be applied as well to build this view, either separately (new QueryBuilder to handle only deleted instances), or in scope of the existing QueryBuilder by adding functionality to handle only deleted instances. However, last variant does not make sense once the new approach for OAI-PMH is implemented since it covers support for deleted instances. In this case, deleted instances support can be postponed until MODOAIPMH-490, MODOAIPMH-491 and MODOAIPMH-492 are completed.