FOLIO Harvester - new FOLIO module (UXPROD-2293)

[UXPROD-4064] GBV. FOLIO module wrapper around existing Harvester (Extended Poppy work) Created: 14/Feb/23  Updated: 05/Feb/24  Resolved: 04/Oct/23

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: Poppy (R2 2023)
Parent: FOLIO Harvester - new FOLIO module

Type: New Feature Priority: P2
Reporter: Charlotte Whitt Assignee: Charlotte Whitt
Resolution: Done Votes: 0
Labels: CBS2FOLIO, GBV, prio1, was-quesnelia-now-poppy
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File Skärmavbild 2021-06-16 kl. 13.11.01.png     PNG File Skärmavbild 2021-06-16 kl. 13.11.27.png     PNG File Skärmavbild 2021-06-16 kl. 13.11.51.png    
Issue links:
Blocks
is blocked by UXPROD-3013 GBV. Consider design for integration ... Closed
Cloners
clones UXPROD-3892 GBV. FOLIO module wrapper around exis... Closed
Defines
is defined by MODHAADM-65 Logs are not written to mod-harvester... Closed
is defined by MODHAADM-66 Fix FOLIO Instance XML to JSON conver... Closed
is defined by MODHAADM-70 Reduce excessive logging in localindi... Closed
is defined by UIHAADM-7 Support management of storage engines Closed
is defined by UIHAADM-73 Permissions - UI Harvester Admin Closed
is defined by UIHAADM-75 Add guards to hide parts of the UI th... Closed
is defined by UIHAADM-80 Stopping a running job does not work Closed
is defined by UIHAADM-81 After deleting a harvestable, the UI ... Closed
is defined by UIHAADM-84 Starting an already-running job, or s... Closed
is defined by UIHAADM-85 Changes made in any record in setting... Closed
is defined by UIHAADM-86 "Provided object contains unexpected ... Closed
is defined by UIHAADM-88 When duplicating a step, only fields ... Closed
is defined by MODHAADM-73 Tighten storage schema: make property... Closed
is defined by UIHAADM-83 Progressive loading for Log pages Closed
is defined by UIHAADM-91 Jobs component: Display number of fai... Closed
is defined by UIHAADM-82 Provide ability to download last log ... Closed
Release: Poppy (R2 2023)
Epic Link: FOLIO Harvester - new FOLIO module
Front End Estimate: XL < 15 days
Front End Estimator: Charlotte Whitt
Back End Estimate: Large < 10 days
Back End Estimator: Niels Erik Nielsen
Development Team: Sif
PO Rank: 0
Rank: Cornell (Full Sum 2021): R5
Rank: GBV (MVP Sum 2020): R1
Rank: Mainz (Full TBD): R1

 Description   

Current situation or problem: Wrap up the work on the FOLIO wrapper for Harvester Admin. The wrapper interacts for now with the legacy Harvester.

In scope: FOLIO module wrapper around existing Harvester for GBV. 1st phase of FOLIO Harvester module.

  1. Logfiles should be stored
  2. UI display of log data

Out of scope

Use case(s): For the German libraries - GBV, Hebis, Mainz then ... (enter text here)

Proposed solution/stories

Links to additional info

Questions

Out of scope:
Harvester version 2.0. 2nd phase FOLIO Harvester module

Use case(s):
As an admin from library in the GBV consortia,
I need to be abel to run (start, stop, general access to) my library's jobs in the Harvester

Proposed solution/stories:
Solution that meets GBV’s requirements for giving different admins access to only their own jobs in the Harvester. It assumes that libraries will not have access to transformation pipelines, transformation steps or back-end (Inventory storage) configurations, which are assumed to be maintained in the existing Harvester Admin UI by a central maintainer.

The basic idea is for a solution that utilizes current FOLIO user management modules and authentication - and applies access control on row level in the existing Harvester database. The idea is thus to discard the existing Harvester admin UI for jobs management but use the existing Harvester admin REST APIs by wrapping these XML based REST APIs in JSON based APIs. This is to make it possible to put a Stripes admin UI on top of them. The underlying MySQL database could be kept as is, initially at least, possibly save for a few tweaks. The harvest scheduler and harvest job running logic would also not change.

The suggestion is to implement access control by tagging all records in the Harvester database with tokens specifying who created the records and who should thus be able to retrieve them again. The simplest approach is possibly to install the Harvester module in a given FOLIO, create a user per library in that FOLIO and filter jobs by user ID. Alternatively it might be possible to control access per tenant.

The existing Harvester database and its REST APIs would in other words become the storage layer for the new JSON APIs.

The FOLIO module, ie mod-harvester-admin, would need to know where the Harvester resides. This could be a FOLIO tenant setting, but that would require that module to have a storage and the UI to provide the settings page. Mod-configuration might be another option. A simpler solution might be to declare the Harvester’s location in the deployment descriptor - pretty much the same way access to PostgreSQL is configured in other FOLIO storage modules.

It should be considered changing the current primary key scheme in the Harvester database from sequence generated primary keys to UUIDs. This would bring the schema closer to FOLIO conventions (and have unrelated configuration maintenance benefits)

The project would thus include:

  • Wrap select XML APIs as JSON APIs, provide unit tests.
  • Perhaps make a few changes to the underlying database for more FOLIO like JSON schemas
  • Write Stripes UIs with unit tests, to replace existing JSF UIs
  • Implement row level access control

Following APIs would need to be implemented in JSON

  • Storages (GET, to populate drop-down in jobs edit page)
  • Jobs (POST, PUT, GET, DELETE)
  • Transformation pipelines (GET, to populate drop-down in jobs edit page)
  • Job logs (GET, possibly just text passthrough rather than JSON)
  • Failed records (GET)

Following existing JSF based UIs would be implemented as Stripes front-ends - not necessarily in a one-to-one fashion but at least so that existing functionality will continue to be supported.

Links to additional info:
https://folio-org.atlassian.net/wiki/pages/viewpage.action?pageId=1387435

Questions


Generated at Fri Feb 09 00:36:59 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.