FOLIO Harvester - new FOLIO module (UXPROD-2293)

[UXPROD-4521] GBV. FOLIO module wrapper (mod-harvester-admin, ui-harvester-admin) around existing Harvester (Ramsons work) Created: 23/Oct/23  Updated: 08/Feb/24

Status: Open
Project: UX Product
Components: None
Affects versions: None
Fix versions: Ramsons (R2 2024)
Parent: FOLIO Harvester - new FOLIO module

Type: New Feature Priority: P2
Reporter: Charlotte Whitt Assignee: Charlotte Whitt
Resolution: Unresolved Votes: 0
Labels: CBS2FOLIO, GBV
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File Skärmavbild 2021-06-16 kl. 13.11.01.png     PNG File Skärmavbild 2021-06-16 kl. 13.11.27.png     PNG File Skärmavbild 2021-06-16 kl. 13.11.51.png    
Issue links:
Cloners
clones UXPROD-4463 GBV. FOLIO module wrapper (mod-harves... In Progress
is cloned by UXPROD-4658 GBV. CBS to FOLIO. Further enhancemen... Open
Defines
defines MODHAADM-82 Add WSAPIs for accessing OAI-PMH serv... Open
is defined by MODHAADM-38 OS license for com.indexdata:masterke... Open
is defined by MODHAADM-64 Display descriptive HTTP statuscode w... Open
is defined by UIHAADM-99 The three tabs Harvestables, Jobs, Fa... Open
is defined by UIHAADM-121 Column-width instability in the jobs ... Open
is defined by MODHAADM-85 Jobs > Only 100 failed records are st... Closed
is defined by UIHAADM-14 Mediate CORS access from UI module to... Closed
is defined by UIHAADM-4 Detect CF engines and CF repositories... Blocked
is defined by MODINVUP-85 Only one OUF-container for all tenants In Refinement
Release: Ramsons (R2 2024)
Epic Link: FOLIO Harvester - new FOLIO module
Front End Estimate: XL < 15 days
Front End Estimator: Charlotte Whitt
Back End Estimate: Large < 10 days
Back End Estimator: Niels Erik Nielsen
Development Team: Sif
PO Rank: 0
Rank: Cornell (Full Sum 2021): R5
Rank: GBV (MVP Sum 2020): R1
Rank: Mainz (Full TBD): R1

 Description   

Current situation or problem: Remaining work on the FOLIO wrapper for Harvester Admin. The wrapper interacts for now with the legacy Harvester.

In scope: FOLIO module wrapper around existing Harvester for GBV. 2nd phase of FOLIO Harvester module.

Out of scope

Use case(s): For the German libraries - GBV, Hebis, Mainz then ... (enter text here)

Proposed solution/stories

Links to additional info

Questions

Out of scope:
Harvester version 2.0. 2nd phase FOLIO Harvester module

Use case(s):
As an admin from library in the GBV consortia,
I need to be abel to run (start, stop, general access to) my library's jobs in the Harvester

Proposed solution/stories:
Solution that meets GBV’s requirements for giving different admins access to only their own jobs in the Harvester. It assumes that libraries will not have access to transformation pipelines, transformation steps or back-end (Inventory storage) configurations, which are assumed to be maintained in the existing Harvester Admin UI by a central maintainer.

The basic idea is for a solution that utilizes current FOLIO user management modules and authentication - and applies access control on row level in the existing Harvester database. The idea is thus to discard the existing Harvester admin UI for jobs management but use the existing Harvester admin REST APIs by wrapping these XML based REST APIs in JSON based APIs. This is to make it possible to put a Stripes admin UI on top of them. The underlying MySQL database could be kept as is, initially at least, possibly save for a few tweaks. The harvest scheduler and harvest job running logic would also not change.

The suggestion is to implement access control by tagging all records in the Harvester database with tokens specifying who created the records and who should thus be able to retrieve them again. The simplest approach is possibly to install the Harvester module in a given FOLIO, create a user per library in that FOLIO and filter jobs by user ID. Alternatively it might be possible to control access per tenant.

The existing Harvester database and its REST APIs would in other words become the storage layer for the new JSON APIs.

The FOLIO module, ie mod-harvester-admin, would need to know where the Harvester resides. This could be a FOLIO tenant setting, but that would require that module to have a storage and the UI to provide the settings page. Mod-configuration might be another option. A simpler solution might be to declare the Harvester’s location in the deployment descriptor - pretty much the same way access to PostgreSQL is configured in other FOLIO storage modules.

It should be considered changing the current primary key scheme in the Harvester database from sequence generated primary keys to UUIDs. This would bring the schema closer to FOLIO conventions (and have unrelated configuration maintenance benefits)

The project would thus include:

  • Wrap select XML APIs as JSON APIs, provide unit tests.
  • Perhaps make a few changes to the underlying database for more FOLIO like JSON schemas
  • Write Stripes UIs with unit tests, to replace existing JSF UIs
  • Implement row level access control

Following APIs would need to be implemented in JSON

  • Storages (GET, to populate drop-down in jobs edit page)
  • Jobs (POST, PUT, GET, DELETE)
  • Transformation pipelines (GET, to populate drop-down in jobs edit page)
  • Job logs (GET, possibly just text passthrough rather than JSON)
  • Failed records (GET)

Following existing JSF based UIs would be implemented as Stripes front-ends - not necessarily in a one-to-one fashion but at least so that existing functionality will continue to be supported.

Links to additional info:
https://folio-org.atlassian.net/wiki/pages/viewpage.action?pageId=1387435

Questions



 Comments   
Comment by Charlotte Whitt [ 15/Nov/23 ]

Hi Felix Hemme and Antje Niemann - did you find time to move over the possible jira stories listed under UXPROD-4463 In Progress , which we safely can postpone to the Ramsons release?

Comment by Felix Hemme [ 17/Nov/23 ]

Hi Charlotte Whitt , our priority this week was to work on the issue we had found in Bremens Inventory ( MODINVUP-81 Closed ). Yesterday Antje and I went through the document of work packages for 2024 in preparation for a talk with Kirstin. With MODHAADM-76 Closed done since yesterday we are now able to update UIHA to v2.0.0 and verify some of the bugs found. After we are done with that, we will take a look at UXPROD-4463 In Progress and see what tickets are higher / lower priority.

Comment by Charlotte Whitt [ 17/Nov/23 ]

Sounds all good Felix Hemme

Generated at Fri Feb 09 00:40:33 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.