[UXPROD-3886] Support 'push' based mechanism for populating Agreements Local KB Created: 31/Oct/22  Updated: 08/Feb/24  Resolved: 09/Oct/23

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: Poppy (R2 2023)

Type: New Feature Priority: TBD
Reporter: Owen Stephens Assignee: Owen Stephens
Resolution: Done Votes: 0
Labels: erm, local_kb
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Defines
is defined by ERM-2630 Add new endpoint for external KB "pus... Closed
Relates
relates to UXPROD-4272 Make it easy to check equity between ... Open
relates to UXPROD-4271 Support date created and updated meta... Closed
relates to ERM-2631 Write service to use the 'push' servi... In Progress
relates to UXPROD-3885 Allow external data sources to be aut... Closed
Release: Poppy (R2 2023)
Development Team: Bienenvolk
PO Rank: 0

 Description   

Current situation or problem:

Currently the main mechanism for populating the Agreements local KB uses:

  • Uses a 'pull' based mechanism where Agreements runs code to fetch data from the remote data source
  • Assumes that data will be provided a complete package at a time (i.e. we always get a full list of titles & coverage for each package we ingest)

Specifically Agreements uses an OAI-PMH feed to synchronise data from GOKb to the local KB. With large packages this can lead to having to process very large chunks of XML and failure can result in incomplete package harvest with no good mechanism for picking up the harvest from where it left off (the whole package has to be re-harvested in the case of failure mid-way through a package).

With a "pull" methodology the code that understands the remote data source API and data format has to live in a particular release of mod-agreements, which can be fragile if the external API or format changes

This feature is to support a "push"  move away from the "pull" and package based synchronisation in order to allow for more incremental synchronisation that is less intensive in terms of processing, deals with more granular data and is able to recover more elegantly from failure

In scope

  • Move away from package based harvesting
  • A new endpoint which accepts data 'pushed' to it to populate the Agreements local KB
  •  

Out of scope

  • Other changes to data sync or data loading from other sources

Links to additional info

  • The new method will be optional, switched at compile time.
  • See also UXPROD-3885 Closed which also touches on synchronisation

 


Generated at Fri Feb 09 00:35:35 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.