[FOLIO-869] Aggregator module Created: 29/Sep/17  Updated: 18/Jan/19

Status: Open
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: New Feature Priority: P3
Reporter: Heikki Levanto Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: 4 hours, 30 minutes
Original estimate: Not Specified

Sprint:
Development Team: Core: Platform

 Description   

We discussed a simple aggregator module in Cph. Summarize design here.



 Comments   
Comment by Heikki Levanto [ 29/Sep/17 ]

The idea is to build something not unlike graphQL, but simplified to the bare minimum that we can get away with. Main points:

  • Read-only access to any Okapi module
  • Schemaless entry point that takes a Json structure that defines what is to be fetched, and what is to be expanded
  • Multi-level expansion, but on fixed paths (userid -> user record -> loan records -> item records) But not (user id -> proxy record -> another user id -> another user record -> proxy record -> ... aritrary depth user record)

In some cases it makes sense to extend the main record with the expanded data, for example when getting a loan record that has a userid, put a whole user record next to it. In other cases less so, for example when getting a list of notes about an item, each of them has a userid who created the note, but these are often the same. Then it is better to return a list of notes with just the userids, and a separate map from userids to user records.

We need to be able to expand in both directions: Starting from a loan record, find the userid, and get the matching user. Or, starting from a user record, take the userid, and find all loans that the user has (and probably expand the items in those too)

Comment by Heikki Levanto [ 29/Sep/17 ]

Over the lunch we agreed that the way is to post the aggregation rules to this module, which will then create endpoints that provide the aggregated data, for example notes-with-user-names. The aggregation rules can be part of a moduleDescriptor, either in the back-end module (mod-notes can provide the notes-with-user-names), or in the UI module, depending what the UI needs.

Comment by Heikki Levanto [ 29/Sep/17 ]

Simple example: endpoint notes-with-users gets notes, and expands the userid into user records. The /notes endpoint returns an object that contains a notes array of note records.

endpoint: /notes-with-users
get: /notes
aggregate {
  each: notes as {note}
  get: /users/{note.metadata.creatorUid}
  into {note}.userdata
}
Comment by Heikki Levanto [ 29/Sep/17 ]

A more complex example:

endpoint: /loans-with-detail?{query},{limit},{maxlocs}
get: /loans?query=$query&limit=$limit
aggregate {
  each: loans[*] as {loan}
  get: /items/{loan.item}
  into: {loan}.itemdetails
  aggregate {
    each: {loan}.itemdetails.locations[*] as {loc}
    get: /locations/{loc.locationcode}?limit={maxlocs}
    into {loc}.locationdetails
  }
}

aggregate {
  join: loans[*].userid with " OR " as {userlist}
  get: /users?query=userid any {userlist}
  into: users[]
  aggregate {
    each: users as {usr}
    get: /perms/?query=user={usr.userid}
    into: {usr}.perms
  }
}

This iterates over all loans, and expands the items in them by looking up locations. Then it takes all the userids, joins them into one list, does a lookup on that, and then iterates over each user looking up permissions too. The join functionality may not be needed in version 1 of the aggregator, but it is here as an example of the directions we may expand the definition language.

Of course the language will be proper Json, we don't want to parse our own string representation of it.

Comment by Heikki Levanto [ 29/Sep/17 ]

The first implementation will be a stand-alone module with only in-memory data, to get a proof of concept up and running quickly. Then one that persists its data, and Okapi inegration, so we can define aggregation endpoints in ModuleDescriptors, using the redirect facility, with a new header that tells what the original path was.

Generated at Thu Feb 08 23:08:55 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.