Skip to end of banner
Go to start of banner

Spike: Changing /titles/{titleId}?include=resources to a paged endpoint

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

MODKBEKBJ-591 - Getting issue details... STATUS

In scope of this spike we need to answer two questions:

  • how does current implementation handle titles with a large number of resources (a few thousand)?
  • can we implement a paged endpoint to get resources of a title that will work as /packages/{packageId}/resources endpoint?

Current implementation for titles

HoldingsIQ

Api '/titles/{titleId}?include=resources' of mod-kb-ebsco java uses following holdingsIQ api:

And there is no parameters for resources so they are returned in response every time:

Mod-kb-ebsco-java

According to the current implementation, resources always come from holdingsIQ and option "include=resources" is used only to include this resources in mod-kb-ebsco response and retrieve tags for related resources from database.

Current implementation for packages

'/packages/{packageId}/resources' endpoint of mod-kb ebsco java uses '/vendors/{providerId}/packages/{packageId}/titles'.

So titles here are treated as resources for packages and this holdingsIQ endpoint has 'count' and 'offset' parameters for pagination, so mod-kb-ebsco endpoint also provides pagination.

HoldingsIQ api:

Summary

HoldingsIQ doesn't provide endpoints to paginate titles' resources currently:

To implement full pagination for mod-kb-ebsco, some endpoint (f.e. /{custId}/titles/{titleId}/resources) must be implemented on HoldingsIQ side.

For now, all resources related to specific title will be returned in HoldingsIQ' response. And if we can't count on new api from HoldingsIQ side and pagination is really needed for mod-kb-ebsco the following approach could be implemented:

  1. Cache HoldingsIQ' title response(with some ttl and invalidation on PUT/POST/DELETE requests).
  2. Add pagination parameters to current '/titles/{titleId}?include=resources' api.
  3. Paginate over cached data(if cache exists, otherwise get data from HoldingsIQ, then cache response and return desired page).

The benefit we gain from this approach is that client traffic will be reduced if user don't need all the resources for title but we don't have in-memory data storage in folio currently and this approach could be resource-consuming for mod-kb-ebsco if we store cache in-memory(it will not even be shared between instances of applications if server run multiple mod-kb-ebsco intances) and bring even more problems if we store cache in postgres (the main problem - cache invalidation).

All in all, i would suggest to implement pagination only in following cases:

  1. HoldingsIQ provides api for pagination over titles' resources.
  2. We have in-memory data storage in folio (f.e. Redis).
  • No labels