Addition to Spike: MODKBEKBJ-459 Ability to filter titles by package(s) + titles/selection status + tags/access status types

MODKBEKBJ-459 - Getting issue details... STATUS

Problem

There is no way for the user to filter to view the desired titles inside the package that s/he wants.

Goals

What do we currently have

There is no RM API endpoint exists so satisfy expected output as one request. In this case, we can analyze what we can be used in the existing system and what changes are required.

I picked out two actions

get a list of unique packages by title name
return data (selection status, title name, package name, publisher name, and type) to UI

To search titles regardless of the package, mod-kb-ebsco-java uses existing GET /{custid}/titles endpoint, which can return package information. This means in return, we have a set of unique titles (by title id) but not packages. So, this endpoint can hardly be used to serve the required behavior without changes.

To get a list of unique packages from endpoint mentioned above we have to perform additional actions:

get the full list of the titles - RM API return batches 100 entries max, so, in case of 1000 titles we have to perform 10 requests to get all of them
find unique package names

and return them on UI.

What can be done

The current implementation does not support filtering titles by packages at one time. Several options can be considered as solutions to provide the desired output.

On RM API side

1. Retrieve information from RM API using separate endpoint

This option assumes creation of a new endpoint on the RM API side with the option to filter titles by package name.

Requirement	Can be done?	Explanation
Filter title results by package(s)	✅	should be supported by the RM API endpoint
This filter must work in tandem with searching titles	✅	should be supported by the RM API endpoint
This filter must work in tandem with selection status	✅	should be supported by the RM API endpoint
NICE TO HAVE - filter must work in tandem with access status types filter	❌	not supported by an existing mod-kb-ebsco-java implementation
NICE TO HAVE - filter must work in tandem with tags filter	❌
Sort the results	✅	should be supported by the RM API endpoint
Paging results	✅	should be supported by the RM API endpoint

2. Additional query parameter to the existing RM API endpoint.

RM API already has an endpoint(GET /{custid}/titles), which can retrieve titles regardless of the package with the list of following filters

Endpoint	/{custid}/titles
Available filters	search	<search_word>
	searchfield	titlename, publisher, isxn, subject and zdbid.
	selection	all, selected, notselected, orderedthroughebsco
	resourcetype	all, journal, newsletter, report, proceedings, website, newspaper, unspecified, book, bookseries, database, thesisdissertation, streamingaudio, streamingvideo, and audiobook
	searchtype	any, contains, exactmatch, beginswith, proximity, exactphrase and advanced

that could be updated by an additional filter to search by package name. Example:

Proposed filter	packagename

The filter name 'packagename' should not be treated as the desired name. It is an example name and should be changed to a reasonable one.

Requirement	Can be done?	Explanation
Filter title results by package(s)	✅	should be supported by the RM API endpoint
This filter must work in tandem with searching titles	✅	should be supported by the RM API endpoint
This filter must work in tandem with selection status	✅	supported by the RM API endpoint
NICE TO HAVE - filter must work in tandem with access status types filter	❌	not supported by an existing mod-kb-ebsco-java implementation
NICE TO HAVE - filter must work in tandem with tags filter	❌
Sort the results	✅	supported by the RM API endpoint
Paging results	✅	supported by the RM API endpoint

Cons

There is no possibility to have this functionality done in the nearest future because of the workload of the RM API team.

If we stick to this option we have two options to consider:

wait until RM API will have the capacity to include this work into the scope
propose to implement needed functionality by the Spitfire team.

On kb-ebsco-java side

The other possibility is to perform filtering on the kb-ebsco-java side.

1. Use stored holdings information

As another option, we can think about using holdings that are load every 5 days for each KB credentials. The information available to get from RM API much more extensive compared to stored info in the holdings table. Let's compare the information stored in the module and provided by RM API.

According to the screenshot at the top of the page, the desired information includes the following fields:

Required info	Stored in holdings	Can be added to holdings info
Selection status	❌	❌
Title name	✅
Package name	❌	✅
Publisher name	✅
Publication/Resource type	✅

Detailed information about holding fields can be found here - GET /{custid}/reports/holdings/deltas/{deltaReportId}

Requirement	Can be done?	Explanation
Filter title results by package(s)	✅	possible because needed data stored in holdings table
This filter must work in tandem with searching titles	✅	possible because needed data stored in holdings table
This filter must work in tandem with selection status	❌	holdings do not have information about the selection status
NICE TO HAVE - filter must work in tandem with access status types filter	❌	not supported by an existing implementation
NICE TO HAVE - filter must work in tandem with tags filter	❌	not supported by an existing implementation
Sort the results	✅	as we operate with holdings table, maintained in the module the sorting and paging will be done on the DB side by SQL query.
Paging results	✅

Cons

view outdated data set - holdings table updates every 5 days for each KB credentials. So, there is a possible situation that the user may see the information about titles that is no longer exists.
RM API returns only selected holdings.
selection status filter can not be applied because the holdings table does not have this information
in case of data absence, we should have "second plan"

2. Load titles data and process it in mod-kb-ebsco-java module

This is the most undesirable option. As it was described in "What do we currently have" section module should perform the work for getting unique package names by loading full titles list. That leads us to the option of storing loaded title list wheater in the cache or in the database.

It means that the package filter will be considered as a secondary filter after all titles loaded we are able to segregate the package list.

Requirement	Can be done?	Explanation
Filter title results by package(s)	✅	search by the package as a secondary filter
This filter must work in tandem with searching titles	✅	search as a first filter
This filter must work in tandem with selection status	✅	supported by the RM API endpoint
NICE TO HAVE - filter must work in tandem with access status types filter	❌	not supported by an existing implementation
NICE TO HAVE - filter must work in tandem with tags filter	❌	not supported by an existing implementation
Sort the results	❌	hard to maintain because of the complicated solution
Paging results	❌	hard to maintain because of the complicated solution

Cons

two separate calls to the same RM API endpoint to get
- page with titles - the result list from RM API is pageable
- list of packages - separate call to load all packages that contain searched title
time cost for loading titles and selecting unique package names
storing packages and titles in mod-kb-ebsco-java module(in cache or db table) - leads to performance issues

Example: below we have a title search example with 10_000 results. To have a unique package list for this request, we have to load all records and store them somewhere, for instance in cache. Then the user decides to change search request and in return also have ~10_000 results and we also have to store it.