/
ARCH-35 Technical approach for Browsing by call number type functionality

ARCH-35 Technical approach for Browsing by call number type functionality

Mar 30, 2023

Overview

(Approach addresses feature https://folio-org.atlassian.net/browse/UXPROD-3569)

It's required to implement items filtering by Call Number Types (LC, NLM, Dewey Decimal, SuDoc and Local Call Numbers) represented as aggregations (facets). Currently Browse functionality is implemented Browsing by Call Number with one available facet Effective location (item).

Current UI/UX

Expected UI/UX

(NOTE: in the recent requirements there's no need to build aggregation view, instead needs to proceed with a dropdown list. See [UIIN-2358] Add new browse options to limit browse by call number type - FOLIO Issue Tracker )

Context on Call Number Types - Technical Designs and Decisions - FOLIO Wiki

Scope

Back-End implementation (considering OpenSearch (mod-search) or (mod-inventory-storage) Postgres schema change, with corresponding amendments on mod-quick-marc)

Front-End representation ui-quick-marc

Potential performance impact analysis

Current state

Browsing is done by call number, refined by effective location facet filtering. Extra capabilities of filtering by call number type have not been implemented yet. 

Nested objects mappings haven't been introduced in any of existing indexes (implies implementation from scratch).

Another concern is on generation of shelving order, as browsing functionality is performed on the first 10 characters from the whole call number that can causes inconsistent search results.

Least efforts approach

To add facet abilities on call number types with codebase changes on UI and BE.

  • Back-end:

    • mod-search: add extra mapping for nested objects relationship on instance → items → call number entry

    • mod-search: API for nested aggregations and queries with corresponding CQL query change (to distinct handling strategy as for nested elements)

  • UI:

    • ui-inventory: Add UI elements facets (drop-down items are considered as another option) for browsing by Call Number Types  

Example of request response:

Facet request callnumber
// response on facet search request with value "callNumberType" // /search/instances/facets?query=...&facet=callNumberType { "facets": { "callNumberTypes": { "values": [ { "id": "NLM", "totalRecords": 1500 }, { "id": "Local Call Number", "totalRecords": 579 }, { "id": "Dewey Decimal", "totalRecords": 789 } ], "totalRecords": 2868 } } }

Technical impediments / implications / concerns

1) There are concerns on introducing another mapping (nested) for items (ones belong to holdings and holdings itself belong to instance) in terms of potential performance impact on indexing and querying by nested query as Elasticsearch implicitly creates separate documents.

Prototype of probation nested objects mapping and querying with measurements of performance, space usage. (link)

2) Used marc4j.jar (v2.9.2) library implements only two kinds of Call Number Types: LC and Dewey. No SuDoc, NLM and others specified. As a solution within ItemEffectiveShelvingOrderProcessor#getFieldValue class getValidShelfKey could be generated separately

3) It's possible to encounter multiple call number types per instance: when there are multiple holdings with items that have different call number types.

4) CQL syntax processing does not support distinction in nested queries or not. To be clarified users expectations.

5) Local call number types (with source "local") are not supposed to be distinguished among each other during indexing and filling a new field typedCallNumber.

Solutions

Solution 1: Application-side join

Steps to describe this approach on trying to Browse by call number with type specified:

  1. Check amount of instances by call number type

  2. Get matched instances by call number

  3. Filter by call number type on application side

  4. In case if after filtering amount of left instances is less than expected and less than total amount, continue search in two directions: backward and forward from anchor with corresponding offsets, filtering by call number type. 

 

Short description

Comments

 

Short description

Comments

Pros

  1. No need to reimplement indexing, mapping

  2. No impact on other search / browsing functionality

  3. No complex queries to Elasticsearch

  1. Initial query is done by call number that is a regular query to mod-search (Elasticsearch implied)

  2. Due to index retains the same there is no impact on other functionality tied to mod-search interaction

  3. No nested or other types of queries introduced as its regular query with post handling on application side

Cons

  1. Potential excessive queries

  2. Time consuming operation in some cases

  3. Could not return valid results eventually

  1. In case if found lots of records that exceed paging limit needs to filter page-by-page and re-query data again from Elasticsearch to get the next portion of records and filter ones by call number type

  2. In case of millions records indexed and resource limits (CPU, RAM) it can perform slowly due to necessity to handle all records found by call number to filter by call number type

  3. In some cases process can take a while (in comparison to nested queries for example) and paging results could be complicated (needs to store all filtered instances in memory or repeat queries and handling for the next pages)

Cons to be analyzed more.

Solution 2: Nested objects, nested queries and nested aggregation

In case of nested objects introduction there will be possibility to browse by both datasets (call numbers and call number types) as ones will belong to implicitly indexed document (e.g. item that is nested to holding and correspondingly holding is nested to instance - or similar mappings to be considered if required). Nested Objects concept explained - Technical Designs and Decisions - FOLIO Wiki

 

Short description

Comment

Pros

  1. Quick and effective search by nested queries and nested aggregations

  2. No need to handle found results and extra filtering afterwards once records found

  1. As indexing is done in another way that implies separate documents for nested objects querying of ones is performed effectively

  2. Results of found instances are sufficient in case of regular nested query or nested aggregation

Cons

  1. Direct impact on another search / browse functionality due to indexing / mapping reimplementation

  2. Complexity of queries increased due to nested queries introduction

  3. More space for index allocation

  4. Potentially slower indexing 

  1. Introducing of nested structure for inner objects within instance (like items and holdings) will lead to non-working queries for the whole search functionality of instance, holdings, items. Needs to do refinements in implementation of ones

  2. Queries could be shown in examples or in Elasticsearch docs

  3. Nested structure will take more space as implicitly nested objects are separate documents. Actual measurements to be provided

  4. Assumption needs to be assured by actual performance testing

Trade-off

could be additional mapping instead of substitution to keep back-compatibility for all potentially affected searches 

there could be added another sibling mapping for items like (items_nested) with corresponding nested mapping(s) for the entry

effectiveCallNumberComponents

Actual impact on indexing performance (and ways to speed up one) to be analyzed more, however according to what's probated within the prototype running - impact on indexing performance is far from critical.

Implementation details (Solution 2)

ui-inventory

 

 

For ui-inventory module the next changes are implied:

  • altering request(s) for browsing (by nested queries) with optional call number type parameter inclusion. Particularly request semantic implies to stay the same, except extra parameter adding for refined request.

  • (facets requirements are considered as skipped https://folio-org.atlassian.net/browse/UIIN-2358) For facet filtering nested bucket aggregation query is implied. There are two ways of how it could be performed: 

    • 1st is to count all instances grouped by call number types

    • 2nd is to break results of browsing by search term on groups of call number types (note, only results for current, previous and next pages are counted. If implement aggregation functionality needs to define if it's required to count in aggregations all instance records or just affine around browsing value).