Scope
In order to address issues related to the poor performance of the Inventory app, the Technical Council decided during the meeting in October 2020 to implement an alternative Inventory search using Elastic search. The scope of the work was determined to include following deliverables:
Back-end:
- Sending update notification messages from inventory and source record storage (SRS)
- Providing Inventory and SRS APIs for fetching view for indexing by ids
- Extract common library for using it in other modules
- Build infrastructure necessary to support Elasticsearch
Front-end:
- Update Stripes Components to support new Search API
- Provide “switch” to allow using Elasticsearch or existing search
Infrastructure:
- Add Elasticsearch cluster to CI/CD and setup it on environments (k8s conf)
- Check configuration of existing Kafka cluster
In December 2020 it was determined that sending update notification messages from source record storage (SRS) are out of scope for Elasticsearch and became a separate feature (UXPROD-2791)
Delivered functionality
Back-end:
- Sending add/update/delete notification messages from Inventory
- Built Search APIs for searching and faceting
- Combined instances + holding + items into a single index
- Implemented re-index process for existing inventory DB
- Spring base implementation that supports:
- Up to five language-specific analyzers configured on the tenant level
- Near real-time inserts, updates and deletions
- Boolean operators (AND, OR, NOT)
- Nested search using brackets
- All or Any keyword search
- Exact phrase search
- Left-, right-hand truncation, wildcards searches in some fields
Front-end:
Due to rigid structure of the existing Inventorys app Search Component, in order to be able to present existing Inventory functionality and be able to provide an UI for the work implemented in the back end, we built an alternative UI (Inventory ES app) that allowed non-technical users compare performance between the existing search and the search powered by Elasticsearch. Inventory ES app introduced:
- New UI components for advanced search that include:
- auto-resized textbox,
- supported fields and operators auto-suggestion
- Boolean operators support
- Nested search using brackets
- New UI components for filters and facets
- Default results sort by ranking
- Preserved other non-search related Inventory app functionality
Infrastructure:
- Added Elasticsearch cluster to CI/CD and set it up on the reference environments
- Updated existing Kafka cluster configuration
- Introduced option of setting up performance testing environment in the community
As the result of the work, following search options and filters are supported:
Search options:
Instance | Holdings | Items |
Keyword search (title, contributor, identifier) | Keyword search (title, contributor, identifier) | Keyword search (title, contributor, identifier) |
Contributors | ISBN | Barcode |
Title (all) | ISSN | ISBN |
Identifiers (all) | Call Number | ISSN |
ISBN | Holdings HRID | Material type |
ISSN | Call Number | |
Subject | Item HRID | |
Instance HRID | ||
Instance UUID | ||
| Notes (public)* | ||
Electronic access (all fields) |
Filters and facets:
Effective location | Effective location (item) | Item Status |
Language | Holdings permanent location | Effective location |
Resource type | Suppress from discovery | Holdings permanent location |
Format* | Tags | Material type |
Mode of issuance | Suppress from discovery | |
Nature of content | Tags | |
Staff suppress | ||
Suppress from discovery | ||
Date created (from, to)* | ||
Date updated (from, to)* | ||
Source | ||
Tags |
*Back-end only
POC evaluation
The evaluation of the POC took place from April 5th to April 9th and it was conducted in the the Bugfest environment (~8 millions records) by eEight evaluators representing Chicago University (2 participants), Chicago University (2 participants), Duke University, Missouri State University (2 participants), Simmons University, EBSCO and Index Data.
Almost all evaluation happened trough UI. 75% of those who participate, found the implementation successful but all participants saw the room for improvements. The team addressed following issues raised:
Issue | Solution |
Noisy search results | Implemented searches supporting keyword “all” or “any” limiting the number of matches: MSEARCH-91 |
Expected results not found | All provided examples were related to the special characters in the Title that were searched using ASCII representation. The problem will be addressed in scope of MSEARCH-67 |
Bug in sorting by title | |
Support phrase search | |
Ranking refinement | Refinement of the default ranking system will require further analysis to be in the scope of a separate feature |
Discrepancy in saving UUIDs from Action menu | MSEARCH-93 and UISEES-58 |
UI enhancements and bugfixes | UISEES-47, UISEES-57, UISEES-61, UISEES-62, UISEES-48, UISEES-49 |
Those evaluators who deemed the POC a failure, provided following reasons:
- Expected to perform complex queries of multiple fields and across record types (including MARC fields).
- Expected a different UI more like a catalog or discovery system advanced search.
- Expected support for additional operators (not equal to, starts with, etc.).
- UI not user friendly.
- Preferred a simple left-anchored search than the provided relevancy ranking
Proposed next steps
- Use mod-search endpoints for searching
- Redesign Inventory UI Search component so that it can include new UI components created by POC, especially filters and facets
- Conduct usability study for advanced search textbox
- Conduct analysis of ranking refinements (weights and boosts)
- Conduct analysis of further search refinements
- Define and prioritize work for cross app/cross record types searches
- Define UI for cross app/cross record types searches
- Define requirements for cross-tenant searches