Page Comparison

...

Scope:

In October of 2020, during order to address issues related to the poor performance of the Inventory app, the Technical Council meeting it was decided to look into Elasticsearch as a possible alternative for the existing Inventory search due to its poor performance and inability to provide exact count of records matching search criteria.The development team has been formed in December 2020 and a PO was assigned to the team.  The decided during the meeting in October 2020 to implement an alternative  Inventory search using Elastic search.   The scope of the work was limited to what was limited to the back-end work due to the limited time for delivery.  The details of what was discussed are here

Scope:

...

determined to include following deliverables: 

Back-end:

• Sending update notification messages from inventory and source record storage (SRS)
• Providing Inventory and SRS APIs for fetching view for indexing by ids
• Extract common library for using it in other modules
• Build infrastructure necessary to support ElasticsearchDeliverables

Front-

...

end:

• Update Stripes Components to support new Search API
• Provide “switch” to allow using Elasticsearch or existing searchDeliverables - infrastructure

Infrastructure:

• Add Elasticsearch cluster to CI/CD and setup it on environments (k8s conf)
• Check configuration of existing Kafka cluster

Sending In December 2020 it was determined that sending update notification messages from source record storage (SRS) - determined as are out of scope for Elasticsearch  and became a separate feature (UXPROD-2791)

Delivered functionality:

Back-end:

• Sending add/update/delete notification messages from Inventory
• Built Search APIs for searching and faceting
• Combined instances + holding + items into a single index
• Implemented reindex re-index process for existing inventory DBFront-endBuilt UI for alternate Inventory search app that represents searching capabilities of Elasticsearch
• Spring base implementation that supports:
  • Up to five language-specific analyzers configured on the tenant level
  • Near real-time inserts, updates and deletions
  • Boolean operators (AND, OR, NOT)
  • Nested search using brackets
  • All or Any keyword search
  • Exact phrase search
  • Left-, right-hand truncation, wildcards searches in some fields

 Front-end:

Due to rigid structure of the existing Inventorys app Search Component, in order to be able to present existing Inventory functionality and be able to provide an UI for the work implemented in the back end, we built an alternative UI (Inventory ES app) that allowed non-technical users compare performance between the existing search and the search powered by Elasticsearch.  Inventory ES app introduced:

• New UI components for advanced search that include:
  • auto-resized textbox,
  • supported fields and operators auto-suggestion
  •  Boolean operators support
  • Nested search using brackets
• New UI components for filters and facets
• Default results sort by ranking
• Preserved other non-search related Inventory app functionality

Infrastructure:

• Added Elasticsearch cluster to CI/CD and set it up on the reference environments
• Updated existing Kafka cluster configuration
• Introduced option of setting up performance testing environment in the community

As the result of the work, following search options and filters are supported:

Search options:

Filters and facets:

*Back-end only

POC evaluation:

The evaluation of the POC took place from April 5th to April 9th and it was conducted in the the Bugfest environment (~8 millions records) by eEight evaluators representing Chicago University (2 participants), Chicago University (2 participants), Duke University, Missouri State University (2 participants), Simmons University, EBSCO and Index Data.  

Almost all evaluation happened trough UI.  75% of those who participate, found the implementation successful but all participants saw the room for improvements. The team addressed following issues raised:

Those evaluators who deemed the POC a failure, provided following reasons:

• Expected to perform complex queries of multiple fields and across record types (including MARC fields).
• Expected a different UI more like a catalog or discovery system advanced search.
• Expected support for additional operators (not equal to, starts with, etc.).
• UI not user friendly.
• Preferred a simple left-anchored search than the provided relevancy ranking

Proposed next steps:

• Use mod-search endpoints for searching
• Redesign Inventory UI Search component so that it can include new UI components created by POC, especially filters and facets
• Conduct usability study for advanced search textbox
• Conduct analysis of ranking refinements (weights and boosts)
• Conduct analysis of further search refinements
• Define and prioritize work for cross app/cross record types searches
• Define UI for cross app/cross record types searches
• Define requirements for cross-tenant searches

Additional links:

• TC Meeting on October 14, 2020
• UXPROD-2791 SRS MARC Query API
• Search – Technical documentation
• POC Evaluation survey
• Responses Detailed responses to raised issues
• POC overview for MM SIG (presentation)
• POC overview for MM SIG (recording)
• 110 -111 Sprints Demo