Search - using PostgreSQL (all releases up to Juniper)

Search and filters

FOLIO’s current technical approach for search is using PostgreSQL, JSONB columns, CQL and RAML Module Builder. Using Elastic Search as a long term solution is being explored Autumn 2020 - Spring 2021 - see more. 

In inventory, we have implemented segmented control search where it's possible to toggle between searching on an Instance, Holdings or individual Item level. Search in the Instance segment supports search on bibliographic data. Search in the Holdings and Item segments are a combination of key data from the Instance record, combined with holdings and item specific data elements. The filters reflect each of the individual segments and include search on effective location.

This is work in development, but as of end of January the following search  and filter options are present in FOLIO Snapshot:

Search and filter options can be combined.

When searching in a filter, the search typed will be highlighted. E.g.:

If more than 100 hits are retrieved, the first 100 will load, and then you will see a "Load more" button at the bottom of the page.

When clicked, it will display the next 100:

From the FOLIO version Juniper onwards the concept of loading more records has been redesigned. You can navigate through the records in steps of 100 by clicking the "next/previous" buttons at the bottom of the page.

There is also a Query search function, which can search across records on all data elements. See below.

Search using Query language (Instance, Holdings, Item segment)

See list of property names in Query Search - Inventory Metadata Elements, or in the API Documentation in the /instance-storage/instances POST, /holdings-storage/holdings POST and /item-storage/items POST.

See more CQL tips written up by Julian Ladisch Explain CQL string matching. For experts: RAML Module Builder CQL documentation.

See: Tips: Use Query Search in Inventory for export MARC records from FOLIO

See list of defined supporting database indexes: db_scripts/schema.json – Skip to the "tableName": "instance", "tableName": "holdings_record" or "tableName": "item" section and look into the "uniqueIndex", "index", and "fullTextIndex" subsections. The id field is automatically indexed and not listed in schema.json.

See misc. test of search performance in Bug fest Goldenrod (https://bugfest-goldenrod.folio.ebsco.com/) in Document Inventory Search Response time (Q3 Timebox)

Tips: When searching by an ID, in the Query search it's better to use "==" as it uses the B-tree index to search, whereas a CQL query using "=" will result in a full text search, which is not efficient for searching when there is already an index on the ID.

Examples on CQL syntax (WIP)

Boolean search within records and across instance/holdings/item:

Search queries that currently do not seem to work

Item search

Sort

Change the sort field in the URL.

Example:

https://folio-snapshot.dev.folio.org/inventory?query=*&sort=title becomes
https://folio-snapshot.dev.folio.org/inventory?query=*&sort=publicationPeriod.start

We need to use publicationPeriod.start because there can be multiple dateOfPublication entries in the publication array.

Retrieve UUIDs

For some CQL queries you need the UUID of the value, because you cannot search for the actual term. To do this, you can use the developer tools that are built into your browser. In Chrome, press F12 to open the developer tools and switch to the "network" tab. Clear out any previous data by pressing "clear" and refresh the Folio page. Please note, that you'll first have to open a detail instance record in the third pane. After the page has loaded, you should look in the network data for the record UUID. Select it to see the record data in JSON format. 

More documentation on search indexes (from a developer's perspective)

Following list of resources is gathered by Julian Ladisch:

Existing documentation:

• https://dev.folio.org/faqs/explain-cql/
• https://dev.folio.org/faqs/explain-database-schema/
• https://github.com/folio-org/raml-module-builder#the-post-tenant-api about likeIndex, ginIndex, uniqueIndex, index, fullTextIndex
• https://github.com/folio-org/raml-module-builder#cql-relations with several sections about CQL
• https://github.com/folio-org/raml-module-builder#cql2pgjson-multi-field-index with several sections about CQL2PgJSON indexes
• https://github.com/folio-org/mod-inventory-storage/blob/master/src/main/resources/templates/db_scripts/schema.json is the schema.json of mod-inventory-storage including several indexes for instance, holdings_record and item fields.

All fields can be indexed, simply add an index entry to schema.json on GitHub. The indexing is the same for all institutions and a change requires a new release of the module.
Currently institutions cannot customize indexing at runtime and cannot have different indexing. GBV libraries have extensive indexing customization in the inventory of their current system (OCLC LBS) and will likely request this for FOLIO. Index customization use cases for inventory should be discussed in the metadata management sig.