[MODINVSTOR-650] Bugfest - will solve using Elasticsearch. Item segment. Filtering by Effective location and Item Status returns unexpected additional records Created: 17/Dec/20 Updated: 01/Nov/21 Resolved: 01/Nov/21 |
|
| Status: | Closed |
| Project: | mod-inventory-storage |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Bug | Priority: | P3 |
| Reporter: | Lisa Sjögren | Assignee: | Unassigned |
| Resolution: | Won't Do | Votes: | 0 |
| Labels: | elastic-search, support | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||
| Issue links: |
|
||||||||||||||||
| Sprint: | Prokopovych - Sprint 106, Prokopovych - Sprint 105 | ||||||||||||||||
| Development Team: | Prokopovych | ||||||||||||||||
| Affected Institution: |
Chalmers, Simmons
|
||||||||||||||||
| Tester Assignee: | Charlotte Whitt | ||||||||||||||||
| Description |
|
Overview: Customer Priority is set to be Low since Chalmers currently has a workaround using APIs for creating lists of missing items for each library. It would have been prioritized higher had it been possible to display item data – status + data identifying and locating the item – in the results list (which would make Inventory an attractive option for generating and working with precise lists of e.g. missing items). Steps to reproduce:
Expected result: Actual result: Additional comments: Marc Johnson wrote up following scope for the expectations:
Interested parties: Maryam Vardeh (Chalmers) |
| Comments |
| Comment by Kelly Drake [ 17/Dec/20 ] |
|
Charlotte Whitt - Is this something you are aware of? |
| Comment by Charlotte Whitt [ 17/Dec/20 ] |
|
Kelly Drake, Lisa Sjögren - no this is a new bug, which I have not heard about until now. This problem is not about that the item segment is selected but actually searching instances ... but I would need to look into this, and see if I can mimik the error in Bugfest Honeysuckle. |
| Comment by Charlotte Whitt [ 17/Dec/20 ] |
|
Bugfest Honeysuckle behaves as expected: Steps:
Lisa Sjögren Could you maybe try the same steps in the environment you are testing, and check if you still see any errors. Please notice, that the result list always will return a list of instances, and then you need to verify each result for which of the instance has the item status missing. (That's the current display, but only an interim solution). |
| Comment by Lisa Sjögren [ 17/Dec/20 ] |
|
Hi Charlotte Whitt! Sorry for not being clear about the version. I was testing in Snapshot Stable. |
| Comment by Charlotte Whitt [ 17/Dec/20 ] |
|
Hi Lisa Sjögren you're screen dump looks a bit old. Please notice that FOLIO Snapshot Stable is not maintained since 9/29/2020 - https://dashboard.folio.ebsco.com/ Try FOLIO Snapshot instead: https://folio-snapshot.dev.folio.org/ |
| Comment by Lisa Sjögren [ 17/Dec/20 ] |
|
Charlotte Whitt Are you sure that, when you did your search, there existed a title where AcqMono BO's item was available but another location's item was missing? When I tried your search in Honeysuckle I did initially only get the three results, but once I added a missing item for another location to an instance that has an available AcqMono BO item, that title showed up in the result list. SInce AcqMono BO item is not missing, I would not expect that to be returned when I filter on Missing and AcqMono BO. |
| Comment by Charlotte Whitt [ 17/Dec/20 ] |
|
And you did clear your searches between the two searches? That's important in order to make sure there is nothing in the catch spoking? |
| Comment by Charlotte Whitt [ 17/Dec/20 ] |
|
It works for me in Bugfest Honeysuckle. Now I don't get the missing item with 'new' location. Do you want me to call you, and I can share my screen. |
| Comment by Lisa Sjögren [ 17/Dec/20 ] |
|
Thanks for the Snapshot link, Charlotte Whitt! I'll use that one in the future. It looks like this one and Stable have the same okapi URL though – folio-snapshot-okapi.dev.folio.org – so I would assume that the API behaviour would have been the same. |
| Comment by Charlotte Whitt [ 17/Dec/20 ] |
|
Yes, and the title Semantic web, has also a Missing item and location Main library right. Then that's the expected result. That's the way it works right now, until we can get it refined, right. |
| Comment by Lisa Sjögren [ 17/Dec/20 ] |
|
Charlotte Whitt I'm in another meeting, but I think we are speaking past each other. The issue is not with the Semantic Web, but The Girl on the Train. Main Library's copy of that title is not missing. Where the use case is to find all missing items at a certain library, titles where only another library's copy is missing are not relevant. |
| Comment by Charlotte Whitt [ 17/Dec/20 ] |
|
Lisa Sjögren - re.
Says nothing about the build and the current version of the software. Here you need to check: https://folio-snapshot.dev.folio.org/settings/about |
| Comment by Charlotte Whitt [ 17/Dec/20 ] |
okay, let me check |
| Comment by Charlotte Whitt [ 17/Dec/20 ] |
|
Lisa Sjögren - I know the explenaition now: I tried a new search in Bugfest Honeysuckle (and Bugfest Snapshot - is the same). The search do correctly search: All items missing and effective location, e.g. acq bind https://bugfest-honeysuckle.folio.ebsco.com/inventory/view/22f98c5a-d4ea-433f-81fe-7859c5a9ea9d?filters=itemStatus.Missing%2CeffectiveLocation.fac5de34-26ee-456d-86b1-f04fdf680d65&segment=items&sort=title*https://bugfest-honeysuckle.folio.ebsco.com/ But because one specific instance has two items
That's a bug, and we must look into how to fix that. |
| Comment by Lisa Sjögren [ 18/Dec/20 ] |
|
Great! That is exactly what I was trying to say. I do think it's worth noting that the query
{{okapi-url}}/inventory/items?limit=100&query=(status.name=="Missing" and effectiveLocationId="fcd64ce1-6995-48f0-840e-89ffa2288371") sortby title
does return the expected results, in case a library that runs into this decides to do the search outside the UI. |
| Comment by Charlotte Whitt [ 18/Dec/20 ] |
|
Cool, sounds all good. I'll see if I can get it groomed at today's Core Functional meeting. |
| Comment by Zak Burke [ 18/Dec/20 ] |
|
This looks to me like a back-end issue. e.g. the query generated for an item query when both "Item status" and "Effective location" filters are present contains an "and" clause like this: GET https://folio-snapshot-okapi.dev.folio.org/inventory/instances?limit=100&query=%28item.status.name%3D%3D%22Available%22%20and%20item.effectiveLocationId%3D%3D%22fcd64ce1-6995-48f0-840e-89ffa2288371%22%29%20sortby%20title { "instances" : [ { "@context" : "http://folio-snapshot-okapi.dev.folio.org/inventory/instances/context", "id" : "69640328-788e-43fc-9c3c-af39e243f3b7", "hrid" : "inst000000000001", "source" : "FOLIO", "title" : "ABA Journal", "alternativeTitles" : [ ], "editions" : [ ], "series" : [ ], "identifiers" : [ { "identifierTypeId" : "913300b2-03ed-469a-8179-c1092c991227", "value" : "0747-0088" }, { "identifierTypeId" : "c858e4f2-2b6b-4385-842b-60732ee14abb", "value" : "84641839" } ], "contributors" : [ ], "subjects" : [ ], "classifications" : [ ], "publication" : [ { "publisher" : "American Bar Association", "place" : "Chicago, Ill.", "dateOfPublication" : "1915-1983", "role" : null } ], "publicationFrequency" : [ ], "publicationRange" : [ ], "electronicAccess" : [ ], "instanceTypeId" : "6312d172-f0cf-40f6-b27d-9fa8feaf332f", "instanceFormatIds" : [ ], "physicalDescriptions" : [ ], "languages" : [ ], "notes" : [ ], "discoverySuppress" : false, "statisticalCodeIds" : [ ], "statusUpdatedDate" : "2020-12-18T01:43:49.852+0000", "metadata" : { "createdDate" : "2020-12-18T01:43:49.852+00:00", "createdByUserId" : null, "updatedDate" : "2020-12-18T01:43:49.852+00:00", "updatedByUserId" : null }, "tags" : { "tagList" : [ ] }, "natureOfContentTermIds" : [ "0abeee3d-8ad2-4b04-92ff-221b4fce1075" ], "links" : { "self" : "http://folio-snapshot-okapi.dev.folio.org/inventory/instances/69640328-788e-43fc-9c3c-af39e243f3b7" } }, { "@context" : "http://folio-snapshot-okapi.dev.folio.org/inventory/instances/context", "id" : "7fbd5d84-62d1-44c6-9c45-6cb173998bbd", "hrid" : "inst000000000006", "source" : "FOLIO", "title" : "Bridget Jones's Baby: the diaries", "alternativeTitles" : [ ], "editions" : [ "First American Edition" ], "series" : [ ], "identifiers" : [ { "identifierTypeId" : "5d164f4b-0b15-4e42-ae75-cfcf85318ad9", "value" : "ocn956625961" } ], "contributors" : [ { "contributorNameTypeId" : "2b94c631-fca9-4892-a730-03ee529ffe2a", "name" : "Fielding, Helen", "contributorTypeId" : null, "contributorTypeText" : null, "primary" : null } ], "subjects" : [ "Jones, Bridget", "Pregnant women", "England", "Humorous fiction", "Diary fiction" ], "classifications" : [ { "classificationNumber" : "PR6056.I4588", "classificationTypeId" : "ce176ace-a53e-4b4d-aa89-725ed7b2edac" } ], "publication" : [ { "publisher" : "Alfred A. Knopf", "place" : "New York", "dateOfPublication" : "2016", "role" : null } ], "publicationFrequency" : [ "A frequency description" ], "publicationRange" : [ "A publication range" ], "electronicAccess" : [ { "uri" : "http://www.folio.org/", "linkText" : "Electronic resource (audio streaming)", "materialsSpecification" : "Novel", "publicNote" : "Access to audio file", "relationshipId" : null } ], "instanceTypeId" : "6312d172-f0cf-40f6-b27d-9fa8feaf332f", "instanceFormatIds" : [ ], "physicalDescriptions" : [ "219 pages ; 20 cm." ], "languages" : [ "eng" ], "notes" : [ { "instanceNoteTypeId" : "6a2533a7-4de2-4e64-8466-074c2fa9308c", "note" : "Bridget Jones finds herself unexpectedly pregnant at the eleventh hour. However, her joyful pregnancy is dominated by one crucial but awkward question --who is the father? Could it be honorable, decent, notable human rights lawyer, Mark Darcy? Or, is it charming, witty, and totally despicable, Daniel Cleaver?", "staffOnly" : false } ], "previouslyHeld" : true, "staffSuppress" : true, "discoverySuppress" : true, "statisticalCodeIds" : [ ], "statusUpdatedDate" : "2020-12-18T01:43:49.701+0000", "metadata" : { "createdDate" : "2020-12-18T01:43:49.701+00:00", "createdByUserId" : null, "updatedDate" : "2020-12-18T01:43:49.701+00:00", "updatedByUserId" : null }, "tags" : { "tagList" : [ ] }, "natureOfContentTermIds" : [ ], "links" : { "self" : "http://folio-snapshot-okapi.dev.folio.org/inventory/instances/7fbd5d84-62d1-44c6-9c45-6cb173998bbd" } }, { "@context" : "http://folio-snapshot-okapi.dev.folio.org/inventory/instances/context", "id" : "eb5dc7cf-5c81-4172-b2e3-f0cc30dc1c6c", "hrid" : "in00000000005", "source" : "FOLIO", "title" : "New test title", "alternativeTitles" : [ ], "editions" : [ ], "series" : [ ], "identifiers" : [ ], "contributors" : [ ], "subjects" : [ ], "classifications" : [ ], "publication" : [ ], "publicationFrequency" : [ ], "publicationRange" : [ ], "electronicAccess" : [ ], "instanceTypeId" : "a2c91e87-6bab-44d6-8adb-1fd02481fc4f", "instanceFormatIds" : [ ], "physicalDescriptions" : [ ], "languages" : [ ], "notes" : [ ], "previouslyHeld" : false, "staffSuppress" : false, "discoverySuppress" : false, "statisticalCodeIds" : [ ], "statusUpdatedDate" : "2020-12-18T02:14:17.089+0000", "metadata" : { "createdDate" : "2020-12-18T02:14:17.089+00:00", "createdByUserId" : "a5a9f6c5-30c1-57d6-b4a8-0fe0d40bb319", "updatedDate" : "2020-12-18T02:14:17.089+00:00", "updatedByUserId" : "a5a9f6c5-30c1-57d6-b4a8-0fe0d40bb319" }, "tags" : { "tagList" : [ ] }, "natureOfContentTermIds" : [ ], "links" : { "self" : "http://folio-snapshot-okapi.dev.folio.org/inventory/instances/eb5dc7cf-5c81-4172-b2e3-f0cc30dc1c6c" } }, { "@context" : "http://folio-snapshot-okapi.dev.folio.org/inventory/instances/context", "id" : "cf23adf0-61ba-4887-bf82-956c4aae2260", "hrid" : "inst000000000024", "source" : "FOLIO", "title" : "Temeraire", "alternativeTitles" : [ ], "editions" : [ ], "series" : [ ], "identifiers" : [ { "identifierTypeId" : "8261054f-be78-422d-bd51-4ed9f33c3422", "value" : "1447294130" }, { "identifierTypeId" : "8261054f-be78-422d-bd51-4ed9f33c3422", "value" : "9781447294130" } ], "contributors" : [ { "contributorNameTypeId" : "2b94c631-fca9-4892-a730-03ee529ffe2a", "name" : "Novik, Naomi", "contributorTypeId" : null, "contributorTypeText" : null, "primary" : null } ], "subjects" : [ ], "classifications" : [ ], "publication" : [ ], "publicationFrequency" : [ ], "publicationRange" : [ ], "electronicAccess" : [ ], "instanceTypeId" : "6312d172-f0cf-40f6-b27d-9fa8feaf332f", "instanceFormatIds" : [ ], "physicalDescriptions" : [ ], "languages" : [ ], "notes" : [ ], "discoverySuppress" : false, "statisticalCodeIds" : [ ], "statusUpdatedDate" : "2020-12-18T01:43:50.025+0000", "metadata" : { "createdDate" : "2020-12-18T01:43:50.025+00:00", "createdByUserId" : null, "updatedDate" : "2020-12-18T01:43:50.025+00:00", "updatedByUserId" : null }, "tags" : { "tagList" : [ ] }, "natureOfContentTermIds" : [ ], "links" : { "self" : "http://folio-snapshot-okapi.dev.folio.org/inventory/instances/cf23adf0-61ba-4887-bf82-956c4aae2260" } }, { "@context" : "http://folio-snapshot-okapi.dev.folio.org/inventory/instances/context", "id" : "30fcc8e7-a019-43f4-b642-2edc389f4501", "hrid" : "inst000000000003", "source" : "FOLIO", "title" : "The American Journal of Medicine", "alternativeTitles" : [ { "alternativeTitleTypeId" : null, "alternativeTitle" : "The American journal of medicine (online)" }, { "alternativeTitleTypeId" : null, "alternativeTitle" : "Am. J. med" }, { "alternativeTitleTypeId" : null, "alternativeTitle" : "Green journal" } ], "editions" : [ ], "series" : [ ], "identifiers" : [ { "identifierTypeId" : "593b78cb-32f3-44d1-ba8c-63fd5e6989e6", "value" : "AJMEAZ" }, { "identifierTypeId" : "913300b2-03ed-469a-8179-c1092c991227", "value" : "0002-9343" }, { "identifierTypeId" : "c858e4f2-2b6b-4385-842b-60732ee14abb", "value" : "med49002270" } ], "contributors" : [ ], "subjects" : [ "Clinical medicine-Periodicals", "Medicine", "Geneeskunde" ], "classifications" : [ { "classificationNumber" : "RC60 .A5", "classificationTypeId" : "ce176ace-a53e-4b4d-aa89-725ed7b2edac" }, { "classificationNumber" : "W1 AM493", "classificationTypeId" : "a7f4d03f-b0d8-496c-aebf-4e9cdb678200" } ], "publication" : [ { "publisher" : "Dun-Donnelley Pub. Co. ", "place" : "New York", "dateOfPublication" : "1946-", "role" : null } ], "publicationFrequency" : [ ], "publicationRange" : [ ], "electronicAccess" : [ ], "instanceTypeId" : "6312d172-f0cf-40f6-b27d-9fa8feaf332f", "instanceFormatIds" : [ ], "physicalDescriptions" : [ "v., ill. 27 cm." ], "languages" : [ "eng" ], "notes" : [ { "instanceNoteTypeId" : "6a2533a7-4de2-4e64-8466-074c2fa9308c", "note" : "Print subscription cancelled by Dec. 2016.", "staffOnly" : false }, { "instanceNoteTypeId" : "6a2533a7-4de2-4e64-8466-074c2fa9308c", "note" : "May 1988-: A Yorke medical. Also known as the Green journal", "staffOnly" : false }, { "instanceNoteTypeId" : "6a2533a7-4de2-4e64-8466-074c2fa9308c", "note" : "Publisher: Excerpta Medica, 2008-; New York, NY : Elsevier Inc. 2013-", "staffOnly" : false }, { "instanceNoteTypeId" : "6a2533a7-4de2-4e64-8466-074c2fa9308c", "note" : "Supplements issued irregularly, 1982.", "staffOnly" : false }, { "instanceNoteTypeId" : "6a2533a7-4de2-4e64-8466-074c2fa9308c", "note" : "Official journal of the Association of Professors of Medicine 2005-", "staffOnly" : false }, { "instanceNoteTypeId" : "6a2533a7-4de2-4e64-8466-074c2fa9308c", "note" : "Indexed quinquennially in: American journal of medicine 5 year cumulative index", "staffOnly" : false } ], "discoverySuppress" : false, "statisticalCodeIds" : [ ], "statusUpdatedDate" : "2020-12-18T01:43:49.645+0000", "metadata" : { "createdDate" : "2020-12-18T01:43:49.645+00:00", "createdByUserId" : null, "updatedDate" : "2020-12-18T01:43:49.645+00:00", "updatedByUserId" : null }, "tags" : { "tagList" : [ ] }, "natureOfContentTermIds" : [ "0abeee3d-8ad2-4b04-92ff-221b4fce1075" ], "links" : { "self" : "http://folio-snapshot-okapi.dev.folio.org/inventory/instances/30fcc8e7-a019-43f4-b642-2edc389f4501" } }, { "@context" : "http://folio-snapshot-okapi.dev.folio.org/inventory/instances/context", "id" : "f31a36de-fcf8-44f9-87ef-a55d06ad21ae", "hrid" : "inst000000000012", "source" : "FOLIO", "title" : "The Girl on the Train", "alternativeTitles" : [ { "alternativeTitleTypeId" : null, "alternativeTitle" : "First alternative title" }, { "alternativeTitleTypeId" : null, "alternativeTitle" : "Second alternative title" } ], "editions" : [ ], "series" : [ ], "identifiers" : [ { "identifierTypeId" : "7f907515-a1bf-4513-8a38-92e1a07c539d", "value" : "B01LO7PJOE" } ], "contributors" : [ { "contributorNameTypeId" : "2e48e713-17f3-4c13-a9f8-23845bb210aa", "name" : "Creator A", "contributorTypeId" : null, "contributorTypeText" : null, "primary" : null }, { "contributorNameTypeId" : "e8b311a6-3b21-43f2-a269-dd9310cb2d0a", "name" : "Creator B", "contributorTypeId" : null, "contributorTypeText" : null, "primary" : null } ], "subjects" : [ ], "classifications" : [ ], "publication" : [ ], "publicationFrequency" : [ ], "publicationRange" : [ ], "electronicAccess" : [ ], "instanceTypeId" : "6312d172-f0cf-40f6-b27d-9fa8feaf332f", "instanceFormatIds" : [ ], "physicalDescriptions" : [ ], "languages" : [ ], "notes" : [ ], "discoverySuppress" : false, "statisticalCodeIds" : [ ], "statusUpdatedDate" : "2020-12-18T01:43:49.683+0000", "metadata" : { "createdDate" : "2020-12-18T01:43:49.683+00:00", "createdByUserId" : null, "updatedDate" : "2020-12-18T01:43:49.683+00:00", "updatedByUserId" : null }, "tags" : { "tagList" : [ ] }, "natureOfContentTermIds" : [ "44cd89f3-2e76-469f-a955-cc57cb9e0395" ], "links" : { "self" : "http://folio-snapshot-okapi.dev.folio.org/inventory/instances/f31a36de-fcf8-44f9-87ef-a55d06ad21ae" } } ], "totalRecords" : 6 } |
| Comment by Kelly Drake [ 04/Jan/21 ] |
|
Support SIG - this issue would appear to effect all production libraries. Should it be higher priority? |
| Comment by Charlotte Whitt [ 04/Jan/21 ] |
|
Yes, Kelly Drake - the ranking was based on Chalmers comment re. their work around, but do agree, and will bump it up. |
| Comment by Marc Johnson [ 05/Jan/21 ] |
|
Holly Mistlebauer Charlotte Whitt This has been placed in Sprint 105 for the Core Functional team. Does that mean that the team should be actively picking up this issue? If so, is it more important than the upgrading of modules to RAML Module Builder 32 for 2021 R1? |
| Comment by Marc Johnson [ 05/Jan/21 ] |
|
I agree with Zak Burke's statement above that this is likely a back end issue. It was estimated on the 18th December (at a session I wasn't present for) yet the issue was not moved to a back end project. Can anyone provide me context as to whether the estimate was based upon it being front end or back end work? |
| Comment by Marc Johnson [ 05/Jan/21 ] |
|
If I were to speculate (and we would want to investigate and confirm this), my thought would be that this CQL translates to find any instance where there is an item that is missing and an item whose effective location is the chosen location, however these might not be the same item. If this turns out to be the case, this would be a side effect of how multi-record CQL is translated to SQL and would likely need Core Platform involvement to resolve. |
| Comment by Charlotte Whitt [ 05/Jan/21 ] |
|
I think you are right Marc Johnson (the comment above) - should we ask if Julian Ladisch could help out here? |
| Comment by Holly Mistlebauer [ 05/Jan/21 ] |
|
Marc Johnson: Chalmers has a workaround, so I would not describe this as a higher priority than upgrading modules for RAML. |
| Comment by Marc Johnson [ 08/Jan/21 ] |
Should it be in the current sprint for developers to pick up? |
| Comment by Charlotte Whitt [ 08/Jan/21 ] |
|
Yes, good point Marc Johnson. This is very important to get fixed, while it affects all libraries in production, and of course also all libraries testing their sandbox environments, preparing to go live some time this year. |
| Comment by Marc Johnson [ 08/Jan/21 ] |
|
Charlotte Whitt Holly Mistlebauer
Does that mean it should be in the current sprint and take priority over other back end work for 2021 R1? |
| Comment by Charlotte Whitt [ 11/Jan/21 ] |
|
Marc Johnson - it means that if at all possible we should fit it into the current sprint, or the following sprint. And we definitely should solve it for 2021 R1. |
| Comment by Marc Johnson [ 11/Jan/21 ] |
|
Charlotte Whitt Thank you for providing more context for the prioritisation of this.
What do you want to do? Should we leave this in the current sprint? If the cause of this turns out to be the SQL generated from the CQL query, there might be a significant lead time to resolve this (assuming we can). If the resolution requires changes to RAML Module Builder it is unlikely that these will be completed in time for the freeze on the 15th January. |
| Comment by Julian Ladisch [ 11/Jan/21 ] |
|
Marc is correct that this CQL query
{{okapi-url}}/inventory/instances?limit=100&query=(item.status.name=="Missing" and item.effectiveLocationId="fcd64ce1-6995-48f0-840e-89ffa2288371") sortby title
is translated into this this database query:
If "item.*" is used several times it can refer to different items of the instance. If multiple filters should apply to the same item use the /inventory/items API. Lisa already provided the solution:
{{okapi-url}}/inventory/items?limit=100&query=(status.name=="Missing" and effectiveLocationId="fcd64ce1-6995-48f0-840e-89ffa2288371") sortby title
|
| Comment by Marc Johnson [ 11/Jan/21 ] |
Thanks, I was just setting out to investigate and replicate this.
As convenient a solution as that would be, I don't think that would meet the expected behaviour. As I understand it, all searches in the inventory UI are intended to find instances not items, which I think would mean this approach isn't appropriate as that API responds with items. Charlotte Whitt Holly Mistlebauer Zak Burke Is this the case? Is it a sensible understanding that the behaviour we are trying to achieve is to find all instances which are associated to an item with a given status and effective location? |
| Comment by Charlotte Whitt [ 11/Jan/21 ] |
|
Hi Marc Johnson - yes, that's correct: Charlotte Whitt Holly Mistlebauer Zak Burke Is this the case? Is it a sensible understanding that the behaviour we are trying to achieve is to find all instances which are associated to an item with a given status and effective location? We have labelled this ticket as a SPIKE, while this work might reveal that this is a general issue when we are performing a search as a combinations of filters. |
| Comment by Charlotte Whitt [ 11/Jan/21 ] |
|
Marc Johnson - should I assign the ticket to you? |
| Comment by Marc Johnson [ 11/Jan/21 ] |
I believe Julian Ladisch has confirmed that is the case in his comment above. Julian Ladisch Would it be reasonable to generalise what you said above that any combination of search clauses e.g. filters that refer to any record type other than instances e.g. items, holdings records will result in the behaviour where each is considered a separate match on these related records?
I don't think so, I just unassigned myself from it after Julian Ladisch confirmed my hypothesis. Julian Ladisch has outlined that this is intended behaviour in the tooling (RAML Module Builder) that mod-inventory-storage uses. Without changes to that tooling, I don't think it is likely we would be able to meet the expected behaviour. What are you expecting the next steps to be for this work? cc: Jakub Skoczen |
| Comment by Lisa Sjögren [ 11/Jan/21 ] |
While that is indeed how inventory works, I think it's worth noting that this "intention" might not be shared by the user. As I pointed out, Chalmers expected to be able to use the combination item segment + effective location filter + status to identify items that belong to a certain location and have a certain status – to a librarian that is a search intended to find items, although Inventory (here, with some difficulty) translates it into a search for instances. The need to search for specific items in Inventory was also highlighted (so to speak) in https://folio-org.atlassian.net/browse/UIIN-906. |
| Comment by Marc Johnson [ 11/Jan/21 ] |
Agreed, that may well be the case. My understanding comes from Charlotte Whitt's comment above and what we have been asked to build (it saddens me when that doesn't match what folks actually need). Alas, I get to have very little direct contact with potential or actual users of the system :-/ Given Charlotte Whitt answer above and your suggestion that behaviour might not be what is wanted, I think this becomes more of a user needs and experience conversation. Does that make sense? And fit with your understanding? |
| Comment by Charlotte Whitt [ 11/Jan/21 ] |
|
hmm Lisa Sjögren - I'm sure you are aware that the current solution, displaying instance in the result list is the thin thread implementation using the MCL component. Eventually (multiple fingers crossed Having the search term being highlighted not just for barcodes, but as a general solution, is definitely a vision we share. But that work was paused due to Elastic Search. When we did
Charlotte Whitt added a comment - 12/Dec/19 4:36 PM We could use search on barcode as a SPIKE for this functionality. CC: Siska Humlesjö |
| Comment by Charlotte Whitt [ 11/Jan/21 ] |
|
Marc Johnson - please see my comment to Lisa above. Lisa Sjögren comments are only to be understood as 'statements' re. where we eventually want to go - and this work is both agreed upon by the MM-SIG and definitely expected to happen long term. |
| Comment by Lisa Sjögren [ 11/Jan/21 ] |
|
So, this is what I originally wrote in the description and what I understand the overarching, bordering on philosophical, issue to be:
So I absolutely think this belongs in a greater discussion about what you should expect to be able to do in Inventory. The Customer priority for Chalmers on this is Low, so they do not expect it to be fixed soon. |
| Comment by Charlotte Whitt [ 11/Jan/21 ] |
|
Lisa Sjögren - maybe it's low for Chalmers due to you have a work around, and are can search in the Union Catalogue. We have a Support SIG meeting later today - and I'll loop in Kelly Drake on this. |
| Comment by Charlotte Whitt [ 11/Jan/21 ] |
|
Lisa Sjögren - I'd be happy anytime to give you a walk through of the planned work for Inventory, Inventory Search, and Inventory UI |
| Comment by Lisa Sjögren [ 11/Jan/21 ] |
|
I'm wondering if there needs to be several Customer priority fields, to reflect the fact that different customers have different priorities (which may also be different from development priority). Is this something you have discussed in the Support SIG, Kelly Drake?
Yes, and I believe that could open up a world of exciting uses for Inventory search – at which point this issue would become much more tangible. |
| Comment by Marc Johnson [ 11/Jan/21 ] |
It seems to me that we are talking about two different things
Is this a reasonable statement? Is this issue specifically about the former? |
| Comment by Charlotte Whitt [ 11/Jan/21 ] |
|
Marc Johnson - yes this issue is solemnly about:
The future need, and when work on these features are to be expected, we can not say anything more concrete about until we get an updated road map for 2021 and 2022. |
| Comment by Marc Johnson [ 11/Jan/21 ] |
|
Charlotte Whitt Thanks for confirming my understanding. Based upon Julian Ladisch comments above, we need the involvement of the Core Platform, as (if my understanding is right) resolving this with the current tooling means changing shared behaviour that has been stated to work as expected. Jakub Skoczen please can you advise how we move this work forward. Are folks thinking we need to either change the shared behaviour or introduce a new mechanism? I think there could be some challenges in this, as the search instances?query=(item.status.name=="Missing" and item.effectiveLocationId="fcd64ce1-6995-48f0-840e-89ffa2288371") is sufficiently ambiguous that it could legitimately mean either:
I'm not sure how we would disambiguate this. Are proposing changing the interpretation of CQL involving multiple record types to have to understand that all of the criteria for a specific record type are intended to be used together. |
| Comment by Julian Ladisch [ 11/Jan/21 ] |
|
The same ambiguity can happen when querying both holding and item: instances?query=(item.status.name=="Missing" and holdingsRecords.permanentLocationId="fcd64ce1-6995-48f0-840e-89ffa2288371") In current CQL implementation this means:
|
| Comment by Marc Johnson [ 11/Jan/21 ] |
Agreed. And it gets even more ambiguous, in that the item might not be part of that specific holdings record. |
| Comment by Julian Ladisch [ 11/Jan/21 ] |
|
No back-end change is needed if the front-end runs this query
{{okapi-url}}/inventory/items?limit=100&query=(status.name=="Missing" and effectiveLocationId="fcd64ce1-6995-48f0-840e-89ffa2288371") sortby title
and displays the result. |
| Comment by Marc Johnson [ 11/Jan/21 ] |
Agreed. However I believe Charlotte Whitt has confirmed that this is an unsuitable resolution for this issue as the search still has to be for instances (for the moment).
How would that query present instances, it is a query for items? |
| Comment by Julian Ladisch [ 11/Jan/21 ] |
|
The GET /inventory/items API fetches the instance title and merges it into the item record as a read-only field: https://s3.amazonaws.com/foliodocs/api/mod-inventory/inventory.html#inventory_items_get |
| Comment by Marc Johnson [ 11/Jan/21 ] |
It does. I don't think that means this API can be used to present the same search results as the instance API. Is that what you are suggesting? Also, I thought that only properties on the primary record type could be used for sorting. Have I remembered that correctly? |
| Comment by Julian Ladisch [ 11/Jan/21 ] |
|
This is correct for RMB CQL in RMB based storage modules.
{{okapi-url}}/inventory/items?limit=100&query=(status.name=="Missing" and effectiveLocationId="fcd64ce1-6995-48f0-840e-89ffa2288371")
This avoids sorting on the "title" field that doesn't exists at the time when running the CQL query against the mod-inventory-storage item API. |
| Comment by Julian Ladisch [ 11/Jan/21 ] |
The item record contains the holdingsId. The UI can fetch the holding record that contains the instanceId. The UI can fetch the instance record. This should be sufficient. |
| Comment by Zak Burke [ 12/Jan/21 ] |
|
Julian Ladisch, yes, I think latency is a concern here, though I'm not certain how moving the instance-id up into the item record would address that. That would allow us to generate an un-ordered list of IDs, but then we'd have to write a second query with those IDs to actually get at the data we wanted. This is problematic for multiple reasons:
In other words, we could change mod-inventory as you suggest, but it would (a) require substantial UI work in order to actually function and (b) doing so would go against established UI patterns of loading records in batches and providing a "Load more" button for additional batches which would (c) compound they latency problem. What I am hearing is basically this: while the end-user expectation is that multiple filters of are joined with "AND" (this is how instance-level filtering works), this is not possible for holdings-level and item-level filters applied to the inventory/instances endpoint (effectively, multiple filters are joined with "OR"). |
| Comment by Julian Ladisch [ 12/Jan/21 ] |
|
Yes, it gets that difficult if the items should be sorted by instance title. |
| Comment by Marc Johnson [ 12/Jan/21 ] |
That is basically my understanding. Julian Ladisch stated that this behaviour is the intended behaviour provided by the shared tooling (RAML Module Builder), which means that we would need to change that across the board to change it. |
| Comment by Marc Johnson [ 12/Jan/21 ] |
Do you mean instance rather than items, as the search presents instances not items? I understand you are trying to come up with solutions that don't involve changing that shared underlying tooling and appreciate the endeavour. My concern is that the solutions presented have compromises that when discussed previously with product owners, have been unacceptable e.g. disabling sorting search results by the properties of the instance presented. |
| Comment by Julian Ladisch [ 12/Jan/21 ] |
I mean that the search run against the item endpoint as proposed by Lisa is a solution that can be implemented without any back-end development. However, it requires that title sort is disabled. If the result list should show the instance title there will be some latency. Running the query against the item endpoint is the natural way when all sub-expressions should match the same item. This solution requires some changes in the UI but has low development effort, it can be released earlier than other solutions. It's up to the implementing organizations and the PO to weight the pros and cons. |
| Comment by Marc Johnson [ 12/Jan/21 ] |
Apologies for my misunderstanding.
Charlotte Whitt Lisa Sjögren is it acceptable that sorting be disabled for some search results in the inventory UI? Based upon previous conversations, my understanding is that this limitation is not acceptable.
I believe Zak Burke has already raised concerns about the front end work involved in this approach
How would this work if the search also included properties from the instance or the holdings record? As I understand it, we would encounter the same challenge in reverse. |
| Comment by Julian Ladisch [ 12/Jan/21 ] |
|
Querying against the item endpoint does NOT match different items at the same time. It matches one item only. Adding subqueries with holdingsRecords.* and instance.* to the query matches the holding and instance the item belongs to but does NOT allow to match other items. This is the reason why Lisa and I point to this approach. |
| Comment by Zak Burke [ 12/Jan/21 ] |
|
I have strong reservations about what I'm about to suggest, but in the interest of moving this conversation forward rather than debating whether it's necessary to be able to sort the results of a search in a library's catalog: If we point the holdings and items queries at the holdings and items endpoints, respectively, in order to have multiple filters combine in a way that meets user expectations, we could process that output through a secondary query to the instances endpoint, as is done to handle client-side joins in ui-agreements. md331 noted in the PR where he implemented this pattern that it's cleaner than fetching directly via GET mutators in lifecycle methods. The downside, as I noted above, is that this doesn't accommodate "large" result sets (> ~40 records). Maaaaybe we could figure out how to show a "narrow your filter; too many results" warning if the initial query returns too many rows. Or maybe we suck it up and use those lifecycle methods, along with a query that hangs off the results of a holdings/items query, to retrieve all related instance records (up to some threshold, 100? 1000?) in a loop. More gross, but also more powerful, at least from a search results point of view. To be clear, I don't love this, but we have some patterns for it (even if they're really anti-patterns sprung from the need to handle client-side joins) so maybe it's worth a shot. i.e. having already gone through the Gedankenexperiment of how to make this happen, maybe that hard part is mostly done. |
| Comment by Charlotte Whitt [ 13/Jan/21 ] |
|
Thank you so much Zak Burke for bringing the conversation forward, and pointing at a solution I have only one comment and that is, that a result set as > ~40 records, is probably to expect, so we probably from the very beginning need to go for the more powerful solution from a search result point of view. CC: Holly Mistlebauer lew235 and Magda Zacharska (if this is to be addressed by Elastic Search) |
| Comment by Marc Johnson [ 13/Jan/21 ] |
I want to check my understanding of this statement. Are you saying that search results of more than 40 records should be expected? And that would make the solution that Zak Burke proposed an unacceptably limited solution for this? |
| Comment by Charlotte Whitt [ 13/Jan/21 ] |
Yes that's correct, and it would be an unacceptable limited solution. |
| Comment by Marc Johnson [ 13/Jan/21 ] |
|
Charlotte Whitt Lisa Sjögren Holly Mistlebauer I'm getting the sense that both Julian Ladisch and Zak Burke's proposed solutions are too limited to be acceptable. My understanding of the expectations for this are are as follows:
Is this a reasonable summary of the expectations? |
| Comment by Charlotte Whitt [ 13/Jan/21 ] |
|
Thanks Marc Johnson - yes that's the expectations. |
| Comment by Julian Ladisch [ 13/Jan/21 ] |
|
Are there use cases where different items of one instance should be queried? item.effectiveLocationId="fcd64ce1-6995-48f0-840e-89ffa2288371" AND item.effectiveLocationId="8188760d-2641-44ca-a1f2-c111810201db" If this is the case then the current implementation is still useful and this Jira should be changed to be a feature request (and not a bug). |
| Comment by Charlotte Whitt [ 13/Jan/21 ] |
|
Hi Julian Ladisch - this ticket is about:
The behavior documented here is clearly a bug, while an unwanted instance (The girl on the train) > holdings > items is returned. |
| Comment by Julian Ladisch [ 13/Jan/21 ] |
|
Charlotte Whitt You do not answer my question, sorry that I did not explain it clearly enough. Do I understand correctly that there will never be a multi-item use case where searching different items of the same instance is a useful feature like the example I posted above, and that the RMB CQL multi-item feature is completely wrong and should be thrown away? |
| Comment by Charlotte Whitt [ 13/Jan/21 ] |
|
Hi Julian Ladisch - you are right I did not answer your question, and that is because what you describe is a different scenario, and do not really have anything to do with this bug ticket. But I can write up another ticket for your question re multi-item use case - no problem. Real live Use case: |
| Comment by Marc Johnson [ 13/Jan/21 ] |
Does that mean that there are circumstances in the inventory UI where folks want to find instances that have an item that matches only some of the criteria (e.g. only one of two filters specified)? |
| Comment by Charlotte Whitt [ 13/Jan/21 ] |
|
This is three different searches, right?
Marc Johnson and Julian Ladisch - I expect that
When we have solved: |
| Comment by Zak Burke [ 14/Jan/21 ] |
I think this should be "Find me all results for Items which have item status missing AND (location is Main library or location is Annex library). i.e. that multiple filters are AND-ed together (any matching must match all filters) while multiple values within a filter are OR-ed together (matching a filter means matching any one of its selected values). This is the behavior, for example, in ui-users when ticking the boxes for Status - Active, Patron group - Faculty, and Patron group - Staff. Selected users must be active AND must belong to faculty or staff. I understand ui-users is a simpler use case (there is no join from instance to holdings to item) but from the point of view of a choosing the "Item" pill and then choosing some filters, I see why folks would expect the same behavior (filter-values OR-ed, multiple filters AND-ed). |
| Comment by Marc Johnson [ 01/Feb/21 ] |
|
Holly Mistlebauer Lisa Sjögren Charlotte Whitt This issue is in the Core Functional team's current sprint. What are folks expectations about progressing this? From what I can tell from the conversations above. The current tooling (RAML Module Builder's implementation of multi-record type queries) won't allow us to satisfy this kind of search (except for the potential workaround to use the items API which sacrifices sorting and may have challenges when combined with other record types). I think that leaves us with the following choices:
Jakub Skoczen Julian Ladisch is that a reasonable summary of the situation? |
| Comment by Lisa Sjögren [ 01/Feb/21 ] |
|
As I've stated above, Chalmers who reported this to EBSCO does not consider this issue urgent (thus their Customer priority: Low), since for their intents and purposes Inventory search, as long as it only returns instances, does not lend itself very well to the type of searches that would be the most affected by this. What I expect, considering the nature of the problem, is a solid long term solution which is well anchored in the architecture of Inventory. I cannot really speak to what that solution should be. |
| Comment by Marc Johnson [ 01/Feb/21 ] |
I apologise for asking a question that you had answered previously I had gotten the impression, from the current priority (P2, meaning it needs resolving this release), the current customer priority (Important) and the involved conversation, that there was a need to move this issue forward soon. Charlotte Whitt Holly Mistlebauer Kelly Drake Please could you advise on this? It seems that EBSCO / Chalmers (based upon what Lisa Sjögren just said) consider this a low priority piece of work, yet it has been classified as an urgent and important piece of work. |
| Comment by Lisa Sjögren [ 01/Feb/21 ] |
|
Absolutely no need to apologize Marc Johnson! Lots of comments to keep track of here. 🙂 The current Customer Priority value was added by Charlotte Whitt to reflect other libraries' needs. |
| Comment by Charlotte Whitt [ 01/Feb/21 ] |
|
Yes, this ticket has been discussed at the past 3-4 Support SIG meetings ... and the talks have been lengthy. At today's Support SIG meeting we decided that a talk with relevant MM-SIG SMEs, Elastic Search people and the developers who have been involved in the current Search in Inventory should be held, and then we must review our options, and decide for how to move along. I'll try if I can find a suitable meeting time this week. I'll send out a doodle poll ASAP. |
| Comment by Charlotte Whitt [ 02/Feb/21 ] |
|
Zak Burke - I had to test using Query Search: Following query search gives the expected result (test data in FOLIO Snapshot as of today) I do (of corse) understand, that eventually all other item states should also be listed with NOT Order closed, NOT Awaiting delivery, etc. etc. If I do the filtering on Item status: Missing, and Effective location: Main library, then I (with my test data) get the not wanted instance title: The girl on the train (as we already have documented above): |
| Comment by Julian Ladisch [ 02/Feb/21 ] |
|
This query doesn't solve the problem.
This is a different query. If you change the status of the "The Girl on the Train" barcode=7031 item from "Available" to some other status the query will also return the instance "The Girl on the Train". |
| Comment by Charlotte Whitt [ 03/Feb/21 ] |
|
Julian Ladisch - I did also write:
So I'm fully aware that the query can be more complicated – but my approach do solve the issue here. Julian Ladisch I'm no super expert on CQL, so there might be away to express: that only item status Missing is wanted here. That would make the CQL much prettier. |
| Comment by Holly Mistlebauer [ 03/Feb/21 ] |
|
Charlotte Whitt: If this issue doesn't need to be fixed for R1 2021, or isn't ready to be fixed in R1 2021, we need to change the status from P2 to P3. P2 is reserved for bugs that must be fixed in the next release. Thanks... |
| Comment by Charlotte Whitt [ 03/Feb/21 ] |
|
Okay, I can lower it to P3. |
| Comment by Julian Ladisch [ 10/Aug/21 ] |
|
If I search on an item attribute and an instance has both a matching item and a non-matching item should both been shown or only the matching item? |
| Comment by Charlotte Whitt [ 11/Aug/21 ] |
|
Hi Julian Ladisch - we should not invest more effort in fixing this in the current search solution. We are planning for doing the swap of Inventory using PostgreSQL to be using Elastic Search. I have added `Elastic-search` as a label - so we keep track, and make sure that it's not being dropped under the radar. |
| Comment by Charlotte Whitt [ 11/Aug/21 ] |
|
Anya - should I remove the `support` label? |
| Comment by Julian Ladisch [ 11/Aug/21 ] |
|
Charlotte Whitt: The PostgreSQL tables will continue to exists, Elastic Search doesn't replace them, Elastic Search adds an additional search option. Adding an additional front-end view where only matching items are shown is needed regardless of the underlying search machine. |
| Comment by Marc Johnson [ 28/Oct/21 ] |
|
This is currently marked as needing to be fixed for 2021 R3 Bug Fix. The title suggests that the intention was to resolve this in the ES based search rather than regular inventory. As
|
| Comment by Marc Johnson [ 01/Nov/21 ] |
|
Superseded by ES based work. Charlotte Whitt Lisa Sjögren please reopen if this is incorrect. |