Query request where offset value > totalRecords returns 0 reords even if actual number of records > totalRecords

Description

Steps to reproduce

(It should be able to reproduce them using only item-storage/items, but this is the actual scenario I had.)

  1. With /item-storage/items: Using consecutive queries and paging through the results with a limit 1000 and an offset that increments by 1000 for each query, fetch all existing items where (effectiveLocationId=="f4c5afee-261a-43d1-9708-cc123aaba50a")

  2. With /inventory/items: Using consecutive queries and paging through the results with a limit 100 and an offset that increments by 100 for each query, fetch all existing items where (effectiveLocationId=="f4c5afee-261a-43d1-9708-cc123aaba50a")

Expected result

All items matching the criterium are returned. mod-invetory and mod-inventory-storage return the same number of items.

Actual result

mod-inventory-storage returns 11 351 items, mod-inventory only 11 000. Not only is the latter number too low, but it's suspiciously even. N.B. that these numbers come from counting the actual records returned, not from the totalRecords value. The totalRecords value returned is, in both cases, 11 075.

Investigation

Knowing that mod-inventory-storage returns 11 351 items matching this criterium, let's make this request in Postman:

It should return records 11 100-11 200. Alas, instead it returns this:

Could this have something to do with the fact that our offset value (11 100) is higher than the estimated totalRecords (11 075).

Let's test that idea by lowering the offset to 11 000, and changing the limit to 400, so that the range encompasses the actual number of records. Lo and behold, this returns 351 items. (totalRecords is still 11075.)

Interpretation

If the offset value is higher than totalRecords, the query request will return 0 items even if there are actually more records.

Implications

In many cases this probably wouldn't be noticeable, but if the limit used (in order to avoid too long query string errors) is less than the difference

actual number of records - totalRecords, how do you get to those last records?
For example this query

If I set the limit to a higher value the request will return a 500 error.

totalRecords is 11075, but the actual number of records is 11351. (I know the actual number of records because I tried the same query but with a higher limit in item-storage.)
The first 111 requests, with offsets between 0 and 11000 return 100 items each. But then I get to

which returns an empty items list.

CSP Request Details

None

CSP Rejection Details

None

Potential Workaround

None

Attachments

5

Checklist

hide

TestRail: Results

Activity

Show:

Julian Ladisch November 30, 2020 at 9:54 AM

For mod-inventory-storage the fix has been shipped with Q2 2020 hotfix 3 (or some earlier hotfix): https://github.com/folio-org/platform-complete/blob/q2-2020-hotfix-3/okapi-install.json#L11
Other storage modules have shipped the RMB-684 fix (= upgrade to RMB >= 30.2.5) with a different Goldenrod hotfix release or with Honeysuckle only.

Cate Boerema November 30, 2020 at 9:00 AM

, it looks like the fix for this (RMB-684) went out with the first Goldenrod hotifx

Lisa Sjögren November 26, 2020 at 4:41 PM

was this included in Goldenrod, or will Chalmers get it Honeysuckle?

it would be great if we could verify this by comparing to a number from the database.

Cate Boerema September 9, 2020 at 11:11 AM

Just changing this from In Bugfix Review to In Code Review because that is the status that maps to "Developer Done (Awaiting Deployment)" on the Chalmers Kanban board (https://folio-org.atlassian.net/secure/RapidBoard.jspa?rapidView=132)

, let me know if this is not right.

Julian Ladisch September 9, 2020 at 10:31 AM

The fix RMB-684 has been released as RMB 30.2.5 that was released with mod-inventory-storage 19.3.2 (MODINVSTOR-559).

Done

Details

Assignee

Reporter

Priority

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs
Created April 20, 2020 at 8:13 AM
Updated November 30, 2020 at 9:54 AM
Resolved September 9, 2020 at 10:32 AM
TestRail: Cases
TestRail: Runs