[MODINVSTOR-395] Adding a Instance record with a few more contributors than normal throws "Values larger than 1/3 of a buffer page cannot be indexed" exception Created: 05/Nov/19  Updated: 15/Sep/20  Resolved: 06/Jan/20

Status: Closed
Project: mod-inventory-storage
Components: None
Affects versions: None
Fix versions: 18.2.1, 19.0.0

Type: Bug Priority: P2
Reporter: Theodor Tolstoy (One-Group.se) Assignee: Julian Ladisch
Resolution: Done Votes: 0
Labels: back-end, platform-backlog, q4-2019, q4-2019-bugfix, q4-2019-bugfix-release-created, q4-2019-bugtest, q4-2019-deployed-to-bugfest
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File Skärmavbild 2020-01-06 kl. 12.10.36.png     PNG File Skärmavbild 2020-01-07 kl. 17.02.40.png     PNG File Skärmavbild 2020-01-07 kl. 17.05.59.png     File modestly_sized_record.json    
Issue links:
Blocks
blocks FOLIO-2402 platform-related features and fixes f... Closed
is blocked by MODINVSTOR-418 Update RAML Module Builder (RMB) to 2... Closed
is blocked by RMB-498 Truncate b-tree string for 2712 index... Closed
Relates
relates to MODINVSTOR-390 Adding an Instance with a longer titl... Closed
Sprint: CP: sprint 79
Story Points: 1
Development Team: Core: Platform
Tester Assignee: Charlotte Whitt

 Description   

This behaviour has been observed both in Five Colleges tenant and in Snapshot-stable

Steps to reproduce:
1. Post the attached Instance record to /inventory/instances

What should happen:
Instance is created successfully

What actually happens:
An exception is thrown. Message from API:

ErrorMessage(fields=[(Severity, ERROR), (V, ERROR), (SQLSTATE, 54000), (Message, index row size 2984 exceeds maximum 2712 for index "instance_contributors_idx"), (Hint, Values larger than 1/3 of a buffer page cannot be indexed.
Consider a function index of an MD5 hash of the value, or use full text indexing.), (s, diku_mod_inventory_storage), (t, instance), (n, instance_contributors_idx), (File, nbtinsert.c), (Line, 584), (Routine, _bt_findinsertloc)])



 Comments   
Comment by Theodor Tolstoy (One-Group.se) [ 05/Nov/19 ]

Anya Do you keep track on FC-related issues in any way?

Comment by Theodor Tolstoy (One-Group.se) [ 05/Nov/19 ]

I got the hosting team to remove the index, and then it worked, but this needs to get fixed in the source.

Comment by Charlotte Whitt [ 06/Nov/19 ]

charles and Wayne Schneider - have you noticed this error in the ID sandbox environments?

Comment by charles [ 06/Nov/19 ]

Yes. I received the following three errors while loading Simmons instances last week:

"Youth's companion / edited by Lovell Thompson" (1393456)
{"message":"0a3011d7-9bd3-4821-ab09-ff92baea0dd9: ErrorMessage(fields=Map(s -> sim_daisy_mod_inventory_storage, n -> instance_contributors_idx, t -> instance, Line -> 684, Hint -> Values larger than 1/3 of a buffer page cannot be indexed.\nConsider a function index of an MD5 hash of the value, or use full text indexing., File -> nbtinsert.c, SQLSTATE -> 54000, Routine -> _bt_findinsertloc, V -> ERROR, Message -> index row size 2832 exceeds maximum 2712 for index \"instance_contributors_idx\", Severity -> ERROR))","level":"error","service":"user-service"}

"Norton recorded anthology of western music." (35326170)
{"message":"4adb0659-7dc8-4429-927e-d39350627f72: ErrorMessage(fields=Map(s -> sim_daisy_mod_inventory_storage, n -> instance_contributors_idx, t -> instance, Line -> 684, Hint -> Values larger than 1/3 of a buffer page cannot be indexed.\nConsider a function index of an MD5 hash of the value, or use full text indexing., File -> nbtinsert.c, SQLSTATE -> 54000, Routine -> _bt_findinsertloc, V -> ERROR, Message -> index row size 3384 exceeds maximum 2712 for index \"instance_contributors_idx\", Severity -> ERROR))","level":"error","service":"user-service"}

"American poetry : the twentieth century / advisory board, Robert Hass [and others]." (42290811)
{"message":"9f045e58-c625-4d30-936f-d7c3aa016954: ErrorMessage(fields=Map(s -> sim_daisy_mod_inventory_storage, n -> instance_contributors_idx, t -> instance, Line -> 684, Hint -> Values larger than 1/3 of a buffer page cannot be indexed.\nConsider a function index of an MD5 hash of the value, or use full text indexing., File -> nbtinsert.c, SQLSTATE -> 54000, Routine -> _bt_findinsertloc, V -> ERROR, Message -> index row size 4704 exceeds maximum 2712 for index \"instance_contributors_idx\", Severity -> ERROR))","level":"error","service":"user-service"}

This error goes away if we reduce the number of contributors to under 100.

Comment by Theodor Tolstoy (One-Group.se) [ 06/Nov/19 ]

Why did I not think of that...

Comment by Charlotte Whitt [ 06/Nov/19 ]

:-D I hear you Theodor Tolstoy (One-Group.se) - this bug is now filed in the top of the Core functional teams backlog.

Comment by Theodor Tolstoy (One-Group.se) [ 06/Nov/19 ]

Great Charlotte Whitt!

Comment by Theodor Tolstoy (One-Group.se) [ 07/Nov/19 ]

When you are figuring out a solution, that solution needs to reflect the fact that there are Scientific reports/Articles etc, especially in Physics, where the number of contributors can reach well over 3000 names. I am not joking. Three thousand.

Usually these reports are kept in the Institutional repositories of the Universities, but I am sure one or two will end up in FOLIO eventually, so we need to take that into consideration.

Comment by Cate Boerema (Inactive) [ 08/Nov/19 ]

Per Core Functional grooming, this bug belongs with Core Platform (CF is far less familiar with this stuff than CP). FYI Jakub Skoczen and Oleksii Popov since I think Jakub is OOO

Comment by Jakub Skoczen [ 19/Dec/19 ]

Theodor Tolstoy (One-Group.se) Cate Boerema this problem is fixed with RMB 29.1.4 and it will be shipped with mod-inventory-storate 18.2.0 – please review if the problem is gone on your end.

Comment by Charlotte Whitt [ 06/Jan/20 ]

Manual test in FOLIO Snapshot version @folio/inventory 1.13.1000758, using Chrome.

I created an instance record with extremely many contributors. All look good - https://folio-snapshot.aws.indexdata.com/inventory/view/a85d810e-717a-43b6-8d33-dd896e658483?query=goodbye&sort=Title. I'll close the ticket as solved.

Here's how the same record look in LIBRIS - https://libris.kb.se/bib/11871576?vw=full&tab3=marc

Comment by Charlotte Whitt [ 06/Jan/20 ]

Cate Boerema - the code can be released to BugFest environment.

CC: Holly Mistlebauer

Comment by Cate Boerema (Inactive) [ 06/Jan/20 ]

Awesome! Thanks for testing, Charlotte Whitt. I am just waiting for these two to be closed before requesting that https://github.com/folio-org/mod-inventory-storage/releases/tag/v18.2.1 be deployed to BugFest

MODINVSTOR-379 Closed and MODINVSTOR-139 Closed

Comment by Cate Boerema (Inactive) [ 06/Jan/20 ]

Deployed to bugfest: https://github.com/folio-org/mod-inventory-storage/releases/tag/v18.2.1

Comment by Charlotte Whitt [ 07/Jan/20 ]

Loaded .mrc file which previously failed caused to too many contributors. Todays test in Q4 2019 bugfest passed successfully, and all is fine. I have removed retest label.

Generated at Thu Feb 08 23:19:58 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.