[MODEXPW-255] Bulk edit: 10528 upstream timed out (110: Connection timed out) Error Created: 15/Sep/22  Updated: 07/Nov/22  Resolved: 27/Oct/22

Status: Closed
Project: mod-data-export-worker
Components: None
Affects versions: None
Fix versions: 2.0.0

Type: Bug Priority: P2
Reporter: Olga Kondratenko Assignee: Siarhei Charniak
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified
Environment:

https://ptf-rmb.int.aws.folio.org/(lcp2)
https://bugfest-mg.int.aws.folio.org


Attachments: File 10k-item-barcodes.csv     File MODEXPW-255_2022-10-12_14h34_02.mp4     PNG File are you sure.png     File barcodes-10000.csv     PNG File chrome_LFRLYhIOSz.png     PNG File chrome_ZzbIVPO2vA.png     PNG File chrome_sjj1ug4Lbm.png     PNG File image-2022-10-13-23-08-15-588.png     File log-events-viewer-result (1).csv     File logs-insights-results (4).csv     PNG File result.png     PNG File screenshot-1.png     PNG File screenshot-2.png     PNG File screenshot-3.png    
Issue links:
Blocks
blocks PERF-288 PTF - Establish performance baseline ... Closed
Defines
defines UXPROD-3665 Firebird - Nolana R3 2022 Enhancement... Closed
Relates
relates to MODEXPW-150 A job stays in indefinitely "In progr... Closed
relates to MODEXPW-267 Long delay in populating preview scre... Closed
relates to UIBULKED-176 Loader disappears before data is loaded Closed
relates to FAT-3133 Review of C365620 Test Case Closed
Sprint: Firebird - Sprint 150, Firebird - Sprint 151
Story Points: 3
Development Team: Firebird
Release: Nolana (R3 2022)
RCA Group: TBD

 Description   

If there are multiple concurrent bulk edit jobs of the larger data sets (~10k records) and some of them have subset of records that do not need change in value then the Preview on the "Are you sure" form takes a long time to load and fails on timeout, due to [error] 50#50: *10528 upstream timed out (110: Connection timed out)

Steps to Reproduce:

  1. Navigate to the "Bulk Edit" => Select "Inventory-Items" app => Select "Items barcodes" from "Record identifier" dropdown
  2. Upload a .csv file with 5000 Items barcodes (e.g. with status "Available")
  3. Click "Actions" menu => Select "Start Bulk edit" option
  4. Select action from the dropdown that will not affect any record (for example, the same status that Items are related yet:  status = "Available")
  5. Click "Confirm changes" button

Expected Results:

*"Are you sure?" form opens
*"Preview of the records to be changed" uploaded and  contains all valid Ids from the uploaded file, even if they are not going to be updated
Actual Results:

"Preview of the records to be changed" is NOT uploaded, request fails on timeout

Log

2022/09/15 11:44:07 [error] 50#50: *10528 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.23.35.53, server: , request: "POST /mod-data-export-worker/bulk-edit/abfe0d91-7fb0-4599-9d73-c6331b55ad0c/items-content-update/upload HTTP/1.1", upstream: "http://127.0.0.1:8082/bulk-edit/abfe0d91-7fb0-4599-9d73-c6331b55ad0c/items-content-update/upload", host: "pvt.lb.lcp2.folio-eis.us-east-1:8051"

Response time for POST /bulk-edit/$

{jobId}

/items-content-update/upload

For 100 records - about 10-20 seconds
For 1000 records -about 2-3 minutes
For 10k records - unable to get a successful response
For 100k records - unable to get a successful response

Attached log files

Additional information:
From Yauheniya Kryshtafovich: For example, if the user updates item status from to Available to Missing we see following behavior:
1. If the file contains 10K items and some of them (more than 4000 or so) do not need to be updated (already have Status_1= Missing), we may wait a long time for an answer from the endpoint and the preview will time out
2. If the file contains 10K items, and part of them (less than 4000 or so) do not need to be updated (already with Status_1= Missing), the preview will take a long time to be downloaded, but up to 10 minutes, and the preview will be updated



 Comments   
Comment by Ann-Marie Breaux (Inactive) [ 16/Sep/22 ]

HI Magda Zacharska There was no dev team on this bug, but I think it's Firebird. Please adjust if that's not correct. Thank you!

Comment by Mikita Siadykh [ 20/Sep/22 ]

expected to be fixed in MODEXPW-150 Closed

Comment by Magda Zacharska [ 26/Sep/22 ]

The issue will be fixed in Nolana as it closely relates to MODEXPW-267 Closed

Comment by Magda Zacharska [ 07/Oct/22 ]

Hi Siarhei Charniak - can this ticket be verified on bulk edit rancher environment?

Comment by Magda Zacharska [ 10/Oct/22 ]

Hi Siarhei Charniak - snapshot environments don't have enough data to do the verification. How this fix could be verified?

Comment by Siarhei Charniak [ 12/Oct/22 ]

Hi Magda Zacharska, Yauheniya Kryshtafovich

Verified on folio-snapshot-2 environment.
1. Created 10001 items
2. Followed steps to reproduce

Response for POST /bulk-edit/$

{jobId}

/items-content-update/upload for 10k items was received in a few seconds:

Comment by Yauheniya Kryshtafovich [ 12/Oct/22 ]

Hi, Magda Zacharska and Siarhei Charniak The story verified on snapshot-2:
1. On the snapshot-2 env were added 10.000 Items records
2. Identifiers (barcodes) of those Items added to .csv file ( barcodes-10000.csv )
3. Navigate to the Bulk edit app => Select "Inventory-Items" app from the "Record types" accordion => Select " Item barcodes" from "Records identifier" dropdown
4. Upload a .csv file with "Item barcodes" by dragging it on the Drag & drop area => The "Preview of record matched" accordion shows a table populated with top 10 matched Items records, all Items have parameters:
Permanent location = empty
Temporary location = Main library
Status = Available
5. Click "Actions" menu => Select "Start bulk edit" => Modify items by selecting:
Temporary location = Annex (new location)
Permanent location = empty (no changes needed)
Status = Available (no changes needed)

6. The "Are you sure?" form is uploading (approximately 5 sec)
Once the "Are you sure?" form uploaded, the notification " 10000 of records will be changed if the "Commit changes" button is clicked" appears
Populated with top 10 records "Preview of the records to be changed" is displayed
NO timeout error occurred
MODEXPW-255_2022-10-12_14h34_02.mp4

Comment by Magda Zacharska [ 14/Oct/22 ]

Hi Yauheniya Kryshtafovich and Siarhei Charniak - I uploaded 10k item barcodes (file attached)
Here are the changes in the in-app:

After I clicked "Confirm changes" I got the following screen for more than ~10 minutes now.

There is no entry in Export manager and the developer tools show that the job is Stalled:

After one hour, the response returned 504 and Are you sure? form shows 0 records:

There is still no entry in the Export Manager. Please check the logs on the server side - to see if there is an explanation for this behavior.

Comment by Siarhei Charniak [ 14/Oct/22 ]

Hi Magda Zacharska, those changes are not yet deployed on perf environment.

Comment by Yauheniya Kryshtafovich [ 17/Oct/22 ]

Hi, Magda Zacharska and Siarhei Charniak The story was checking on https://folio-dev-firebird-diku.ci.folio.org/ , the "Are you sure" form is populated in several seconds (~5 sec), but after clicking on "Commit changes" button* the process of updating takes a long time*. For example, I've uploaded file with 10K Items barcodes >> Changed Available status and Temporary location to the same values >> The progress bar appeared >>* In 1 hour* (according to DevTools) only 1100 Items were updated (approx. 11%)

Comment by Yauheniya Kryshtafovich [ 25/Oct/22 ]

Hi, Magda Zacharska
The story was verified on https://folio-perf-bulk-edit-fs09000000.ci.folio.org/ env for Users and Items records types. For Holdings records it is not possible to verify correctly due to MODEXPW-266 Closed
The results of test scenarios you can see at the document below:
https://docs.google.com/spreadsheets/d/1B4BaIp99W3ec3Yp6zCnAQuyyFFc21yoMTIQYr8jMbi8/edit?usp=sharing

Generated at Thu Feb 08 22:30:45 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.