Issue with holdings load when 2 processes are running in parallel

Description

A few points to note:

1. Two tenants - fs00000000 and fs00001000 are running in the same container on the same EC2 instance.
2. Both tenants are using different RM API accounts - fs00000000 is using "ns187805" and fs00001000 is using "s3911979"

Please see attached logs:
1. Few interesting things to note in logs are while the load process is running for fs00000000 at 20:48:00:092 and imported 100,000 at 20:48:01:458 we receive signal to start the process again and it restarts for tenant fs00000000 eventually resulting in error
2. Soon after at 20:48:02:721, we see an error with tenant fs00001000 and error is from RM API as below:

which results in an Internal Server error in our code.

Either RM API holdings process or our holdings process is having an issue with parallel processes getting holdings information although using 2 different custids.

Note: Please do not use custid = s3911979 for testing purposes. It is a real production account. Use 'apidvcorp' and 'apidvgvmt' production accounts.

Holdings Status table for fs00000000 - this is probably stuck there because we removed RM API configuration since this was a tenant not in use

Holdings status table for fs00001000 -

Acceptance Criteria:
1. Investigate the root cause and see if we can come up with a solution.
2. See if we already fixed this in newer version of code. These logs are from q3.1 release version.
3. Create user stories to fix the issue as necessary

CSP Request Details

None

CSP Rejection Details

None

Potential Workaround

None

Attachments

8

Checklist

hide

TestRail: Results

Activity

Show:

Sobha Duvvuri October 15, 2019 at 2:18 PM

Yes, understand. thank you!

Andrii Paias October 15, 2019 at 11:13 AM

,
Yes, that is the fix that we added in 3.0.0 . Multiple calls from okapi can't break loading process now, but if okapi makes calls with enough time between them (e.g. second okapi container makes a call 5 hours after the first one), then loading will be done multiple times.

Sobha Duvvuri October 14, 2019 at 5:14 PM

Speaking with Adam about this issue - timer interface is called once per okapi process - since we have 3 okapi containers running - 3 processes - so timer interface is called thrice and that is the right thing to do. This should be taken care of by the lock that is put in place in our code though - because if one holdings process for a tenant + rm api creds combo is in progress, another request from okapi should not interfere with the already running process is what I understand - correct ?

Andrii Paias October 3, 2019 at 9:46 AM

For the issue of multiple calls to /loadHoldings interfering with each other, this should be fixed in 3.0.0 release.

Andrii Paias October 3, 2019 at 9:44 AM

Created issue in Okapi project: https://folio-org.atlassian.net/browse/OKAPI-769

Done

Details

Assignee

Reporter

Priority

Story Points

Sprint

Development Team

Spitfire

Fix versions

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs
Created September 27, 2019 at 7:01 PM
Updated October 18, 2019 at 1:22 PM
Resolved October 18, 2019 at 1:22 PM
TestRail: Cases
TestRail: Runs