Importing 10,000 MARC authority records > Completes with errors due to timeout

Description

Overview:
Steps to Reproduce:

  1.  Login https://bugfest-orchid.int.aws.folio.org 

  2. Import a record with at least 10,000 authority records (see attached - ignore file name - it can be used) 

  3. Select the Default - Create SRS  MARC Authority job profile 

  4. Run the data import job

Expected Results:  All authority records are either updated or created 
Actual Results: Some records are created but most are not created (see job summary of errors - https://bugfest-orchid.int.aws.folio.org/data-import/job-summary/42552791-209a-452d-a36c-799de2bb98ab?errorsOnly=true&entity=SRS_MARC) 
**

Error returned: io.vertx.core.impl.NoStackTraceThrowable: Timeout

 

NOTE: mod-source-record-storage v5.6.2 has a huge performance issue with DB. Investigating which changes can affect it.

cc: and  

 

UPD

Next configurations were tests, but still failed with the same message:

  • SRS and SRM DB Poolsize 35, SRS DB Timeout 40;

  • SRS and SRM DB Poolsize 30, SRS DB Timeout 45;

  • SRS and SRM DB Poolsize 40, SRS DB Timeout 40;

  • SRS and SRM DB Poolsize 30, SRS DB Timeout 50.

 

mod-source-record-storage number of instances: 2.

mod-source-record-manager number of instances: 2.

Number of partitions for each DI topic: 2.

CSP Request Details

None

CSP Rejection Details

None

Potential Workaround

None

Attachments

1

Checklist

hide

TestRail: Results

Activity

Show:

Kateryna Senchenko March 30, 2023 at 10:07 AM

In scope of this issue indexes for 010 and 035 MARC fields were added. Additional changes to improve imports stability will be done in scope of . No further verification is required for this ticket, closing it.

JenkinsNotifications March 30, 2023 at 6:18 AM

Deployed to the {BugfestEnv} bf env. Moved status to In bugfix review from status Awaiting deployment. Please proceed with verification

Denis Anischenko March 27, 2023 at 2:36 PM

Hi and can you please clarify when this changes will be released ?

CC:  

Volodymyr Rohach March 27, 2023 at 8:52 AM

Hello ! So I merged PR with new indices to srs, and added 2 points to the release notes: Orchid (R1 2023) Release Notes - Releases - FOLIO Wiki.

So I'll move this ticket to In Review status.

CC:  

Ann-Marie Breaux March 23, 2023 at 2:16 PM
Edited

will add a release note explaining more details; check with if any questions about how to do that

Done

Details

Assignee

Reporter

Priority

Story Points

Sprint

Development Team

Folijet

Fix versions

Release

Orchid (R1 2023) Bug Fix

RCA Group

Environmental/deployment issue

Affected releases

Orchid (R1 2023)

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs
Created March 9, 2023 at 7:58 PM
Updated February 28, 2024 at 12:03 PM
Resolved March 27, 2023 at 8:52 AM
TestRail: Cases
TestRail: Runs