NFR: R2 2021 Juniper Data Import Stabilization and Reliability work

Description

Architectural and Infrastructure work to make Data Import more stable and performant

Based on feedback from the PTF and architects

As of 14 April 2021, PTF has tested importing files of 50K MARC to create Bibs, Instances, Holdings, Item

Additional testing still needed

  • Multiple imports triggered by different users at the same time

  • Imports that update SRS and Inventory records (so have matching portion in the job profile, which creates does not have)

  • EDIFACT - to get a large renewal invoice from Anne C, so that we can see if that causes any issues

Priority

Fix versions

Development Team

Folijet

Assignee

Solution Architect

Parent Field Value

None

Parent Status

None

Checklist

hide

TestRail: Results

Activity

Show:

Ann-Marie Breaux June 17, 2021 at 6:49 AM

Architectural changes moved to Kiwi feature

Ann-Marie Breaux May 26, 2021 at 4:02 PM

After internal conversation, decided to deprioritize some of these architectural changes for Juniper, in favor of bugfixes and automated tests

Ann-Marie Breaux May 18, 2021 at 3:25 PM
Edited

Based on number of stories and total points, we likely will need to split this feature, and continue the work in R3

Ann-Marie Breaux April 29, 2021 at 7:50 AM

Hi We'll be putting together details on the wiki in the next week or two. We started with Creates for SRS Bibs, and all 3 Inventory record types. Now we are also working with multiple updates to the same records. We have 5 job profiles that should result in successes for all updates, 1 job profile that will fail for holdings and item updates, and 1 job profile that will fail on item updates. We are not yet making them as varied as you'll encounter in a real use case. Another large part of the testing is related to processing time, so we want predictable results before we start introducing more unpredictability. That whole walk before you run thing.

With a few adjustments to environment setup details, a load of 50K creating or updated 50K SRS, Instances, Holdings, and Items completes in ca. 2.5 hours. We think there are still some steps that we can take to improve that performance, but the devs are still analyzing the various runs, adjusting settings, and trying again. More soon, I promise!

Jacquie Samples April 28, 2021 at 1:21 PM
Edited

I am curious to know what kind of testing has happened on the data that was loaded in the 50k import.  Has anyone tested updating or overlaying those bib and/or order data?  In other words, was the load successful in that those data are now actionable and effective? 

I am also interested to know how long it took to load the 50k bib records.

Thanks!

Done

Details

Reporter

PO Rank

115

Front End Estimate

Very Small (VS) < 1day

Front End Estimator

Front-End Confidence factor

Medium

Back End Estimate

Jumbo: > 45 days

Back End Estimator

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs

Created March 31, 2021 at 12:56 PM
Updated August 31, 2021 at 6:57 AM
Resolved July 28, 2021 at 3:01 PM
TestRail: Cases
TestRail: Runs