Data Import Test Report (Juniper HF3)
In-Progress
Overview
This document contains the results of testing Data Import in Kiwi and Juniper Hot Fix 3, and compare their results against each other to detect performance trends.
Software versions
Juniper
- mod-data-import v2.1.4
- mod-source-record-manager v3.1.3
- mod-source-record-storage v5.1.6, started out with v5.1.6 but switched to v5.1.7 to match Juniper Bugfest
- mod-inventory v17.0.6
- mod-inventory-storage v21.0.9
(This set of Juniper modules requires the unique item barcode to be applied to the mod_inventory_storage.item barcode column because of MODINV-508 - Getting issue details... STATUS . Without it any Data Import job would not be able to proceed at all.)
MODSOURCE-422 Upgrade Vertx from 4.0.0 to 4.1.2 (Juniper HF5)
MODSOURCE-421 Add possibility to adjust parameter DB_MAXPOOLSIZE for configuring connection pool (Juniper HF5)
Results
Import durations are recorded in the following table. Note that we did a couple of imports and their durations are recorded here for completeness sake, separated by a comma.
Import Durations
| Profile | Juniper HF3 | HF3 with mod-inv v17.0.7 | HF5 Vertex 4.1.2 | HF5 Vertex 4.1.2 +DB_MAXPOOLSIZE (25) conf | |
|---|---|---|---|---|---|
5K MARC Create | PTF - Create 2 | 15, 17 min | 15 min | 11 min | |
5K MARC Update | PTF - Updates Success - 1 | 12 mins, FAILED | 32 min | 23 min | |
10K MARC Create | PTF - Create 2 | 28 min | 26 min | 24 min | |
10K MARC Update | PTF - Updates Success - 1 | FAILED | 1 hr | 1 hr 14 min | |
20K MARC Create | PTF - Create 2 | 56 min | 46 min | ||
20K MARC Update | PTF - Updates Success - 1 | DNR - slow | FAILED | ||
| 25K MARC Create | PTF - Create 2 | 1 hr 7 min | 1 hr 36 min | ||
| 25K MARC Update | PTF - Updates Success - 1 | 2 hr 33 min | 2 hr 24 min | ||
| 50K MARC Create | PTF - Create 2 | 2 hr 7 min Failed | 2 hr 3 min * | ||
| 50K Update | PTF - Updates Success - 1 | DNR |
Observations Juniper HF3
- mod-srs v5.1.6 tasks crashed every time doing an UPDATE import, which prompted us to use v5.1.7 to see if this problem was resolved.
- UDPATEs on Juniper HF3 most of the time does not work. Getting various errors.
- When
* Observations Juniper HF5 ( HF5 changes list)
- 50K Create is unstable on HF5 and may fail with 25 connections DB_MAXPOOLSIZE due to Error Connection is not active now, current status: CLOSED.
HF 5 DB_MAXPOOLSIZE experiments
notable observations:
- More connections from SRS to DB means higher service memory usage;
- Looks like DB_MAXPOOLSIZE parameter is the maximum connections per service (not task).
| Test # | DB_MAXPOOLSIZE | duration |
|---|---|---|
| 1 | 5 | 38 |
| 2 | 50 | 53 |
| 3 | 20 | 38 |
| 4 | 15 | 41 |
Juniper resources usage
