| Table of Contents |
|---|
...
Compared with results in previous test report: Data Import Create MARC holdings records [non-ECS] [QuesneliaPoppy]
Summary
- Data import create holdings jobs perform faster in Quesnelia release about 40% in Average comparing with Poppy.
Recommendations & Jiras
Errors
- about 40% in Average comparing with Poppy.
- Number of associated holdings to one instance in files used in tests do not affect duration of data import in non-ECS environments.
- Top CPU utilization: mod-inventory-b - 73%, nginx-okapi - 65%, mod-quick-marc-b - 57%, mod-source-record-storage-b - 35%
- Top Memory consumption:
- Set #1: mod-inventory-storage-b - 87%, mod-inventory-b - 72%, mod-data-import-b - 59%, Spikes for mod-inventory-storage-b were observed and after tests finished it returned into "before tests" state.
- Set #2: mod-inventory-storage-b - 24%, mod-inventory-b - 56%, mod-data-import-b - 58%, mod-users-b - 53%. During
- RDS CPU utilization was on level 95% for all DI tests except of test with 1k file.
- RDS DB connections were 860
Recommendations & Jiras
- Investigate memory growing trend for mod-inventory-storage in tests set #1 (using 1 instance HRID to create all Holdings).
Test Runs
Profile used for testing - Default - Create Holdings and SRS MARC Holdings
| Set of tests № | Scenario | Test Conditions |
|---|---|---|
| 1 | DI Holdings Create (previous* approach) | 1K, 5K, 10K, 80K sequentially |
| 2 | DI Holdings Create (new** approach) | 1K, 5K, 10K, 80K sequentially |
*previous approach - Data import Holdings with mrc file where 1 instance HRID is associated to all holdings (1k, 5k, 10k, 80k)
**new approach - Data import Holdings with mrc file where 1 instance HRID is associated to 1000 holdings
Test Results
...
Test | File | Duration: Quesnelia [non-ECS] Set #1 | Duration: Quesnelia [non-ECS] Set #2 | Status and Errors Quesnelia [non-ECS] Set #1, Set #2 |
|---|---|---|---|---|
| 1 | 1k | 19 sec | 25 sec | Success |
| 2 | 5k | 1 min 17 sec | 1 min 24 sec | Success |
| 3 | 10k | 2 min 32 sec | 2 min 40 sec | Success |
| 4 | 80k | 19 min 54 sec | 21 min 44 sec | Success |
Comparison
Test | File | Duration: Poppy [non-ECS] | Duration: Quesnelia [non-ECS] Set #1 | Delta, sec | %, Delta/Poppy Duration |
|---|---|---|---|---|---|
| 1 | 1k | 32 sec | 19 sec | 13 sec | 40.63% |
| 2 | 5k | 2 min 14 sec | 1 min 17 sec | 57 sec | 42.54% |
| 3 | 10k | 4 min 35 sec | 2 min 32 sec | 2 min 3 sec | 44.73% |
| 4 | 80k | 36 min 25 sec | 19 min 54 sec | 16 min 31 sec | 45.35% |
...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
UPDATE fs09000000_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *
INSERT INTO fs09000000_mod_inventory_storage.holdings_record (id, jsonb) VALUES ($1, $2) RETURNING jsonb
WITH input_rows(record_id, holdings_id) AS (
VALUES ($1::uuid,$2::uuid)
)
, ins AS (
INSERT INTO fs09000000_mod_inventory.records_holdings(record_id, holdings_id)
SELECT * FROM input_rows
ON CONFLICT (record_id) DO UPDATE SET record_id=EXCLUDED.record_id
RETURNING record_id::uuid, holdings_id::uuid
)
SELECT record_id, holdings_id
FROM ins
UNION ALL
SELECT c.record_id, c.holdings_id
FROM input_rows
JOIN fs09000000_mod_inventory.records_holdings c USING (record_id);
|
...
| Name | Memory GIB | vCPUs | Engine version |
|---|---|---|---|
db.r6g.xlarge | 32 GB | 4 vCPUs | 16.1 |
- MSK tenant
- 2 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
- DB records
- fs09000000
- instances - 25901331
- items - 27074913
- holdings - 25871735
- fs07000001
- nstances - 10100620
- items - 1484850
- holdings - 10522266
- fs07000002
- nstances - 1161275
- items - 1153548
- holdings - 1153548
- fs09000000
| Expand | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
...