Data Import in Central and Member Tenants

Owned by Hanna Tuhuzbayeva
Last updated: Jan 09, 2024 by Leonid Kolesnykov

Overview

This document contains the results of testing Data Import in central and member tenants with ECS and DI file splitting feature enabled in the Poppy release.

Tickets: PERF-753 - Getting issue details... STATUS and PERF-754 - Getting issue details... STATUS  

Summary

  • DI create and update jobs have a better performance compared with test results without ECS enabled on Poppy release. More details in comparison table.
  • DI Create job results in central tenant and member tenant do not differ much for  from 1k to 50k. DI with 100k records file worked faster in central tenant.
  • DI Update jobs perform faster in central tenant and for 100k records file it was 5 minutes faster than in member tenant.
  • Service CPU utilization of mod-inventory in central tenant didn't exceed 130% during DI Update jobs. In member tenant it was 160%.
  • CPU utilization decreased for major of modules in both central and member tenants if to compare with non-ECS results. The only module which shows insignificant growth is mod-inventory-storage (+6%).
  • After an additional set of 100k DI create jobs on pcon (to populate DB with instances on member tenants) it was observed a growth trend of CPU utilization with subsequent Out Of Memory issue for mod-inventory after spike 361%.
  • Memory consumption for mod-inventory was 100% in average for both tenants.
  • Memory consumption increased for mod-source-record-storage (20%) and mod-source-record-manager (10%) in both central and member tenants if to compare with non-ECS results. It decreased for data-import module (15%).
  • RDS utilized 98% with DI MARC Bib Create jobs and 94% with DI MARC Bib Update jobs in central tenant and 95% with DI MARC Bib Create jobs and 90% with DI MARC Bib Update jobs in member tenant.

  • OpenSearch Service CPU utilization - 97% and Memory consumption - 99%.

Recommendations and Jiras

  • After an additional set of 100k DI create jobs on pcon (to populate DB with instances on member tenants) it was observed a growth trend of CPU utilization with subsequent Out Of Memory issue for mod-inventory after spike 361%.
  • Jira ticket was created PERF-764 - Getting issue details... STATUS and investigated. Results of heap dump analysis attached as a comment.
  • Consider more CPU units allocation to mode-data-import taking into account that during DI Update jobs CPU utilization for mod-data-import module exceed 100%.

Test Runs 

Test #

Scenario

Load levelComment
1DI MARC Bib Create1K, 5K, 10K, 25K, 50K, 100K consecutively (with 5 min pause)Central tenant only
2DI MARC Bib Update1K, 5K, 10K, 25K, 50K, 100K consecutively (with 5 min pause)
3DI MARC Bib Create1K, 5K, 10K, 25K, 50K, 100K consecutively (with 5 min pause)Member tenant only
4DI MARC Bib Update1K, 5K, 10K, 25K, 50K, 100K consecutively (with 5 min pause)

Test Results

ProfileMARC FileDI Duration (hh:mm:ss)
Central tenantMember tenant
DI MARC Bib Create (PTF - Create 2)1K.mrc00:00:3800:00:33
5K.mrc00:02:1300:02:02
10K.mrc00:03:5400:03:54
25K.mrc00:09:4400:10:03
50K.mrc00:18:4900:18:50
100K.mrc00:37:4600:39:33
DI MARC Bib Update (PTF - Updates Success - 1)1K.mrc00:00:4400:00:33
5K.mrc00:02:2600:02:39
10K.mrc00:04:5700:05:20
25K.mrc00:12:0500:13:21
50K.mrc00:24:2700:26:43
100K.mrc00:49:1500:54:29

Comparison Table

ProfileMARC FileDI Duration (hh:mm:ss)
pconpcp1pcon/pcp1
Central tenantMember tenantCentral tenantDelta, central tenant
DI MARC Bib Create (PTF - Create 2)1K.mrc00:00:3800:00:3300:00:3900:00:01
5K.mrc00:02:1300:02:0200:02:3900:00:26
10K.mrc00:03:5400:03:5400:05:0000:01:06
25K.mrc00:09:4400:10:0300:11:1500:01:31
50K.mrc00:18:4900:18:5000:22:1600:03:27
100K.mrc00:37:4600:39:3300:49:5800:12:12
DI MARC Bib Update (PTF - Updates Success - 1)1K.mrc00:00:4400:00:3300:00:3400:00:10
5K.mrc00:02:2600:02:3900:02:2800:00:02
10K.mrc00:04:5700:05:2000:05:3100:00:34
25K.mrc00:12:0500:13:2100:14:5000:02:45
50K.mrc00:24:2700:26:4300:32:5300:08:26
100K.mrc00:49:1500:54:2901:14:3900:25:24

* - the results of DI without Check-in/Check-out in Poppy release were taken from the report Data Import with Check-ins Check-outs (Poppy)

Service CPU Utilization

CPU utilization decreased for major of modules in both central and member tenants if to compare with non-ECS tests results. The only module which shows insignificant growth is mod-inventory-storage (+ 6%).

ModuleCentral tenantMember tenant
Create JobsUpdate JobsCreate JobsUpdate Jobs
ECSNon-ECSECSNon-ECSECSNon-ECSECSNon-ECS
mod-inventory-b101%125%132%220%112%125%158%220%
mod-inventory-storage-b31%25%36%25%36%25%32%25%
mod-source-record-storage-b52%60%36%50%48%60%33%50%
mod-source-record-manager-b29%35%22%45%29%35%20%45%
mod-di-converter-storage-b68%80%68%90%56%80%52%90%
mod-data-import135%200%254%96% 25k file179%200%161%96% 25k file

This table provides Average CPU utilization in ECS and Non-ECS test results in 100k records file.

Central tenant

During create jobs the highest cpu utilization was with 100k record file by mod-inventory - 99%. During update jobs mod-inventory module utilized 130% with 100k record file. The spikes were observed in mod-data-import at the very beginning of each job that was expected. The highest spike was in update job with 100k records file - 250%.


Create jobs: 

The highest 101% of resource utilization was observed for mod-inventory. And at the end of the test we see that mod-quick-marc-b (98%) module begin to consume more than than other modules. Such behaviour was observed for all create jobs.

Average for mod-inventory-b - 101%, mod-inventory-storage-b - 31%, mod-source-record-storage-b - 52%, mod-source-record-manager-b - 29%, mod-di-converter-storage-b - 68%, , mod-data-import - 135% spike for 100k job.

Non-ECS tests*: Average for mod-inventory-b - 125%, mod-inventory-storage-b - 25%, mod-source-record-storage-b - 60%, mod-source-record-manager-b - 35%, mod-di-converter-storage-b - 80%, , mod-data-import - 200% spike for 100k job and mod-data-import - 86% spike for 25k job.

Update jobs:

The highest 134% of resource utilization was observed for mod-inventory.

Average for mod-inventory-b - 132%, mod-inventory-storage-b - 36%, mod-source-record-storage-b - 36%, mod-source-record-manager-b - 22%, mod-di-converter-storage-b - 68%, , mod-data-import - 254% spike for 100k job and mod-data-import - 85% spike for 25k job.

Non-ECS tests results*: Average for mod-inventory-b - 220%, mod-inventory-storage-b - 25%, mod-source-record-storage-b - 50%, mod-source-record-manager-b - 45%, mod-di-converter-storage-b - 90%, , mod-data-import - 96% spike for 25k job.

Member tenant

During create jobs the highest cpu utilization was with 100k record file by mod-inventory - 117%. During update jobs mod-inventory module utilized 160% with 100k record file. The spikes were observed in mod-data-import at the very beginning of each job that was expected.

Create jobs: 

The highest 112% of resource utilization was observed for mod-inventory. And at the end of the test we see that mod-quick-marc-b(122%) module begin to consume more than than other modules. Such behaviour was observed for all create jobs.

Average for mod-inventory-b - 112%, mod-inventory-storage-b - 36%, mod-source-record-storage-b - 48%, mod-source-record-manager-b - 29%, mod-di-converter-storage-b - 56%, , mod-data-import - 179% spike for 100k job and mod-data-import - 70% spike for 25k job.

on-ECS tests results*: Average for mod-inventory-b - 125%, mod-inventory-storage-b - 25%, mod-source-record-storage-b - 60%, mod-source-record-manager-b - 35%, mod-di-converter-storage-b - 80%, , mod-data-import - 200% spike for 100k job.

Update jobs:

The highest 158% of resource utilization was observed for mod-inventory.

Average for mod-inventory-b - 158%, mod-inventory-storage-b - 32%, mod-source-record-storage-b - 33%, mod-source-record-manager-b - 20%, mod-di-converter-storage-b - 52%, , mod-data-import - 161% spike for 100k job and mod-data-import - 73% spike for 25k job.

Non-ECS tests results*: Average for mod-inventory-b - 220%, mod-inventory-storage-b - 25%, mod-source-record-storage-b - 50%, mod-source-record-manager-b - 45%, mod-di-converter-storage-b - 90%, , mod-data-import - 96% spike for 25k job.

Memory Utilization

Memory consumption increased for mod-source-record-storage (20%) and mod-source-record-manager (10%) in both central and member tenants if to compare with non-ECS tests results. It decreased for data-import module (15%).

This table provides Average Memory Consumption in ECS and Non-ECS test results. Mod-di-converter-storage-b in the Non-ECS tests scenario is not provided, as indicated by the "-".

ModuleECSNon-ECS
Central tenantMember tenant
Create JobsUpdate JobsCreate JobsUpdate Jobs
mod-inventory-b96%96%101%101%90%
mod-inventory-storage-b18%21%20%23%18%
mod-source-record-storage-b64%74%71%71%46%
mod-source-record-manager-b49%49%46%43%38%
mod-di-converter-storage-b32%33%33%33%-
mod-data-import38%38%35%37%53%

Central tenant

Memory consumption for module mod-inventory grew gradually to 96 % during create jobs and didn't change during update jobs. No memory leaks were detected.

Create jobs: 

Average for mod-inventory-b - 96%, mod-inventory-storage-b - 18%, mod-source-record-storage-b - 64%, mod-source-record-manager-b - 49%, mod-di-converter-storage-b - 32%, , mod-data-import - 38%.

Update jobs:

Average for mod-inventory-b - 96%, mod-inventory-storage-b - 21%, mod-source-record-storage-b - 74%, mod-source-record-manager-b - 49%, mod-di-converter-storage-b - 33%, , mod-data-import - 38%.

Non-ECS tests results*:

Average for mod-inventory-b - 90%, mod-inventory-storage-b - 18%, mod-source-record-storage-b - 46%, mod-source-record-manager-b - 38%, mod-di-converter-storage-b - %, , mod-data-import - 53%.

Member tenant

Memory consumption for module mod-inventory grew gradually to 101% during create jobs and didn't change during update jobs. No memory leaks were detected.

Create jobs: 

Average for mod-inventory-b -101 %, mod-inventory-storage-b - 20%, mod-source-record-storage-b - 71%, mod-source-record-manager-b - 46%, mod-di-converter-storage-b - 33%, , mod-data-import - 35%.

Update jobs:

Average for mod-inventory-b - 101%, mod-inventory-storage-b - 23%, mod-source-record-storage-b - 71%, mod-source-record-manager-b - 43%, mod-di-converter-storage-b - 33%, , mod-data-import - 37%.

Non-ECS tests results*:

Average for mod-inventory-b - 90%, mod-inventory-storage-b - 18%, mod-source-record-storage-b - 46%, mod-source-record-manager-b - 38%, mod-di-converter-storage-b - %, , mod-data-import - 53%.

DB CPU Utilization

Central tenant

RDS utilized 98% with DI MARC Bib Create jobs and 94% with DI MARC Bib Update jobs


Member tenant

RDS utilized 95% with DI MARC Bib Create jobs and 90% with DI MARC Bib Update jobs

DB Connections

Central tenant

In central tenant DB connections did not exceed 312 for DI MARC Bib Create jobs and 290 for DI MARC Bib Update jobs.

Member tenant

DB connections did not exceed 312 for DI MARC Bib Create jobs and 300 for DI MARC Bib Update jobs.

DB Load

Central tenant

Waits

SQL

Member tenant

Waits

SQL

Top SQL

Central tenant

 Top SQL:

INSERT INTO cs00000int_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO cs00000int_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

INSERT INTO cs00000int_mod_search.consortium_instance (tenant_id, instance_id, json, created_date, updated_date) VALUES ($1, $2, $3::json, $4, $5) ON CONFLICT (tenant_id, instance_id) DO UPDATE SET json = EXCLUDED.json, updated_date = EXCLUDED.updated_date

UPDATE cs00000int_mod_inventory_storage.instance SET jsonb = $1::jsonb WHERE id=?

COMMIT

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO cs00000int_mod_inventory_storage.item (id, jsonb) VALUES ($1, $2) RETURNING jsonb

INSERT INTO cs00000int_mod_inventory_storage.instance (id, jsonb) VALUES ($1, $2) RETURNING jsonb

INSERT INTO cs00000int_mod_inventory_storage.holdings_record (id, jsonb) VALUES ($1, $2) RETURNING jsonb

UPDATE cs00000int_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *

Member tenant


 Top SQL:

INSERT INTO cs00000int_0001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO cs00000int_0001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

INSERT INTO cs00000int_mod_search.consortium_instance (tenant_id, instance_id, json, created_date, updated_date) VALUES ($1, $2, $3::json, $4, $5) ON CONFLICT (tenant_id, instance_id) DO UPDATE SET json = EXCLUDED.json, updated_date = EXCLUDED.updated_date

COMMIT

UPDATE cs00000int_0001_mod_inventory_storage.instance SET jsonb = $1::jsonb WHERE id=?

UPDATE cs00000int_0001_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *

INSERT INTO cs00000int_0001_mod_inventory_storage.instance (id, jsonb) VALUES ($1, $2) RETURNING jsonb

INSERT INTO cs00000int_0001_mod_inventory_storage.item (id, jsonb) VALUES ($1, $2) RETURNING jsonb

SELECT cs00000int_0001_mod_inventory_storage.count_estimate(?)

INSERT INTO cs00000int_0001_mod_inventory_storage.holdings_record (id, jsonb) VALUES ($1, $2) RETURNING jsonb

OpenSearch Service

CPU utilization (Percent)

Central tenant

Member tenant

Maximum memory utilization (Percent

Central tenant

Member tenant

Indexing Data Rate (operations/min)

Central tenant

Member tenant

Appendix

Methodology/Approach

DI tests were started from UI with 5 min pauses between tests.

Infrastructure

pcon

  • mod_source_record_storage.marc_records_lb = 2171059
  • mod_source_record_storage.raw_records_lb = 2171059
  • mod_source_record_storage.records_lb = 2171059
  • mod_source_record_storage.marc_indexers =  160397101
  • mod_source_record_storage.marc_indexers with field_no 010 = 1102025
  • mod_source_record_storage.marc_indexers with field_no 035 = 6621158
  • mod_inventory_storage.authority = 0
  • mod_inventory_storage.holdings_record = 392000
  • mod_inventory_storage.instance = 1604898
  • mod_inventory_storage.item = 392000

pcp1

  • mod_source_record_storage.marc_records_lb = 28000638
  • mod_source_record_storage.raw_records_lb = 28032783
  • mod_source_record_storage.records_lb = 28032783
  • mod_source_record_storage.marc_indexers =  611119607
  • mod_source_record_storage.marc_indexers with field_no 010 = 1119740
  • mod_source_record_storage.marc_indexers with field_no 035 = 22621766
  • mod_inventory_storage.authority = 7402975
  • mod_inventory_storage.holdings_record = 24165674
  • mod_inventory_storage.instance = 24206317
  • mod_inventory_storage.item = 25375853

Environment: PCON
Release: Poppy (2023 R2)

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)
  • 2 instances of db.r6.xlarge database instances, one reader, and one writer 
  • MSK tenant
    • 4 brokers

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • og.retention.minutes=480
    • default.replication.factor=3

Modules

ModuleTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabled
pcon-pvt
Wed Dec 20 09:08:19 UTC 2023
mod-search2mod-search:3.0.3225922480204814405121024FALSE
mod-data-import1mod-data-import:3.0.01204818442561292384512FALSE
mod-authtoken1mod-authtoken:2.14.021440115251292288128FALSE
mod-inventory-storage1mod-inventory-storage:27.0.024096369020483076384512FALSE
mod-source-record-storage1mod-source-record-storage:5.7.025600500020483500384512FALSE
mod-inventory1mod-inventory:20.1.022880259210241814384512FALSE
mod-di-converter-storage1mod-di-converter-storage:2.1.02102489612876888128FALSE
mod-users2mod-users:19.2.02102489612876888128FALSE
mod-source-record-manager1mod-source-record-manager:3.7.025600500020483500384512FALSE
mod-quick-marc1mod-quick-marc:5.0.01228821761281664384512FALSE
okapi-b2okapi:5.1.13168414401024922384512FALSE