Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Table of Contents
stylenone

...

Retest after this issue will be fixed

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyMODDATAIMP-1064

Test Runs

Important! Before each test, data-import MRC files were prepared using python script ensure creation of MARC Authority records.

...

DB connections were changing values from 550 connection to 870

...

Database load 

...

Top SQL

...

Open Search metrics
Master node CPU utilization

CPU utilization percentage for the master node reached maximum 35%

...

Maximum CPU utilization percentage for all data nodes.

...

Maximum memory usage percentage for all data nodes.

...

Amazon Managed Streaming

...

...


Test 2. DI Central tenant 1k-5K-10K-25K-50K + CI/CO 5VU

...

Expand
title0.22 WITH cte AS (SELECT count(*) AS total_count FROM cs00000int_mod_source_recor
Code Block
WITH cte AS (SELECT count(*) AS total_count FROM cs00000int_mod_source_record_manager.job_execution WHERE subordination_type <> 'PARENT_MULTIPLE' AND TRUE AND status IN ('COMMITTED', 'ERROR', 'CANCELLED') AND job_profile_id NOT IN ('d0ebb7b0-2f0f-11eb-adc1-0242ac120002', '91f9b8d6-d80e-4727-9783-73fb53e3c786') AND file_name NOT IN ('No file name') AND subordination_type NOT IN ('COMPOSITE_PARENT') AND NOT job_profile_hidden AND NOT is_deleted) SELECT j.*, cte.*, p.total_records_count total, p.succeeded_records_count + p.error_records_count currently_processed, (select jsonb_agg(x) composite_data from (select status, count(1)  cnt, sum(p1.total_records_count) total_records_count, sum(p1.succeeded_records_count + p1.error_records_count) currently_processed FROM cs00000int_mod_source_record_manager.job_execution j1 LEFT JOIN cs00000int_mod_source_record_manager.job_execution_progress p1 ON j1.id = p1.job_execution_id where j1.parent_job_id = j.id and j1.id != j1.parent_job_id and j1.subordination_type = 'COMPOSITE_CHILD' group by status) x) composite_data FROM cs00000int_mod_source_record_manager.job_execution j LEFT JOIN cs00000int_mod_source_record_manager.job_execution_progress p ON  j.id = p.job_execution_id LEFT JOIN cte ON true WHERE subordination_type <> 'PARENT_MULTIPLE' AND TRUE AND status IN ('COMMITTED', 'ERROR', 'CANCELLED') AND job_profile_id NOT IN ('d0ebb7b0-2f0f-11eb-adc1-0242ac120002', '91f9b8d6-d80e-4727-9783-73fb53e3c786') AND file_name NOT IN ('No file name') AND subordination_type NOT IN ('COMPOSITE_PARENT') AND NOT job_profile_hidden AND NOT is_deleted ORDER BY completed_date desc LIMIT $1 OFFSET $2

image-20240621-144736.pngImage Added

Maximum CPU utilization percentage for all data nodes.

...

Maximum memory usage percentage for all data nodes.

...

Amazon Managed Streaming

...

Test 3. Multitenant DI 50K-22K-10K-5K-1K.

...

Expand
titleBlue SQL query

image-20240611-123437.png

Code Block
languagesql
select distinct on ("records_lb"."id") "records_lb"."id", "records_lb"."snapshot_id", "records_lb"."matched_id", "records_lb"."generation", "records_lb"."record_type", "records_lb"."external_id", "records_lb"."state", "records_lb"."leader_record_status", "records_lb"."order", "records_lb"."suppress_discovery", "records_lb"."created_by_user_id", "records_lb"."created_date", "records_lb"."updated_by_user_id", "records_lb"."updated_date", "records_lb"."external_hrid", "marc_records_lb"."content" as "parsed_record_content", "raw_records_lb"."content" as "raw_record_content", "error_records_lb"."content" as "error_record_content", "error_records_lb"."description" from "records_lb" left outer join "marc_records_lb" on "records_lb"."id" = "marc_records_lb"."id" left outer join "raw_records_lb" on "records_lb"."id" = "raw_records_lb"."id" left outer join "error_records_lb" on "records_lb"."id" = "error_records_lb"."id" join "marc_indexers_010" on "records_lb"."id" = "marc_indexers_010"."marc_id" join "marc_records_tracking" on ("marc_records_tracking"."marc_id" = "marc_indexers_010"."marc_id" and "marc_records_tracking"."version" = "marc_indexers_010"."version") where ("records_lb"."record_type" = $1::"record_type" and "records_lb"."state" = $2::"record_state" and ("marc_indexers_010"."value" in ('n  027310230486 ') and "marc_indexers_010"."ind1" LIKE '#' and "marc_indexers_010"."ind2" LIKE '#' and "marc_indexers_010"."subfield_no" = 'a')) offset $3 rows fetch next $4 rows only


Open Search metrics
Master node CPU utilization

CPU utilization percentage for the master node reached maximum 47%

...

Maximum CPU utilization percentage for all data nodes.

...

Maximum memory usage percentage for all data nodes.

...

Amazon Managed Streaming

...

image-20240621-150324.pngImage Added

Appendix

Infrastructure

PTF - environment Quesnelia (QCON)

...