Service CPU Utilization

Here we can see that mod-inventory used 125% CPU, mod-di-converter-storage and nginx-okapi used 65%-70% CPU.

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 95%.

DB Connections

Max number of DB connections was 1650.

DB load

Top SQL-queries

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO cs00000int_0001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

 INSERT INTO cs00000int_mod_search.consortium_instance (tenant_id, instance_id, json, created_date, updated_date)
  VALUES ($1, $2, $3::json, $4, $5)
  ON CONFLICT (tenant_id, instance_id)
  DO UPDATE SET json = EXCLUDED.json, updated_date = EXCLUDED.updated_date

WITH deleted_rows AS 
( delete from marc_indexers mi where exists( select 1 from marc_records_tracking mrt 
where mrt.is_dirty = true and mrt.marc_id = mi.marc_id and mrt.version > mi.version )
returning mi.marc_id), deleted_rows2 AS ( delete from marc_indexers mi 
where exists( select 1 from records_lb where records_lb.id = mi.marc_id and records_lb.state = 'OLD' )
returning mi.marc_id) INSERT INTO marc_indexers_deleted_ids SELECT DISTINCT marc_id FROM deleted_rows UNION SELECT marc_id FROM deleted_rows2

INSERT INTO cs00000int_0002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

Appendix

Infrastructure

PTF - environment Quesnelia (qcon)

• 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

• 1 instance of db.r6.xlarge database instance: Writer instance

• OpenSearch

  • domain: fse

  • Data nodes Instance type: r6g.xlarge.search

  • Number of nodes: 9

  • Version: OpenSearch_2_7_R20240502

• MSK - tenat

  • 4 kafka.m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=3

  • Kafka consolidated topics enabled

Methodology/Approach

DI tests scenario(DI MARC Bib Update) were started from UI.

Test set:

• Test 1: Manually tested 25k records files DI started on College tenant(cs00000int_0001) only.
• Test 2: Manually tested 25k records files DI started on College tenant(cs00000int_0001) and Professional tenant(cs00000int_0002) concurrently, step 30%.
• Test 3: Manually tested 25k records files DI started on College tenant(cs00000int_0001), Professional tenant(cs00000int_0002) and School tenant(cs00000int_0003) concurrently, step 30%.

To get status and time range for import jobs the query used: 

select file_name,started_date,completed_date, completed_date - started_date as duration ,status
from cs00000int_0001_mod_source_record_manager.job_execution order by started_date desc limit 1000;

select file_name,started_date,completed_date, completed_date - started_date as duration ,status
from cs00000int_0002_mod_source_record_manager.job_execution order by started_date desc limit 1000;

select file_name,started_date,completed_date, completed_date - started_date as duration ,status
from cs00000int_0003_mod_source_record_manager.job_execution order by started_date desc limit 1000;