Overview

• This document contains the results of testing  Data Import for MARC Bibliographic records with an update job on the Quesnelia [ECS] release on qcon environment. 

Summary

• Data Import tests finished successfully on qcon environment using the PTF - Updates Success - 2 profile and file with 25k records.
• Comparing with previous testing results Poppy and Quesnelia releases
  • Data Import processed all jobs including test on 3 tenants concurrently without errors for Quesnelia releases.
  • Data Import durations stayed in the same time range in Average for Quesnelia releases but it works stable and without errors.
• During testing, we noticed that mod-permission did not have any spikes and used 12% CPU for Quesnelia releases. For Poppy releases we had error.

Test Results

This table contains durations for Data Import. 

...

College
(cs00000int_0001)

...

Comparison

This table contains durations comparison between Poppy and Quesnelia releases.

...

...

...

Service CPU Utilization

Here we can see that mod-inventory used 125% CPU, mod-di-converter-storage and nginx-okapi used 65%-70% CPU.

Image Removed

Service Memory Utilization

Here we can see that all modules show a stable trend.

Image Removed

DB CPU Utilization

DB CPU was 95%.

Image Removed

DB Connections

Max number of DB connections was 1650.

Image Removed

DB load

  Image Removed                                                                                                                   

Top SQL-queries

Image Removed

...

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

...

INSERT INTO cs00000int_0001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

...

 INSERT INTO cs00000int_mod_search.consortium_instance (tenant_id, instance_id, json, created_date, updated_date)
  VALUES ($1, $2, $3::json, $4, $5)
  ON CONFLICT (tenant_id, instance_id)
  DO UPDATE SET json = EXCLUDED.json, updated_date = EXCLUDED.updated_date

...

WITH deleted_rows AS 
( delete from marc_indexers mi where exists( select 1 from marc_records_tracking mrt 
where mrt.is_dirty = true and mrt.marc_id = mi.marc_id and mrt.version > mi.version )
returning mi.marc_id), deleted_rows2 AS ( delete from marc_indexers mi 
where exists( select 1 from records_lb where records_lb.id = mi.marc_id and records_lb.state = 'OLD' )
returning mi.marc_id) INSERT INTO marc_indexers_deleted_ids SELECT DISTINCT marc_id FROM deleted_rows UNION SELECT marc_id FROM deleted_rows2

...

INSERT INTO cs00000int_0002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

Appendix

Infrastructure

PTF - environment Quesnelia (qcon)

• 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

• 1 instance of db.r6.xlarge database instance: Writer instance

• OpenSearch

  • domain: fse

  • Number of nodes: 9

  • Version: OpenSearch_2_7_R20240502

• MSK - tenat

  • 4 kafka.m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=3

  • Kafka consolidated topics enabled

...

...

Overview

• This document contains the results of testing Check-in/Check-out and Data Import for MARC Bibliographic records on the Quesnelia [ECS] release on qcon environment. 

Summary

• Data Import with Check In/Check Out tests finished successfully on qcon environment using the PTF - Create 2 and PTF - Updates Success - 2 profiles with files 5K, 10K, 25K, 50K, 100K records.
• Comparing with previous testing results Poppy and Quesnelia releases
  • Data Import durations for create jobs has performance degradation in Average up to 10% for Quesnelia releases.
  • Data Import durations for update jobs has performance degradation in Average up to 40% for Quesnelia releases.
  • Check In/Check Out Response time has slight improvement in Average up to 10% for Quesnelia releases.
• During testing, we noticed that mod-data-import module used maximum 57% CPU for Quesnelia releases. For Poppy releases we mod-data-import module used 130% CPU and 320% in spike.
• No memory leaks are observed.

Test Runs 

Test Results

This table contains durations for Data Import. 

Check-in/Check-out without DI

Comparison

This table contains DI durations with CICO comparison between Poppy and Quesnelia releases.

Service CPU Utilization

Here we can see that mod-data-import used 150% CPU in spikes.

Image Added

Service Memory Utilization

Here we can see that all modules show a stable trend.

Image Added

DB CPU Utilization

DB CPU was 92%.

Image Added

DB Connections

Max number of DB connections was 1690.

Image Added

DB load

 Image Added                                                                                                                    

Top SQL-queries

Image Added

INSERT INTO cs00000int_0001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO cs00000int_0001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) 
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

  INSERT INTO cs00000int_mod_search.consortium_instance (tenant_id, instance_id, json, created_date, updated_date)
  VALUES ($1, $2, $3::json, $4, $5)
  ON CONFLICT (tenant_id, instance_id)
  DO UPDATE SET json = EXCLUDED.json, updated_date = EXCLUDED.updated_date

INSERT INTO cs00000int_0001_mod_inventory_storage.holdings_record (id, jsonb) VALUES ($1, $2) RETURNING jsonb

Service CPU Utilization

Here we can see that mod-data-import used 130% CPU in spikes.

Image Added

Service Memory Utilization

Here we can see that all modules show a stable trend.

Image Added

DB CPU Utilization

DB CPU was 92%.

Image Added

DB Connections

Max number of DB connections was 1685.

Image Added

DB load

 Image Added                                                                                                                    

Top SQL-queries

Image Added

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO cs00000int_0001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO cs00000int_0001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) 
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

INSERT INTO cs00000int_mod_search.consortium_instance (tenant_id, instance_id, json, created_date, updated_date)
  VALUES ($1, $2, $3::json, $4, $5)
  ON CONFLICT (tenant_id, instance_id)
  DO UPDATE SET json = EXCLUDED.json, updated_date = EXCLUDED.updated_date

UPDATE cs00000int_0001_mod_inventory_storage.instance SET jsonb = $1::jsonb WHERE id=?

Appendix

Infrastructure

PTF - environment Quesnelia (qcon)

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 instance of db.r6.xlarge database instance: Writer instance

• OpenSearch

  • domain: fse

  • Number of nodes: 6

  • Version: OpenSearch_2_7_R20240502

• MSK - tenat

  • 4 kafka.m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=3

  • Kafka consolidated topics enabled

Methodology/Approach

DI tests scenario(DI MARC Bib Update) were started from UI.

Test set:

• Test 1: Manually tested 25k records files DI started on College tenant(cs00000int_0001) only.
• Test 2: Manually tested 25k records files DI started on College tenant(cs00000int_0001) and Professional tenant(cs00000int_0002) concurrently, step 30%.
• Test 3: Manually tested 25k records files DI started on College tenant(cs00000int_0001), Professional tenant(cs00000int_0002) and School tenant(cs00000int_0003) concurrently, step 30%.

To get status and time range for import jobs the query used: 

Methodology/Approach

DI tests scenario (DI MARC Bib Create and Update) were started from UI on Quesnelia (qcon) env with  file splitting features enabled on a ecs environment..

Test runs:

• Test 1: Manually tested 5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause) records files, DI (DI MARC Bib Create ) started on College tenant(cs00000int_0001) only, and CICO with 8 users on background.
• Test 2: Manually tested 5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause) records files, DI (DI MARC Bib Update) started on College tenant(cs00000int_0001) only, and CICO with 8 users on background.

At the time of the test run, Grafana was not available. As a result, response times for Check-In/Check-Out were parsed manually from a .jtl files, using the start and finish dates of the data import tests. These results were visualized in JMeter using a Listener (Response Times Over Time).