Overview

This document contains the results of testing Check-in/Check-out and Data Import for MARC Bibliographic records in the Quesnelia release.

Ticket: PERF-824 - Getting issue details... STATUS

Summary

• Data import tests finished successfully for all files. Duration of DI grew in correspondence with the number of records in files.
• Check-in and Check-out with 8 virtual users was performed during DI Create and Update jobs. No issues.
• Data Import in Quesnelia with CICO perform faster than without it.
• Comparing Poppy and Quesnelia releases
  • Check-in / Check-out perform better in Quesnelia.  Response time improved during Create jobs - 30% in Average, and during DI Update jobs - 15% in Average.
  • DI durations do not differ much
• Resource utilization
  • Average CPU utilization did not exceed 150% for all modules. The highest consumption observed from mod-inventory - 144% in DI Update job with 25k file. The same maximum level as in Poppy. 
  • No memory leaks observed during tests.
  • Average DB CPU usage during data import has 5% decreasing in Quesnelia - close to 90%.
  • Average connection count during data import is about 750 connections for create jobs that is 450 connections higher than in Poppy. For update jobs - 730 connections.

Test Runs 

Test Results

Data import

Files for Data Import update jobs prepared during previous tests. So no need to run Data Export.

Check-in/Check-out without DI

Comparison

CICO with DI comparison

DI duration results without Check-In and Check-Out for Quesnelia were taken from the report Data Import test report (Quesnelia)[non-ECS].

The following table compares test results of current release (Quesnelia ) to the previous release (Poppy).

* Poppy DI and CICO results are taken from Data Import with Check-ins Check-outs Poppy

Detailed CICO response time comparison

Detailed CICO response time for CICO with DI in Poppy

*Top-10 requests were taken for analysis.

Response time

DI MARC BIB Create + CICO

DI MARC BIB Update + CICO

Service CPU Utilization

Average CPU utilization did not exceed 150% for all the modules. The highest consumption observed from mod-inventory. The same level as in Poppy. 

Spike for mod-data-import module observed instantly after the start in DI Create job with 100k file by 150%. For other tests it didn't exceed 110%.

DI MARC BIB Create and Update + CICO

Service Memory Utilization

There is memory utilization increasing observed which is caused by previous modules restarting (everyday cluster shut down process).

Top 5 modules with highest memory consumption: mod-inventory - 87%, mod-search - 87%, mod-oa - 75%, mod-dcb - 68%, mod-source-record-manager - 66%, 

Mod-data-export-worker-b was no level of 93% before and after tests.

DB CPU Utilization

Average DB CPU usage during data import is about 90% It decreased 5% from 95% in Poppy.

DB Connections

Average connection count during data import is about 750 connections for create jobs that is 450 connections higher than in Poppy. For update jobs - 730 connections.

DB load

Create jobs

Update jobs

Top SQL-queries:

Create jobs

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

Update jobs

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

UPDATE fs09000000_mod_inventory_storage.instance SET jsonb = $1::jsonb WHERE id='99b071cf-f789-4dfe-a238-d62c40bccfc0'

PTF -environment qcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	2731

• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Task count for modules mod-agreements, mod-serials-management, mod-graphql set to 0 during tests.

Modules

Methodology/Approach

• To run CI/CO - Ubuntu AWS instance was used as load generator

• DI tests were started from UI