Overview

This document contains the results of testing Check-in/Check-out and Data Import for MARC Bibliographic records in the Poppy release.

Ticket: PERF-697 - Getting issue details... STATUS

Summary

There is significant performance improvement for data import in Poppy comparing with Orchid. Durations are closer to Nolana release results. CI/CO response times improved 10% in average with all DI jobs. There's only CI response time degraded in 100k job with 17%.

Comparing Orchid and Poppy releases DI durations in create jobs are up to 10% higher with 5k file and 5% for bigger files with parallel Check-in/Check-out than pure Data import results. DI durations are higher up to 30% in update jobs.

Response times of CI/CO in Poppy release are twice higher with Data Import job with 100k compared with pure CI/CO.

DI create jobs with CI/CO for files 10k, 25k, 50k, 100k perform better in Poppy release and DI update jobs perform better with 25k, 50k, 100k with CI/CO.

No memory leaks are observed.

Average CPU utilization increased for mod-inventory up to 20% comparing with Orchid. So it did not exceed 150% for all the modules. The highest consumption observed from mod-inventory. The rest of services were almost on the same level in the same test in Orchid  Data Import with Check-ins Check-outs Orchid and didn't exceed 60%.

DB needs more connections (in average +20 more) needed for the same tests as in Orchid for all create and update jobs.

Average DB CPU usage is the same as in Orchid - 95%.

Upd: During previous tests on pcp1 and ocp3 there were problems with DI jobs running big files (Create jobs 100k and higher, Update jobs with 25k and higher). The problem was solved after new deployment and updates of 13 modules in scope of ticket  RANCHER-1121 - Getting issue details... STATUS and RANCHER-1114 - Getting issue details... STATUS and the large DI jobs are completing successfully now.

Test Runs 

Test Results

Data import

Total time for all Data Export jobs - 1 hour 16 minutes 47 seconds.

Check-in/Check-out without DI

Comparison

CICO with DI comparison

The following table compares test results of current release (Orchid) to the previous release numbers (Orchid) and to the baselines Poppy results (CICO without DI and DI without CICO).

* Orchid DI and CICO results are taken from Data Import with Check-ins Check-outs Orchid.

*** Completed with errors

Detailed CICO response time comparison

Detailed CICO response time comparison for CICO with DI in Poppy

*Top-10 requests were taken for analysis.

Response time

DI MARC BIB Create + CICO

DI Bib Update + CICO

Service CPU Utilization

Average CPU utilization did not exceed 150% for all the modules. The highest consumption observed from mod-inventory. That is 20% higher than in Orchid release. But the rest of services were almost on the same level in the same test in Orchid  Data Import with Check-ins Check-outs Orchid and didn't exceed 60%.

Spikes of  mod-data-import observed in Data Import jobs with 50k files up to 130%. for jobs  and 320% spike for 100k. For Data Import jobs with 5k, 10k, 25k files CPU utilization didn't exceed 35%

DI MARC BIB Create + CICO

DI MARC BIB Update+ CICO

Service Memory Utilization

There is memory utilization increasing observed which is caused by previous modules restarting (everyday cluster shut down process).

DI MARC BIB Create + CICO

It was observed that during tests mod-source-record-storage grew every 30 minutes with 10% from 38% up to 67% but it's still less than in the same test in Orchid. 

Memory consumption before tests for mod-search was 39% and for mod-inventory - 75%. During test with 100k file mod-search grew up to 75% and mod-inventory up to 91%.

MARC BIB Update + CICO

After 30 minutes of tests start mod-source-record-storage grew from 67% to 75% and didn't change during tests with 50k and 100k. So 75% of memory consumption can be defined as a baseline under load for this service. Mod-inventory grew from 91% to 99%.

DB CPU Utilization

Average DB CPU usage during data import is about 95% The same results if to compare with the same tests in Orchid.

DI MARC BIB Create + CICO

MARC BIB Update + CICO

DB Connections

Average connection count during data import is about 300 connections for create jobs that is 30 connections higher than in Orchid. For update jobs - 280 connections that is higher 

DI MARC BIB Create + CICO

MARC BIB Update + CICO

DB load

DI MARC BIB Create + CICO

Top SQL-queries:

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

UPDATE fs09000000_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

MARC BIB Update + CICO

Top SQL-queries:

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

Appendix

Infrastructure

PTF -environment pcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, writer/reader

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	2731

• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Methodology/Approach

DI tests were started from UI with 5 min pauses between the tests.

Additional links

Grafana dashboard:

MARC Bib Create + CICO

http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1700738030629&to=1700749313428&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All

MARC Bib Update + CICO

http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1700755239000&to=1700766759000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All