Table of Contents

Overview

...

Ticket:

...

There is significant performance improvement for data import in Poppy comparing with Orchid. Durations are closer to Nolana release results. Response CI/CO response times improved 10% in average for with all DI jobs. There's only CI response time degraded in 100k job with 17%.

Comparing Orchid and Poppy releases DI durations in create jobs are up to 20% 10% higher with 5k file and 5% for bigger files with parallel Check-in/Check-out than pure Data import results. DI durations are higher up to 30% in update jobs.

...

Average CPU utilization increased for mod-inventory up to 20% comparing with Orchid. So it did not exceed 150% for all the modules. The highest consumption observed from mod-inventory. The rest of services were almost on the same level in the same test in Orchid  Data Import with Check-ins Check-outs Orchid and didn't exceed 60%.

DB needs more connections (in average +20 more) needed for the same tests as in Orchid for all create and update jobs.

...

Upd: During previous tests on pcp1 and ocp3 there were problems with DI jobs running big files (Create jobs 100k and higher, Update jobs with 25k and higher). The problem was solved after new deployment and updates of 13 modules in scope of ticket 

...

Average CPU utilization did not exceed 150% for all the modules. The highest consumption observed from mod-inventory. That is 20% higher than in Orchid release. But the rest of services were almost on the same level in the same test in Orchid  Data Import with Check-ins Check-outs Orchid and didn't exceed 60%.

Spikes of  mod-data-import observed in Data Import jobs with 50k files up to 130%. for jobs  and 320% spike for 100k. For Data Import jobs with 5k, 10k, 25k files CPU utilization didn't exceed 35%

...

DI MARC BIB Create + CICO

Top SQL-queries:

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

UPDATE fs09000000_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

MARC BIB Update + CICO

Top SQL-queries:

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

Appendix

Infrastructure

PTF -environment pcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, writer/reader

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

...