[Quesnelia] [non-ECS] Multi-tenants DI

Overview

  • This document contains the results of testing Data Import for MARC Bibliographic records with an update job in the Quesnelia release on qcp1 environments with Kafka consolidated topics and file splitting features enabled on a non-ecs environment.

PERF-818 - Getting issue details... STATUS  

Summary

  • Three tests of DI with update jobs with 25k records were carried out on the qcp1 cluster.
  • Comparing the Quesnelia and Poppy releases, we observed improved DI update job durations in the Quesnelia environment. On average, these durations were 1 minute and 23 seconds shorter for a single tenant and 2 minutes shorter for two and three concurrent tenants in the Quesnelia environment.
  • Memory consumption for mod-data-import was 60%, mod-inventory - 128%. No memory leaks are suspected.
  • CPU utilization for mod-data-import spiked to 70%, mod-inventory - 150%.
  • DB CPU utilization was 93%.
  • DB connections - 1273 on average.

Test Results

ProfileTest #TenantMARC File

DI Duration Quesnelia (hh:mm:ss)

Results

DI MARC Bib Update (PTF - Updates Success - 6)1fs0900000025K.mrc00:15:55Completed
2fs0900000025K.mrc00:17:06Completed
fs0700000125K.mrc00:28:17Completed
3fs0900000025K.mrc00:33:57Completed
fs0700000125K.mrc00:36:07Completed
fs0700000225K.mrc00:41:55
Completed

Comparison

Test #QuesneliaPoppy

DI Delta
per tenant
(hh:mm:ss) Quesnelia /Poppy

DI Delta
per test
(hh:mm:ss) Quesnelia /Poppy
ProfileTenant

DI Duration Quesnelia

per tenant

(hh:mm:ss)

DI Duration Quesnelia

per test

(hh:mm:ss)

Profile

Tenant

DI Duration Poppy (hh:mm:ss)

DI Duration Poppy

per test

(hh:mm:ss)

№1

25K.mrc

DI MARC Bib Update

(PTF - Updates Success - 6)

fs0900000000:15:5500:15:55

DI MARC Bib Update

(PTF - Updates Success - 1)

fso900000000:17:1800:17:18-0:01:23-0:01:23

№2

25K.mrc

fs0900000000:17:0600:45:23fso900000000:23:1100:47:27-0:06:05-0:02:04
fs0700000100:28:17fso700000100:24:16+0:04:01

№3

25K.mrc

fs0900000000:33:571:51:59fso900000000:26:361:54:01+0:07:21-0:02:02
fs0700000100:36:07fso700000100:38:01-0:01:54
fs0700000200:41:55
fso700000200:49:24-0:07:29


Resource utilization for Test #1 and Test #2

 Resource utilization table
CPURAM
mod-inventory-b171%mod-inventory-b87%
mod-di-converter-storage-b83%mod-permissions-b79%
nginx-okapi66%mod-data-import-b60%
mod-source-record-storage-b39%okapi-b58%
okapi-b29%mod-source-record-storage-b48%
mod-inventory-storage-b20%mod-inventory-storage-b44%
mod-source-record-manager-b20%mod-source-record-manager-b43%
mod-permissions-b19%mod-di-converter-storage-b40%
mod-data-import-b1.%nginx-okapi5%
pub-okapi0.2%pub-okapi4.7%

Service CPU Utilization

Here we can see that mod-inventory-b module used 171% CPU, mod-di-converter-storage-b used 83% CPU and nginx-okapi used 66% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU in the average was 88%.

DB Connections

DB connections was 1139.

DB load

                                                                                                                     

Top SQL-queries


#TOP 5 SQL statements
1
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
2
INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
3
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
4
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
5
set SEARCH_PATH to 'fs09000000_mod_source_record_manager';
SELECT * FROM get_job_log_entries('b7d9c003-eff8-42a7-8a77-1473defa0a84','source_record_order', 'ASC', 100, 0, false, 'ALL')


Resource utilization for Test #3

 Resource utilization table
CPURAM
mod-di-converter-storage-b137%mod-inventory-b67%
mod-inventory-b117%mod-permissions-b61%
nginx-okapi109%mod-source-record-storage-b45%
mod-data-import-b55%mod-source-record-manager-b44%
okapi-b46%okapi-b38%
mod-source-record-storage-b44%mod-data-import-b36%
mod-source-record-manager-b34%mod-di-converter-storage-b34%
mod-inventory-storage-b20%mod-inventory-storage-b16%
mod-permissions-b18.00%nginx-okapi4%
pub-okapi0.30%pub-okapi4%

Service CPU Utilization

Here we can see that mod-inventory-b module used 140% CPU in average, mod-inventory-b used 117% CPU and nginx-okapi used 109% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 93%.

DB Connections

DB connections was 1273.

DB load

                                                                                                                     

Top SQL-queries

#TOP 5 SQL statements
1
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
2
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
3
INSERT INTO fs07000001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
4
INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
5
INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

Appendix

Infrastructure

PTF - environment Quesnelia (qcp1)

  • 10 db.r6g.xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 1 database  instances, writer

    NameMemory GIBvCPUs

    db.r6g.xlarge

    32 GiBvCPUs
  • MSK ptf-mobius-testing2
    • 4 m5.2xlarge brokers in 2 zones
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=2


 Quesnelia modules memory and CPU parameters
ModuleTask Def. Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

qcp1-pvt
Tue Apr 16 14:25:54 UTC 2024
mod-remote-storage3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.2.024920447210243960512512FALSE
mod-ncip3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.14.42102489612876888128FALSE
mod-finance-storage3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.6.021024896102470088128FALSE
mod-agreements3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:7.0.0215921488128000FALSE
mod-ebsconet3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.2.0212481024128700128256FALSE
mod-organizations3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:1.9.02102489612870088128FALSE
mod-consortia2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-consortia:1.1.023072204812820485121024FALSE
edge-sip22579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-sip2:3.2.0-SNAPSHOT.2092102489612876888128FALSE
mod-serials-management3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-management:1.0.02248023121281792384512FALSE
mod-settings3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.0.32102489620076888128FALSE
mod-data-import6579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:3.1.01204818442561292384512FALSE
edge-dematic3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematic:2.2.01102489612876888128FALSE
mod-search3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:3.2.0225922480204814405121024FALSE
mod-inn-reach2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inn-reach:3.2.0-SNAPSHOT.86236003240102428805121024FALSE
mod-tags3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tags:2.2.0210248961287688