/
[Quesnelia] [non-ECS] Multi-tenants DI

[Quesnelia] [non-ECS] Multi-tenants DI

Apr 29, 2024

Overview

  • This document contains the results of testing Data Import for MARC Bibliographic records with an update job in the Quesnelia release on qcp1 environments with Kafka consolidated topics and file splitting features enabled on a non-ecs environment.

https://folio-org.atlassian.net/browse/PERF-818 

Summary

  • Three tests of DI with update jobs with 25k records were carried out on the qcp1 cluster.

  • Comparing the Quesnelia and Poppy releases, we observed improved DI update job durations in the Quesnelia environment. On average, these durations were 1 minute and 23 seconds shorter for a single tenant and 2 minutes shorter for two and three concurrent tenants in the Quesnelia environment.

  • Memory consumption for mod-data-import was 60%, mod-inventory - 128%. No memory leaks are suspected.

  • CPU utilization for mod-data-import spiked to 70%, mod-inventory - 150%.

  • DB CPU utilization was 93%.

  • DB connections - 1273 on average.

Test Results

Profile

Test #

Tenant

MARC File

DI Duration Quesnelia (hh:mm:ss)

Results

Profile

Test #

Tenant

MARC File

DI Duration Quesnelia (hh:mm:ss)

Results

DI MARC Bib Update (PTF - Updates Success - 6)

1

fs09000000

25K.mrc

00:15:55

Completed

2

fs09000000

25K.mrc

00:17:06

Completed

fs07000001

25K.mrc

00:28:17

Completed

3

fs09000000

25K.mrc

00:33:57

Completed

fs07000001

25K.mrc

00:36:07

Completed

fs07000002

25K.mrc

00:41:55

Completed

Comparison

Test #

Quesnelia

Poppy

DI Delta
per tenant
(hh:mm:ss) Quesnelia /Poppy

DI Delta
per test
(hh:mm:ss) Quesnelia /Poppy

Profile

Tenant

DI Duration Quesnelia

per tenant

(hh:mm:ss)

DI Duration Quesnelia

per test

(hh:mm:ss)

Profile

Tenant

DI Duration Poppy (hh:mm:ss)

DI Duration Poppy

per test

(hh:mm:ss)

№1

25K.mrc

DI MARC Bib Update

(PTF - Updates Success - 6)

fs09000000

00:15:55

00:15:55

DI MARC Bib Update

(PTF - Updates Success - 1)

fso9000000

00:17:18

00:17:18

-0:01:23

-0:01:23

№2

25K.mrc

fs09000000

00:17:06

00:45:23

fso9000000

00:23:11

00:47:27

-0:06:05

-0:02:04

fs07000001

00:28:17

fso7000001

00:24:16

+0:04:01

№3

25K.mrc

fs09000000

00:33:57

1:51:59

fso9000000

00:26:36

1:54:01

+0:07:21

-0:02:02

fs07000001

00:36:07

fso7000001

00:38:01

-0:01:54

fs07000002

00:41:55

fso7000002

00:49:24

-0:07:29

 

Resource utilization for Test #1 and Test #2

CPU

RAM

CPU

RAM

mod-inventory-b

171%

mod-inventory-b

87%

mod-di-converter-storage-b

83%

mod-permissions-b

79%

nginx-okapi

66%

mod-data-import-b

60%

mod-source-record-storage-b

39%

okapi-b

58%

okapi-b

29%

mod-source-record-storage-b

48%

mod-inventory-storage-b

20%

mod-inventory-storage-b

44%

mod-source-record-manager-b

20%

mod-source-record-manager-b

43%

mod-permissions-b

19%

mod-di-converter-storage-b

40%

mod-data-import-b

1.%

nginx-okapi

5%

pub-okapi

0.2%

pub-okapi

4.7%

Service CPU Utilization

Here we can see that mod-inventory-b module used 171% CPU, mod-di-converter-storage-b used 83% CPU and nginx-okapi used 66% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU in the average was 88%.

DB Connections

DB connections was 1139.

DB load

Top SQL-queries

 

#

TOP 5 SQL statements

#

TOP 5 SQL statements

1

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

2

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

3

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

4

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

5

set SEARCH_PATH to 'fs09000000_mod_source_record_manager'; SELECT * FROM get_job_log_entries('b7d9c003-eff8-42a7-8a77-1473defa0a84','source_record_order', 'ASC', 100, 0, false, 'ALL')

 

Resource utilization for Test #3

CPU

RAM

CPU

RAM

mod-di-converter-storage-b

137%

mod-inventory-b

67%

mod-inventory-b

117%

mod-permissions-b

61%

nginx-okapi

109%

mod-source-record-storage-b

45%

mod-data-import-b

55%

mod-source-record-manager-b

44%

okapi-b

46%

okapi-b

38%

mod-source-record-storage-b

44%

mod-data-import-b

36%

mod-source-record-manager-b

34%

mod-di-converter-storage-b

34%

mod-inventory-storage-b

20%

mod-inventory-storage-b

16%

mod-permissions-b

18.00%

nginx-okapi

4%

pub-okapi

0.30%

pub-okapi

4%

Service CPU Utilization

Here we can see that mod-inventory-b module used 140% CPU in average, mod-inventory-b used 117% CPU and nginx-okapi used 109% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 93%.

DB Connections

DB connections was 1273.

DB load

Top SQL-queries

#

TOP 5 SQL statements

#

TOP 5 SQL statements

1

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

2

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

3

INSERT INTO fs07000001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

4

INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

5

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

Appendix

Infrastructure

PTF - environment Quesnelia (qcp1)

  • 10 db.r6g.xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instances, writer

  • MSK ptf-mobius-testing2

    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=2

 

Module

Task Def. Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

qcp1-pvt

Tue Apr 16 14:25:54 UTC 2024

mod-remote-storage

3

579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.2.0

2

4920

4472

1024

3960

512

512

FALSE

mod-ncip

3

579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.14.4

2

1024

896

128

768

88

128

FALSE

mod-finance-storage

3

579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.6.0

2

1024