Versions Compared

Old Version 1

changes.mady.by.user Mykhailo Petryshyn

Saved on

compared with

New Version 2

changes.mady.by.user Mykhailo Petryshyn

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Table of Contents

Overview

This document contains the results of testing Data Export tests on 61 tenants (excluding the central tenant) on the MCPT environment, this report is the second part of the DE testing, the previous report is available https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/7864374/Data+Export+Test+Report+Mobius#Errors-/-Additional-information :

  • T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).

Expand
titleRequirement test run 1

The member tenant with the largest data set runs export as described in Scenario 2 using the default export profile(DEP). The remaining member tenants run exports as described in Scenario 1, also using the default job profile. All exports are run concurrently (or almost concurrently) on all tenants.
Scenario 1:

  1. Member tenants: 500-2K instance records;

  2. Profiles: Default instances export profile

...

Ticket:

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-803

Summary

Data-export duration statistics:

 

Data export

T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).

T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP).

T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)

500,

sec

1000 ,

sec

2000 ,

sec

500 ,

sec

1000 ,

sec

2000,

sec

500 ,

sec

1000 ,

sec

2000,

sec

Average duration

80

256

830

956

2110

4441

215

533

1281

Minimal duration

21

19

74

384

982

1863

45

14

12

Maximal duration

1028

360

987

1075

2257

4624

254

558

1374

75t percentile

77

339

968

1029

2223

4603

244

657

1362

90t percentile

108

356

982

1037

2241

4618

249

661

1368

CS42(MAX RECORDS)

1 hrs 56 min

1 hrs 49 min

2hrs 18 min

2hrs 10 min

2hrs 30 min

1 hrs 49 min

32 min 7 sec

44 min 3 sec

55 min 31 sec

  • T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)

500

1000

2000

MRC

10 seconds

35 seconds

1 minutes 45 seconds

Holdings

8,3 seconds

26 seconds

1 minutes 43 seconds

Holdings MAX

43 minutes 26 seconds

41 minutes 16 seconds

40 minutes 55 seconds

authorityCS00

36 minutes 2 seconds

35 minutes 37 seconds

36 minutes 39 seconds

  • T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

Tenant cs00000001_0003

Test 3. Data-export . Max records. Default export profile

"2024-02-23T11:02:11.934+00:00"

"2024-02-23T11:29:32.025+00:00"

00:27:20.091

Test 3. Data-export . 2000 records. Custom job profile

"2024-02-23T11:02:00.802+00:00"

"2024-02-23T11:03:30.687+00:00"

00:01:29.885

Test 2. Data-export . Max records. Default export profile

"2024-02-23T10:26:59.739+00:00"

"2024-02-23T10:53:33.241+00:00"

00:26:33.502

Test 2. Data-export . 1000 records. Custom job profile

"2024-02-23T10:26:43.425+00:00"

"2024-02-23T10:27:25.347+00:00"

00:00:41.922

Test 1. Data-export . Max records. Default export profile

"2024-02-23T09:56:41.692+00:00"

"2024-02-23T10:23:11.426+00:00"

00:26:29.734

Test 1. Data-export . 500 records. Custom job profile

"2024-02-23T09:56:20.276+00:00"

"2024-02-23T09:56:39.402+00:00"

00:00:19.126

Recommendations & Jiras

  • Investigate Data-exports job that have “FAILED“ status and <Code>NoSuchKey</Code> <Message>The specified key does not exist.</Message>

  • Run data-import and data-export of authority records with default profiles to reproduce 5xx errors on okapi and mod-entity-link high memory utilization.


Test Results

  • T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). This table contains durations for Data-export for jobs with “Default instances export profile” on 61 tenants with 3 sets of data(500 records; 1000 records; 2000 records) except tenant 42, on this tenant data-export run with the largest number of instances and “Default instances export profile”. Additional information about failed jobs is in Additional information/Error section

...

Tenant

Started Date

Compl Date

Duration

Started Date

Compl Date

Duration

Started Date

Compl Date

Duration

 cs00000001

14:30:48.756

15:02:56.495

00:32:07.739

15:25:21.738

16:09:25.170

00:44:03.432

17:29:00.215

18:24:32.154

00:55:31.939

 cs00000001_0001

14:24:42.318

14:24:42.318

00:00:45.211

15:25:15.665

15:31:19.954

00:06:04.289

17:28:49.284

17:44:36.536

00:15:47.252

 cs00000001_0002

14:24:43.082

14:25:31.223

00:00:48.141

15:25:17.867

15:32:31.980

00:07:14.113

17:28:50.497

17:45:49.205

00:16:58.708

 cs00000001_0003

14:24:45.343

14:25:45.888

00:01:00.545

15:25:19.490

15:32:56.805

00:07:37.315

17:28:52.298

17:46:17.206

00:17:24.908

 cs00000001_0004

14:24:47.523

14:26:21.848

00:01:34.325

15:25:19.468

15:32:53.154

00:07:33.686

17:28:55.078

17:47:24.722

00:18:29.644

 cs00000001_0005

14:24:47.497

14:26:03.472

00:01:15.975

15:25:26.453

15:34:08.759

00:08:42.306

17:28:55.055

17:47:08.115

00:18:13.06

 cs00000001_0006

14:24:51.872

14:26:58.001

00:02:06.129

15:25:34.475

15:35:08.741

00:09:34.266

17:29:10.529

17:49:43.272

00:20:32.743

 cs00000001_0007

14:24:51.900

14:27:00.778

00:02:08.878

15:25:39.336

15:35:31.356

00:09:52.02

17:29:10.505

17:49:42.671

00:20:32.166

 cs00000001_0008

14:24:55.497

14:27:40.752

00:02:45.255

15:25:39.358

15:35:36.215

00:09:56.857

17:29:15.627

17:50:15.416

00:20:59.789

 cs00000001_0009

14:24:59.483

14:27:54.691

00:02:55.208

15:25:39.876

15:35:43.352

00:10:03.476

17:29:17.275

17:50:33.670

00:21:16.395

 cs00000001_0010

14:24:59.508

14:28:06.621

00:03:07.113

15:25:41.078

15:35:43.957

00:10:02.879

17:29:22.272

17:50:51.494

00:21:29.222

 cs00000001_0011

14:25:00.195

14:28:14.477

00:03:14.282

15:25:41.086

15:35:49.502

00:10:08.416

17:29:22.244

17:50:48.353

00:21:26.109

 cs00000001_0012

14:25:04.852

14:28:20.675

00:03:15.823

15:25:47.008

15:36:04.014

00:10:17.006

17:29:28.675

17:51:20.287

00:21:51.612

 cs00000001_0013

14:25:05.432

14:28:35.978

00:03:30.546

15:25:46.416

15:36:03.250

00:10:16.834

17:29:29.703

17:51:24.819

00:21:55.116

 cs00000001_0014

14:25:07.135

14:28:41.482

00:03:34.347

15:25:50.212

15:36:16.623

00:10:26.411

17:29:29.732

17:51:24.821

00:21:55.089

 cs00000001_0015

14:25:11.505

14:28:49.677

00:03:38.172

15:25:50.239

15:36:17.917

00:10:27.678

17:29:38.216

17:51:53.405

00:22:15.189

 cs00000001_0016

14:25:11.481

14:28:47.521

00:03:36.04

15:25:55.589

15:36:33.118

00:10:37.529

17:29:37.763

17:51:49.102

00:22:11.339

 cs00000001_0017

14:25:13.285

14:29:03.071

00:03:49.786

15:25:56.102

15:36:34.963

00:10:38.861

17:29:37.763

17:51:49.103

00:22:11.34

 cs00000001_0018

14:25:18.535

14:29:07.800

00:03:49.265

15:25:58.614

15:36:44.878

00:10:46.264

17:29:47.800

17:52:17.763

00:22:29.963

 cs00000001_0019

14:25:20.256

14:29:13.686

00:03:53.43

15:26:06.052

15:36:52.652

00:10:46.6

17:29:48.419

17:52:20.725

00:22:32.306

 cs00000001_0020

14:25:20.281

14:29:21.852

00:04:01.571

15:26:06.028

15:36:51.427

00:10:45.399

17:29:48.396

17:52:17.762

00:22:29.366

 cs00000001_0021

14:25:28.404

14:29:28.699

00:04:00.295

15:26:07.660

15:36:57.123

00:10:49.463

17:29:48.995

17:52:23.675

00:22:34.68

 cs00000001_0022

14:25:30.184

14:29:30.962

00:04:00.778

15:26:09.197

15:37:08.209

00:10:59.012

17:29:49.018

17:52:23.676

00:22:34.658

 cs00000001_0023

14:25:30.184

14:29:31.495

00:04:01.311

15:26:11.778

15:37:10.213

00:10:58.435

17:30:01.919

17:52:42.840

00:22:40.921

 cs00000001_0024

14:25:30.184

14:29:30.435

00:04:00.251

15:26:19.761

15:37:13.777

00:10:54.016

17:30:01.941

17:52:43.450

00:22:41.509

 cs00000001_0025

14:25:30.184

14:29:31.495

00:04:01.311

15:26:19.783

15:37:16.180

00:10:56.397

17:30:03.091

17:52:49.336

00:22:46.245

 cs00000001_0026

14:25:31.763

14:29:44.320

00:04:12.557

15:26:20.346

15:37:16.180

00:10:55.834

17:30:03.112

17:52:50.019

00:22:46.907

 cs00000001_0027

14:25:36.718

14:29:47.116

00:04:10.398

15:26:23.529

15:37:29.781

00:11:06.252

17:30:17.486

17:53:04.582

00:22:47.096

 cs00000001_0028

14:25:43.614

14:29:47.648

00:04:04.034

15:26:25.997

15:37:31.677

00:11:05.68

17:30:17.536

17:53:04.583

00:22:47.047

 cs00000001_0029

14:25:43.635

14:29:48.171

00:04:04.536

15:26:36.318

15:37:33.537

00:10:57.219

17:30:17.587

17:53:05.213

00:22:47.626

 cs00000001_0030

14:25:43.589

14:29:47.648

00:04:04.059

15:26:37.039

15:37:38.474

00:11:01.435

17:30:19.321

17:53:11.505

00:22:52.184

 cs00000001_0031

14:25:43.567

14:29:47.645

00:04:04.078

15:26:36.341

15:37:35.539

00:10:59.198

17:30:28.307

17:53:16.349

00:22:48.042

 cs00000001_0032

14:25:44.235

14:29:49.921

00:04:05.686

15:26:36.992

15:37:36.079

00:10:59.087

17:30:33.304

17:53:16.349

00:22:43.045

 cs00000001_0033

14:25:46.439

14:30:01.215

00:04:14.776

15:26:37.017

15:37:36.079

00:10:59.062

17:30:33.326

17:53:24.641

00:22:51.315

 cs00000001_0034

14:25:53.953

14:30:02.375

00:04:08.422

15:26:40.769

15:37:48.169

00:11:07.4

17:30:33.350

17:53:25.842

00:22:52.492

 cs00000001_0035

14:26:00.662

14:30:02.888

00:04:02.226

15:26:40.793

15:37:48.910

00:11:08.117

17:30:35.980

17:53:30.121

00:22:54.141

 cs00000001_0036

14:26:00.600

14:30:02.888

00:04:02.288

15:26:48.214

15:37:48.911

00:11:00.697

17:30:36.034

17:53:30.122

00:22:54.088

 cs00000001_0037

14:26:00.627

14:30:02.887

00:04:02.26

15:26:53.216

15:37:49.487

00:10:56.271

17:30:40.561

17:53:31.268

00:22:50.707

 cs00000001_0038

14:26:00.707

14:30:04.701

00:04:03.994

15:26:56.783

15:37:50.130

00:10:53.347

17:30:50.085

17:53:32.414

00:22:42.329

 cs00000001_0039

14:26:00.685

14:30:03.360

00:04:02.675

15:26:57.421

15:37:52.235

00:10:54.814

17:30:50.029

17:53:32.414

00:22:42.385

 cs00000001_0040

14:26:02.068

14:30:10.702

00:04:08.634

15:26:57.458

15:37:59.809

00:11:02.351

17:30:50.106

17:53:33.585

00:22:43.479

 cs00000001_0041

14:26:03.496

14:30:13.924

00:04:10.428

15:26:09.179

15:26:24.068

00:00:14.889

17:30:50.051

17:53:32.414

00:22:42.363

 cs00000001_0042

14:25:44.235

14:29:49.921

00:04:05.686

15:26:46.587

15:36:44.876

00:09:58.289

17:29:39.707

17:29:51.801

00:00:12.094

 cs00000001_0043

14:26:11.198

14:30:14.616

00:04:03.418

15:27:01.669

15:38:02.854

00:11:01.185

17:30:10.228

17:39:28.947

00:09:18.719

 cs00000001_0044

14:26:15.107

14:30:14.619

00:03:59.512

15:27:02.233

15:38:02.854

00:11:00.621

17:31:04.416

17:53:43.867

00:22:39.451

 cs00000001_0045

14:26:18.467

14:30:14.618

00:03:56.151

15:27:03.317

15:38:03.511

00:11:00.194

17:31:15.533

17:53:45.158

00:22:29.625

 cs00000001_0046

14:26:20.713

14:30:14.619

00:03:53.906

15:27:13.457

15:38:03.511

00:10:50.054

17:31:15.507

17:53:45.157

00:22:29.65

 cs00000001_0047

14:26:21.798

14:30:14.620

00:03:52.822

15:27:15.707

15:38:03.511

00:10:47.804

17:31:15.481

17:53:45.157

00:22:29.676

 cs00000001_0048

14:26:21.241

14:30:14.620

00:03:53.379

15:27:18.004

15:38:03.512

00:10:45.508

17:31:15.559

17:53:49.697

00:22:34.138

 cs00000001_0049

14:26:21.821

14:30:15.961

00:03:54.14

15:27:20.942

15:38:03.522

00:10:42.58

17:31:20.125

17:53:53.650

00:22:33.525

 cs00000001_0050

14:26:21.846

14:30:22.124

00:04:00.278

14:26:23.532

14:30:22.113

00:03:58.581

17:31:20.147

17:53:54.706

00:22:34.559

 cs00000001_0051

14:26:03.496

14:26:01.226

00:04:10.853

14:26:24.155

14:30:22.114

00:03:57.959

17:31:24.154

17:53:55.404

00:22:31.25

 cs00000001_0052

14:25:44.209

14:30:14.616

00:04:09.499

14:26:28.901

14:30:22.114

00:03:53.213

17:31:24.730

17:53:55.405

00:22:30.675

 cs00000001_0053

14:26:11.198

14:30:14.619

00:04:09.56

14:26:38.502

14:30:22.120

00:03:43.618

17:31:34.686

17:53:55.405

00:22:20.719

 cs00000001_0054

14:26:15.107

14:30:14.618

00:04:08.079

14:26:37.825

14:30:22.114

00:03:44.289

17:31:45.822

17:53:55.405

00:22:09.583

 cs00000001_0055

14:26:18.467

14:30:14.619

00:03:34.212

14:26:40.893

14:30:22.117

00:03:41.224

17:31:46.415

17:54:00.244

00:22:13.829

 cs00000001_0056

14:26:20.713

14:30:14.620

00:03:57.999

14:26:40.849

14:30:22.117

00:03:41.268

17:31:45.788

17:53:55.405

00:22:09.617

 cs00000001_0057

14:26:21.798

14:30:14.620

00:03:58.051

14:26:41.705

14:30:22.117

00:03:40.412

17:31:45.847

17:53:56.620

00:22:10.773

 cs00000001_0058

14:26:21.241

14:30:15.961

00:03:33.063

14:26:44.945

14:30:22.117

00:03:37.172

17:31:45.868

17:53:56.619

00:22:10.751

 cs00000001_0059

14:26:21.821

14:30:22.124

00:03:58.146

14:26:46.887

14:30:22.123

00:03:35.236

17:31:55.343

17:54:00.245

00:22:04.902

 cs00000001_0060

14:26:21.846

14:30:10.702

00:03:57.914

14:26:44.988

14:30:22.121

00:03:37.133

17:31:50.896

17:54:00.245

00:22:09.349

 cs00000001_0061

14:26:00.707

14:30:13.924

00:03:58.086

14:26:37.825

14:30:22.114

00:03:44.289

17:31:53.075

17:54:00.245

00:22:07.17

Resource Utilization

Service CPU Utilization

  • T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). Service CPU usage during the test was quite low, only the mod-data-export-b okapi-b mod-permissions-b services had a slight increase up to 18%. The test lasted after 23:15 bur there were some spikes from another test. but the CPU usage of selected services was the same.

...

  • T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

...

Memory Utilization

  • T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).

...

  • T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

...

DB CPU Utilization

  • T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). Except for the database spikes that every 30 minutes. CPU usage before tets was about 2.5%, during all Data Export of 500, 1000, and 2000 instances, CPU usage increased up to 4%;

...

  • T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

...

DB Connections

  • T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). The maximal number of DB connections was 1342 for 2000 DE.

...

  • T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

...

DB Load

  • T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).

...

  • T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

...

SQL queries

  • T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).

...

  • T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

...

Additional information

Errors

Data-exports jobs failed with the same errors.

...

Expand

This XML file does not appear to have any style information associated with it. The document tree is shown below.

<Error>

<Code>NoSuchKey</Code>

<Message>The specified key does not exist.</Message>

<Key>cs00000001_0010/3fa36a65-40fa-49d9-b028-62485ed48c1a/cs00000001_0010_1k_mrc_bib-368.mrc</Key>

<RequestId>44DP9SJ9FK9RJ4GP</RequestId>

<HostId>tzFaRcMEHCtq7ZuHDxueqpq2uE+w6W7S2jGSoRSRNskejfvcI4T3azSiy786C26Z/Io/oOjSgzA=</HostId>

</Error>

Methodology/Approach

All tests were run using the JMeter script. Data has been pre-generated for all tenants

Infrastructure

PTF -environment mcpt

  • 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database instance, writer

    Name Memory GIBv CPUsmax_connections

    db.r6g.4xlarge 128 GiB; 16 vCPU

  • MSK tenant

    • 2 m5.2xlarge brokers in 1 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

The table contains modules, memory, and CPU parameters

Module

Task Def. Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

mcpt-pvt 

mod-consortia

11

mod-consortia:1.0.3

2

2048

1802

128

768

88

128

FALSE

mod-organizations

6

mod-organizations:1.8.0

2

1024

896

128

700

88

128

FALSE

mod-data-import

7

mod-data-import:3.0.7

1

2048

1844

256

1292

384

512

FALSE

mod-search

12

mod-search:3.0.5

2

2592

2480

2048

1440

512

1024

FALSE

mod-configuration

6

mod-configuration:5.9.2

2

1024

896

128

768

88

128

FALSE

mod-permissions

14

mod-permissions:6.4.0

2

1684

1544

512

1024

384

512

FALSE

mod-entities-links

10

mod-entities-links:2.0.4

2

2592

2480

400

1440

0

1024

FALSE

mod-inventory-storage

10

mod-inventory-storage:27.0.4

2

4096

3690

2048

3076

384

512

FALSE

mod-data-export-worker

9

mod-data-export-worker:3.1.2

2

3072

2800

1024

2048

384

512

FALSE

mod-source-record-storage

10

mod-source-record-storage:5.7.5

2

5600

5000

2048

3500

384

512

FALSE

mod-inventory

10

mod-inventory:20.1.6

2

2880

2592

1024

1814

384

512

FALSE

mod-di-converter-storage

9

mod-di-converter-storage:2.1.5

2

1024

896

128

768

88

128

FALSE

mod-users

11

mod-users:19.2.2

2

1024

896

128

768

88

128

FALSE

mod-source-record-manager

10

mod-source-record-manager:3.7.7

2

5600

5000

2048

3500

384

512

FALSE

mod-quick-marc

10

mod-quick-marc:5.0.1

1

2288

2176

128

1664

384

512

FALSE

nginx-okapi

2

nginx-okapi:2023.06.14

2

1024

896

512

0

0

0

FALSE

okapi-b

1

okapi:5.1.2

3

1684

1440

1024

922

384

512

FALSE

mod-data-export

10

mod-data-export:4.8.7

1

1024

896

1024

768

88

128

FALSE