Table of Contents

Overview

This document contains the results of testing Data Export tests on 61 tenants (excluding the central tenant) on the MCPT environment, this report is the second part of the DE testing, the previous report is available https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/7864374/Data+Export+Test+Report+Mobius#Errors-/-Additional-information :

T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).

Expand

title	Requirement test run 1

The member tenant with the largest data set runs export as described in Scenario 2 using the default export profile(DEP). The remaining member tenants run exports as described in Scenario 1, also using the default job profile. All exports are run concurrently (or almost concurrently) on all tenants.
Scenario 1:

Member tenants: 500-2K instance records;
Profiles: Default instances export profile

...

Ticket:

Jira Legacy

server	System Jira
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-803

Summary

The tested Data-export jobs are quite light for the system and cause an increase in CPU on the services: mod-data-export, okapi and mod-inventory-storage up to 17%;
No memory leaks, memory consumption was stable during all of the tests;
Database CPU usage varied up to a maximum of 8%;
The number of failed DE jobs during concurrent testing was about 7% all with the same errors, but the duration of failed jobs was like the duration of completed, error messages are at the end of the report
The behavior of the export date was the same as in the first part of the report, with the latest processes starting with a delay. On the first tenants, where the file with instance ids was downloaded the data-export process is completed the fastest, but the data-export on the last tenants lasts the longest, as can be seen from the graphs below the tables
General metrics can be found in the tables below

Data-export duration statistics:

Data export	T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).			T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP).			T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)
Data export	500, sec	1000 , sec	2000 , sec	500 , sec	1000 , sec	2000, sec	500 , sec	1000 , sec	2000, sec
Average duration	80	256	830	956	2110	4441	215	533	1281
Minimal duration	21	19	74	384	982	1863	45	14	12
Maximal duration	1028	360	987	1075	2257	4624	254	558	1374
75t percentile	77	339	968	1029	2223	4603	244	657	1362
90t percentile	108	356	982	1037	2241	4618	249	661	1368
CS42(MAX RECORDS)	1 hrs 56 min	1 hrs 49 min	2hrs 18 min	2hrs 10 min	2hrs 30 min	1 hrs 49 min	32 min 7 sec	44 min 3 sec	55 min 31 sec

T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)

500 records

1000 records

2000 records

MRC

MARC	10 seconds	35 seconds	1 minutes 45 seconds
Holdings	8,3 seconds	26 seconds	1 minutes 43 seconds
Holdings MAX	43 minutes 26 seconds	41 minutes 16 seconds	40 minutes 55 seconds
authorityCS00	36 minutes 2 seconds	35 minutes 37 seconds	36 minutes 39 seconds

T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

Tenant cs00000001_0003	Starterted time	Finished time	Duration
Test 3. Data-export . Max records. Default export profile	"2024-02-23T11:02:11.934+00:00"	"2024-02-23T11:29:32.025+00:00"	00:27:20.091
Test 3. Data-export . 2000 records. Custom job profile	"2024-02-23T11:02:00.802+00:00"	"2024-02-23T11:03:30.687+00:00"	00:01:29.885
Test 2. Data-export . Max records. Default export profile	"2024-02-23T10:26:59.739+00:00"	"2024-02-23T10:53:33.241+00:00"	00:26:33.502
Test 2. Data-export . 1000 records. Custom job profile	"2024-02-23T10:26:43.425+00:00"	"2024-02-23T10:27:25.347+00:00"	00:00:41.922
Test 1. Data-export . Max records. Default export profile	"2024-02-23T09:56:41.692+00:00"	"2024-02-23T10:23:11.426+00:00"	00:26:29.734
Test 1. Data-export . 500 records. Custom job profile	"2024-02-23T09:56:20.276+00:00"	"2024-02-23T09:56:39.402+00:00"	00:00:19.126

Recommendations & Jiras

Investigate Data-exports job that have “FAILED“ status and <Code>NoSuchKey</Code> <Message>The specified key does not exist.</Message>
Run data-import and data-export of authority records with default profiles to reproduce 5xx errors on okapi and mod-entity-link high memory utilization.

Test Results

T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). This table contains durations for Data-export for jobs with “Default instances export profile” on 61 tenants with 3 sets of data(500 records; 1000 records; 2000 records) except tenant 42, on this tenant data-export run with the largest number of instances and “Default instances export profile”. Additional information about failed jobs is in the Additional informationInformation/Error section

Tenant

500

1000

2000

Status

Started Date

Compl Date

Duration

Status

Started Date

Compl Date

Duration

Status

Started Date

Compl Date

Duration

cs00000001_0001

COMPLETED

08:41:18.306

08:41:42.679

00:00:24.373

COMPLETED

16:12:56.192

16:13:15.479

00:00:19.287

COMPLETED

20:43:32.753

20:44:47.318

00:01:14.565

cs00000001_0002

COMPLETED

08:41:25.000

08:41:47.885

00:00:22.885

COMPLETED

16:13:01.591

16:13:34.742

00:00:33.151

COMPLETED

20:43:37.932

20:45:58.223

00:02:20.291

cs00000001_0003

COMPLETED

08:41:31.646

08:41:53.671

00:00:22.025

COMPLETED

16:13:09.258

16:13:52.747

00:00:43.489

COMPLETED

20:43:46.351

20:47:26.980

00:03:40.629

cs00000001_0004

COMPLETED

08:41:36.937

08:41:58.192

00:00:21.255

COMPLETED

16:13:17.224

16:14:11.263

00:00:54.039

COMPLETED

20:43:56.513

20:49:11.379

00:05:14.866

cs00000001_0005

COMPLETED

08:41:42.698

08:42:03.951

00:00:21.253

COMPLETED

16:13:24.351

16:14:36.431

00:01:12.08

COMPLETED

20:44:03.713

20:50:33.894

00:06:30.181

cs00000001_0006

COMPLETED

08:41:47.885

08:42:08.914

00:00:21.029

COMPLETED

16:13:30.177

16:14:50.495

00:01:20.318

COMPLETED

20:44:12.660

20:51:55.779

00:07:43.119

cs00000001_0007

FAIL

08:41:53.167

08:42:15.317

00:00:22.15

COMPLETED

16:13:38.634

16:15:12.011

00:01:33.377

COMPLETED

20:44:21.387

20:53:27.125

00:09:05.738

cs00000001_0008

COMPLETED

08:41:59.692

08:42:25.276

00:00:25.584

COMPLETED

16:13:43.573

16:15:26.623

00:01:43.05

COMPLETED

20:44:29.600

20:54:27.147

00:09:57.547

cs00000001_0009

COMPLETED

08:42:04.965

08:42:33.644

00:00:28.679

COMPLETED

16:13:52.774

16:15:50.168

00:01:57.394

COMPLETED

20:44:37.979

20:55:25.725

00:10:47.746

cs00000001_0010

COMPLETED

08:42:10.940

08:42:42.604

00:00:31.664

COMPLETED

16:13:59.297

16:16:09.067

00:02:09.77

COMPLETED

20:44:46.845

20:56:21.134

00:11:34.289

cs00000001_0011

FAIL

08:42:17.625

08:42:50.114

00:00:32.489

COMPLETED

16:14:05.996

16:16:27.609

00:02:21.613

COMPLETED

20:44:54.729

20:57:13.442

00:12:18.713

cs00000001_0012

COMPLETED

08:42:25.296

08:43:00.791

00:00:35.495

COMPLETED

16:14:13.919

16:16:50.901

00:02:36.982

COMPLETED

20:45:00.316

20:57:40.427

00:12:40.111

cs00000001_0013

COMPLETED

08:42:30.529

08:43:09.030

00:00:38.501

COMPLETED

16:14:19.843

16:17:03.193

00:02:43.35

COMPLETED

20:45:09.757

20:58:33.223

00:13:23.466

cs00000001_0014

COMPLETED

08:42:36.508

08:43:19.801

00:00:43.293

COMPLETED

16:14:26.105

16:17:15.922

00:02:49.817

COMPLETED

20:45:16.057

20:58:58.640

00:13:42.583

cs00000001_0015

FAIL

08:42:43.408

08:43:29.083

00:00:45.675

COMPLETED

16:14:35.856

16:17:41.369

00:03:05.513

COMPLETED

20:45:23.453

20:59:23.966

00:14:00.513

cs00000001_0016

COMPLETED

08:42:50.139

08:43:35.961

00:00:45.822

COMPLETED

16:14:41.975

16:17:57.570

00:03:15.595

COMPLETED

20:45:34.973

20:59:48.715

00:14:13.742

cs00000001_0017

COMPLETED

08:42:56.821

08:43:44.103

00:00:47.282

COMPLETED

16:14:50.522

16:18:22.953

00:03:32.431

COMPLETED

20:45:42.889

21:00:13.600

00:14:30.711

cs00000001_0018

COMPLETED

08:43:01.562

08:43:55.090

00:00:53.528

COMPLETED

16:14:57.741

16:18:43.230

00:03:45.489

COMPLETED

20:45:50.660

21:00:37.526

00:14:46.866

cs00000001_0019

FAIL

08:43:09.901

08:44:00.858

00:00:50.957

FAIL

16:15:03.898

16:18:54.326

00:03:50.428

COMPLETED

20:45:58.742

21:01:00.588

00:15:01.846

cs00000001_0020

COMPLETED

08:43:15.211

08:44:12.492

00:00:57.281

COMPLETED

16:15:11.436

16:19:20.244

00:04:08.808

COMPLETED

20:46:07.696

21:01:24.518

00:15:16.822

cs00000001_0021

COMPLETED

08:43:24.138

08:44:18.231

00:00:54.093

FAIL

16:15:18.252

16:19:36.126

00:04:17.874

COMPLETED

20:46:17.326

21:01:47.076

00:15:29.75

cs00000001_0022

COMPLETED

08:43:29.110

08:44:24.802

00:00:55.692

COMPLETED

16:15:25.017

16:19:44.324

00:04:19.307

COMPLETED

20:46:26.716

21:02:09.121

00:15:42.405

cs00000001_0023

COMPLETED

08:43:34.515

08:44:37.741

00:01:03.226

COMPLETED

16:15:33.244

16:20:21.071

00:04:47.827

COMPLETED

20:46:37.567

21:02:30.237

00:15:52.67

cs00000001_0024

FAIL

08:43:44.103

08:44:44.705

00:01:00.602

COMPLETED

16:15:41.477

16:20:41.820

00:05:00.343

COMPLETED

20:46:41.029

21:02:51.477

00:16:10.448

cs00000001_0025

COMPLETED

08:43:49.254

08:44:52.111

00:01:02.857

FAIL

16:15:48.498

16:20:56.085

00:05:07.587

COMPLETED

20:46:48.851

21:02:52.107

00:16:03.256

cs00000001_0026

FAIL

08:43:55.090

08:44:58.450

00:01:03.36

COMPLETED

16:15:50.199

16:21:02.441

00:05:12.242

COMPLETED

20:47:00.344

21:03:11.997

00:16:11.653

cs00000001_0027

COMPLETED

08:44:00.862

08:45:05.258

00:01:04.396

COMPLETED

16:15:59.021

16:21:21.831

00:05:22.81

COMPLETED

20:47:13.135

21:03:30.979

00:16:17.844

cs00000001_0028

COMPLETED

08:44:06.633

08:45:18.533

00:01:11.9

FAIL

16:16:08.034

16:21:28.734

00:05:20.7

COMPLETED

20:47:13.641

21:03:31.564

00:16:17.923

cs00000001_0029

COMPLETED

08:44:13.255

08:45:25.471

00:01:12.216

COMPLETED

16:16:17.182

16:21:45.894

00:05:28.712

COMPLETED

20:47:26.502

21:03:48.240

00:16:21.738

cs00000001_0030

COMPLETED

08:44:19.680

08:45:32.942

00:01:13.262

COMPLETED

16:16:27.637

16:21:58.230

00:05:30.593

COMPLETED

20:47:39.164

21:04:04.837

00:16:25.673

cs00000001_0031

COMPLETED

08:44:26.226

08:45:40.800

00:01:14.574

COMPLETED

16:16:28.185

16:22:14.855

00:05:46.67

COMPLETED

20:47:40.264

21:04:05.925

00:16:25.661

cs00000001_0032

COMPLETED

08:44:31.947

08:45:47.403

00:01:15.456

COMPLETED

16:16:38.980

16:22:22.245

00:05:43.265

COMPLETED

20:47:54.143

21:04:21.329

00:16:27.186

cs00000001_0033

COMPLETED

08:44:37.766

08:45:54.482

00:01:16.716

COMPLETED

16:16:49.355

16:22:31.415

00:05:42.06

COMPLETED

20:47:54.165

21:04:21.329

00:16:27.164

cs00000001_0034

FAIL

08:44:44.734

08:46:02.199

00:01:17.465

COMPLETED

16:16:50.900

16:22:47.503

00:05:56.603

COMPLETED

20:48:11.145

21:04:37.074

00:16:25.929

cs00000001_0035

COMPLETED

08:44:51.413

08:46:10.557

00:01:19.144

COMPLETED

16:17:02.150

16:22:48.836

00:05:46.686

COMPLETED

20:48:10.524

21:04:35.934

00:16:25.41

cs00000001_0036

FAIL

08:44:58.453

08:46:18.757

00:01:20.304

COMPLETED

16:17:03.192

16:23:02.021

00:05:58.829

COMPLETED

20:48:27.791

21:04:49.677

00:16:21.886

cs00000001_0037

COMPLETED

08:45:04.009

08:46:18.767

00:01:14.758

FAIL

16:17:14.904

16:23:15.294

00:06:00.39

COMPLETED

20:48:28.325

21:04:50.943

00:16:22.618

cs00000001_0038

COMPLETED

08:45:10.759

08:46:26.979

00:01:16.22

COMPLETED

16:17:15.921

16:23:15.297

00:05:59.376

COMPLETED

20:48:47.245

21:05:02.441

00:16:15.196

cs00000001_0039

COMPLETED

08:45:17.825

08:46:34.947

00:01:17.122

COMPLETED

16:17:27.367

16:23:27.219

00:05:59.852

COMPLETED

20:48:47.272

21:05:03.655

00:16:16.383

cs00000001_0040

COMPLETED

08:45:24.020

08:46:43.387

00:01:19.367

COMPLETED

16:17:37.584

16:23:27.815

00:05:50.231

COMPLETED

20:49:07.141

21:05:16.105

00:16:08.964

cs00000001_0041

COMPLETED

08:45:25.495

08:46:51.713

00:01:26.218

COMPLETED

16:17:40.856

16:23:38.234

00:05:57.378

COMPLETED

20:49:07.932

21:05:16.106

00:16:08.174

cs00000001_0042

COMPLETED

08:47:41.141

10:44:05.834

01:56:24.693

COMPLETED

16:24:30.590

18:13:37.660

01:49:07.07

COMPLETED

21:06:03.875

00:24:43.478

02:18:39.603

cs00000001_0043

COMPLETED

08:45:39.452

08:46:59.947

00:01:20.495

COMPLETED

16:17:41.400

16:23:02.020

00:05:20.62

COMPLETED

20:48:29.519

20:56:20.036

00:07:50.517

cs00000001_0044

COMPLETED

08:45:46.640

08:47:06.799

00:01:20.159

COMPLETED

16:18:06.311

16:23:49.211

00:05:42.9

COMPLETED

20:49:28.535

21:05:26.314

00:15:57.779

cs00000001_0045

FAIL

08:45:54.498

08:47:12.755

00:01:18.257

COMPLETED

16:18:09.871

16:23:58.163

00:05:48.292

COMPLETED

20:49:48.172

21:05:34.512

00:15:46.34

cs00000001_0046

COMPLETED

08:46:02.202

08:47:18.442

00:01:16.24

COMPLETED

16:18:10.926

16:23:58.164

00:05:47.238

COMPLETED

20:49:48.196

21:05:36.270

00:15:48.074

cs00000001_0047

COMPLETED

08:46:02.225

08:47:18.442

00:01:16.217

FAIL

16:18:24.123

16:24:01.782

00:05:37.659

FAIL

20:50:09.653

21:05:38.160

00:15:28.507

cs00000001_0048

COMPLETED

08:46:10.574

08:47:22.681

00:01:12.107

COMPLETED

16:18:26.401

16:24:06.366

00:05:39.965

COMPLETED

20:50:09.701

21:05:43.870

00:15:34.169

cs00000001_0049

COMPLETED

08:46:18.761

08:47:29.539

00:01:10.778

COMPLETED

16:18:40.848

16:24:13.867

00:05:33.019

COMPLETED

20:50:09.679

21:05:42.628

00:15:32.949

cs00000001_0050

COMPLETED

08:46:26.232

08:47:32.182

00:01:05.95

COMPLETED

16:18:42.610

16:24:13.868

00:05:31.258

FAIL

20:50:33.358

21:05:49.392

00:15:16.034

cs00000001_0051

COMPLETED

08:46:26.260

08:47:34.374

00:01:08.114

COMPLETED

16:18:58.924

16:24:19.617

00:05:20.693

COMPLETED

20:50:33.400

21:05:50.748

00:15:17.348

cs00000001_0052

COMPLETED

08:46:34.943

08:47:34.375

00:00:59.432

COMPLETED

16:18:58.900

16:24:13.870

00:05:14.97

COMPLETED

20:50:33.380

21:05:50.748

00:15:17.368

cs00000001_0053

COMPLETED

08:46:43.383

08:47:36.503

00:00:53.12

COMPLETED

16:19:00.681

16:24:19.617

00:05:18.936

COMPLETED

20:50:58.398

21:05:56.115

00:14:57.717

cs00000001_0054

COMPLETED

08:46:51.739

08:47:37.351

00:00:45.612

COMPLETED

16:19:17.924

16:24:19.618

00:05:01.694

COMPLETED

20:50:58.420

21:05:56.115

00:14:57.695

cs00000001_0055

COMPLETED

08:46:02.225

08:47:18.442

00:02:11.391

COMPLETED

16:19:17.967

16:24:19.621

00:05:01.654

COMPLETED

20:50:57.826

21:05:51.800

00:14:53.974

cs00000001_0056

COMPLETED

08:46:10.574

08:47:22.681

00:01:57.137

COMPLETED

16:19:36.191

16:24:22.672

00:04:46.481

COMPLETED

20:51:25.458

21:05:58.811

00:14:33.353

cs00000001_0057

COMPLETED

08:46:18.761

08:47:29.539

00:01:57.114

COMPLETED

16:19:37.364

16:24:25.511

00:04:48.147

COMPLETED

20:51:24.836

21:05:56.115

00:14:31.279

cs00000001_0058

COMPLETED

08:46:26.232

08:47:32.182

00:01:59.658

COMPLETED

16:19:38.715

16:24:25.510

00:04:46.795

COMPLETED

20:51:25.482

21:05:59.833

00:14:34.351

cs00000001_0059

COMPLETED

08:46:26.260

08:47:34.374

00:01:59.169

COMPLETED

16:19:57.118

16:24:27.002

00:04:29.884

COMPLETED

20:51:26.048

21:05:59.833

00:14:33.785

cs00000001_0060

COMPLETED

08:46:34.943

08:47:34.375

00:01:48.399

COMPLETED

16:19:57.176

16:24:27.003

00:04:29.827

COMPLETED

20:51:54.664

21:05:59.837

00:14:05.173

cs00000001_0061

COMPLETED

08:46:43.383

08:47:36.503

00:01:44.223

COMPLETED

16:19:58.846

16:24:27.006

00:04:28.16

COMPLETED

20:51:54.644

21:05:59.833

00:14:05.189

...

Tenant	Started Date	Compl Date	Duration	Started Date	Compl Date	Duration	Started Date	Compl Date	Duration
cs00000001	14:30:48.756	15:02:56.495	00:32:07.739	15:25:21.738	16:09:25.170	00:44:03.432	17:29:00.215	18:24:32.154	00:55:31.939
cs00000001_0001	14:24:42.318	14:24:42.318	00:00:45.211	15:25:15.665	15:31:19.954	00:06:04.289	17:28:49.284	17:44:36.536	00:15:47.252
cs00000001_0002	14:24:43.082	14:25:31.223	00:00:48.141	15:25:17.867	15:32:31.980	00:07:14.113	17:28:50.497	17:45:49.205	00:16:58.708
cs00000001_0003	14:24:45.343	14:25:45.888	00:01:00.545	15:25:19.490	15:32:56.805	00:07:37.315	17:28:52.298	17:46:17.206	00:17:24.908
cs00000001_0004	14:24:47.523	14:26:21.848	00:01:34.325	15:25:19.468	15:32:53.154	00:07:33.686	17:28:55.078	17:47:24.722	00:18:29.644
cs00000001_0005	14:24:47.497	14:26:03.472	00:01:15.975	15:25:26.453	15:34:08.759	00:08:42.306	17:28:55.055	17:47:08.115	00:18:13.06
cs00000001_0006	14:24:51.872	14:26:58.001	00:02:06.129	15:25:34.475	15:35:08.741	00:09:34.266	17:29:10.529	17:49:43.272	00:20:32.743
cs00000001_0007	14:24:51.900	14:27:00.778	00:02:08.878	15:25:39.336	15:35:31.356	00:09:52.02	17:29:10.505	17:49:42.671	00:20:32.166
cs00000001_0008	14:24:55.497	14:27:40.752	00:02:45.255	15:25:39.358	15:35:36.215	00:09:56.857	17:29:15.627	17:50:15.416	00:20:59.789
cs00000001_0009	14:24:59.483	14:27:54.691	00:02:55.208	15:25:39.876	15:35:43.352	00:10:03.476	17:29:17.275	17:50:33.670	00:21:16.395
cs00000001_0010	14:24:59.508	14:28:06.621	00:03:07.113	15:25:41.078	15:35:43.957	00:10:02.879	17:29:22.272	17:50:51.494	00:21:29.222
cs00000001_0011	14:25:00.195	14:28:14.477	00:03:14.282	15:25:41.086	15:35:49.502	00:10:08.416	17:29:22.244	17:50:48.353	00:21:26.109
cs00000001_0012	14:25:04.852	14:28:20.675	00:03:15.823	15:25:47.008	15:36:04.014	00:10:17.006	17:29:28.675	17:51:20.287	00:21:51.612
cs00000001_0013	14:25:05.432	14:28:35.978	00:03:30.546	15:25:46.416	15:36:03.250	00:10:16.834	17:29:29.703	17:51:24.819	00:21:55.116
cs00000001_0014	14:25:07.135	14:28:41.482	00:03:34.347	15:25:50.212	15:36:16.623	00:10:26.411	17:29:29.732	17:51:24.821	00:21:55.089
cs00000001_0015	14:25:11.505	14:28:49.677	00:03:38.172	15:25:50.239	15:36:17.917	00:10:27.678	17:29:38.216	17:51:53.405	00:22:15.189
cs00000001_0016	14:25:11.481	14:28:47.521	00:03:36.04	15:25:55.589	15:36:33.118	00:10:37.529	17:29:37.763	17:51:49.102	00:22:11.339
cs00000001_0017	14:25:13.285	14:29:03.071	00:03:49.786	15:25:56.102	15:36:34.963	00:10:38.861	17:29:37.763	17:51:49.103	00:22:11.34
cs00000001_0018	14:25:18.535	14:29:07.800	00:03:49.265	15:25:58.614	15:36:44.878	00:10:46.264	17:29:47.800	17:52:17.763	00:22:29.963
cs00000001_0019	14:25:20.256	14:29:13.686	00:03:53.43	15:26:06.052	15:36:52.652	00:10:46.6	17:29:48.419	17:52:20.725	00:22:32.306
cs00000001_0020	14:25:20.281	14:29:21.852	00:04:01.571	15:26:06.028	15:36:51.427	00:10:45.399	17:29:48.396	17:52:17.762	00:22:29.366
cs00000001_0021	14:25:28.404	14:29:28.699	00:04:00.295	15:26:07.660	15:36:57.123	00:10:49.463	17:29:48.995	17:52:23.675	00:22:34.68
cs00000001_0022	14:25:30.184	14:29:30.962	00:04:00.778	15:26:09.197	15:37:08.209	00:10:59.012	17:29:49.018	17:52:23.676	00:22:34.658
cs00000001_0023	14:25:30.184	14:29:31.495	00:04:01.311	15:26:11.778	15:37:10.213	00:10:58.435	17:30:01.919	17:52:42.840	00:22:40.921
cs00000001_0024	14:25:30.184	14:29:30.435	00:04:00.251	15:26:19.761	15:37:13.777	00:10:54.016	17:30:01.941	17:52:43.450	00:22:41.509
cs00000001_0025	14:25:30.184	14:29:31.495	00:04:01.311	15:26:19.783	15:37:16.180	00:10:56.397	17:30:03.091	17:52:49.336	00:22:46.245
cs00000001_0026	14:25:31.763	14:29:44.320	00:04:12.557	15:26:20.346	15:37:16.180	00:10:55.834	17:30:03.112	17:52:50.019	00:22:46.907
cs00000001_0027	14:25:36.718	14:29:47.116	00:04:10.398	15:26:23.529	15:37:29.781	00:11:06.252	17:30:17.486	17:53:04.582	00:22:47.096
cs00000001_0028	14:25:43.614	14:29:47.648	00:04:04.034	15:26:25.997	15:37:31.677	00:11:05.68	17:30:17.536	17:53:04.583	00:22:47.047
cs00000001_0029	14:25:43.635	14:29:48.171	00:04:04.536	15:26:36.318	15:37:33.537	00:10:57.219	17:30:17.587	17:53:05.213	00:22:47.626
cs00000001_0030	14:25:43.589	14:29:47.648	00:04:04.059	15:26:37.039	15:37:38.474	00:11:01.435	17:30:19.321	17:53:11.505	00:22:52.184
cs00000001_0031	14:25:43.567	14:29:47.645	00:04:04.078	15:26:36.341	15:37:35.539	00:10:59.198	17:30:28.307	17:53:16.349	00:22:48.042
cs00000001_0032	14:25:44.235	14:29:49.921	00:04:05.686	15:26:36.992	15:37:36.079	00:10:59.087	17:30:33.304	17:53:16.349	00:22:43.045
cs00000001_0033	14:25:46.439	14:30:01.215	00:04:14.776	15:26:37.017	15:37:36.079	00:10:59.062	17:30:33.326	17:53:24.641	00:22:51.315
cs00000001_0034	14:25:53.953	14:30:02.375	00:04:08.422	15:26:40.769	15:37:48.169	00:11:07.4	17:30:33.350	17:53:25.842	00:22:52.492
cs00000001_0035	14:26:00.662	14:30:02.888	00:04:02.226	15:26:40.793	15:37:48.910	00:11:08.117	17:30:35.980	17:53:30.121	00:22:54.141
cs00000001_0036	14:26:00.600	14:30:02.888	00:04:02.288	15:26:48.214	15:37:48.911	00:11:00.697	17:30:36.034	17:53:30.122	00:22:54.088
cs00000001_0037	14:26:00.627	14:30:02.887	00:04:02.26	15:26:53.216	15:37:49.487	00:10:56.271	17:30:40.561	17:53:31.268	00:22:50.707
cs00000001_0038	14:26:00.707	14:30:04.701	00:04:03.994	15:26:56.783	15:37:50.130	00:10:53.347	17:30:50.085	17:53:32.414	00:22:42.329
cs00000001_0039	14:26:00.685	14:30:03.360	00:04:02.675	15:26:57.421	15:37:52.235	00:10:54.814	17:30:50.029	17:53:32.414	00:22:42.385
cs00000001_0040	14:26:02.068	14:30:10.702	00:04:08.634	15:26:57.458	15:37:59.809	00:11:02.351	17:30:50.106	17:53:33.585	00:22:43.479
cs00000001_0041	14:26:03.496	14:30:13.924	00:04:10.428	15:26:09.179	15:26:24.068	00:00:14.889	17:30:50.051	17:53:32.414	00:22:42.363
cs00000001_0042	14:25:44.235	14:29:49.921	00:04:05.686	15:26:46.587	15:36:44.876	00:09:58.289	17:29:39.707	17:29:51.801	00:00:12.094
cs00000001_0043	14:26:11.198	14:30:14.616	00:04:03.418	15:27:01.669	15:38:02.854	00:11:01.185	17:30:10.228	17:39:28.947	00:09:18.719
cs00000001_0044	14:26:15.107	14:30:14.619	00:03:59.512	15:27:02.233	15:38:02.854	00:11:00.621	17:31:04.416	17:53:43.867	00:22:39.451
cs00000001_0045	14:26:18.467	14:30:14.618	00:03:56.151	15:27:03.317	15:38:03.511	00:11:00.194	17:31:15.533	17:53:45.158	00:22:29.625
cs00000001_0046	14:26:20.713	14:30:14.619	00:03:53.906	15:27:13.457	15:38:03.511	00:10:50.054	17:31:15.507	17:53:45.157	00:22:29.65
cs00000001_0047	14:26:21.798	14:30:14.620	00:03:52.822	15:27:15.707	15:38:03.511	00:10:47.804	17:31:15.481	17:53:45.157	00:22:29.676
cs00000001_0048	14:26:21.241	14:30:14.620	00:03:53.379	15:27:18.004	15:38:03.512	00:10:45.508	17:31:15.559	17:53:49.697	00:22:34.138
cs00000001_0049	14:26:21.821	14:30:15.961	00:03:54.14	15:27:20.942	15:38:03.522	00:10:42.58	17:31:20.125	17:53:53.650	00:22:33.525
cs00000001_0050	14:26:21.846	14:30:22.124	00:04:00.278	14:26:23.532	14:30:22.113	00:03:58.581	17:31:20.147	17:53:54.706	00:22:34.559
cs00000001_0051	14:26:03.496	14:26:01.226	00:04:10.853	14:26:24.155	14:30:22.114	00:03:57.959	17:31:24.154	17:53:55.404	00:22:31.25
cs00000001_0052	14:25:44.209	14:30:14.616	00:04:09.499	14:26:28.901	14:30:22.114	00:03:53.213	17:31:24.730	17:53:55.405	00:22:30.675
cs00000001_0053	14:26:11.198	14:30:14.619	00:04:09.56	14:26:38.502	14:30:22.120	00:03:43.618	17:31:34.686	17:53:55.405	00:22:20.719
cs00000001_0054	14:26:15.107	14:30:14.618	00:04:08.079	14:26:37.825	14:30:22.114	00:03:44.289	17:31:45.822	17:53:55.405	00:22:09.583
cs00000001_0055	14:26:18.467	14:30:14.619	00:03:34.212	14:26:40.893	14:30:22.117	00:03:41.224	17:31:46.415	17:54:00.244	00:22:13.829
cs00000001_0056	14:26:20.713	14:30:14.620	00:03:57.999	14:26:40.849	14:30:22.117	00:03:41.268	17:31:45.788	17:53:55.405	00:22:09.617
cs00000001_0057	14:26:21.798	14:30:14.620	00:03:58.051	14:26:41.705	14:30:22.117	00:03:40.412	17:31:45.847	17:53:56.620	00:22:10.773
cs00000001_0058	14:26:21.241	14:30:15.961	00:03:33.063	14:26:44.945	14:30:22.117	00:03:37.172	17:31:45.868	17:53:56.619	00:22:10.751
cs00000001_0059	14:26:21.821	14:30:22.124	00:03:58.146	14:26:46.887	14:30:22.123	00:03:35.236	17:31:55.343	17:54:00.245	00:22:04.902
cs00000001_0060	14:26:21.846	14:30:10.702	00:03:57.914	14:26:44.988	14:30:22.121	00:03:37.133	17:31:50.896	17:54:00.245	00:22:09.349
cs00000001_0061	14:26:00.707	14:30:13.924	00:03:58.086	14:26:37.825	14:30:22.114	00:03:44.289	17:31:53.075	17:54:00.245	00:22:07.17

Resource Utilization

Service CPU Utilization

T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). Service CPU usage during the test was quite low, only the mod-data-export-b okapi-b mod-permissions-b services had a slight increase up to 18%. The test lasted after 23:15 bur there were some spikes from another test. but the CPU usage of selected services was the same.

...

T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

...

Memory Utilization

T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).

...

T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

...

DB CPU Utilization

T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). Except for the database spikes that every 30 minutes. CPU usage before tets was about 2.5%, during all Data Export of 500, 1000, and 2000 instances, CPU usage increased up to 4%;

...

T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

...

DB Connections

T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). The maximal number of DB connections was 1342 for 2000 DE.

...

T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

...

DB Load

T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).

...

T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

...

SQL queries

T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).

...

T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

...

Additional information

Errors

Data-exports jobs failed with the same errors.

...

Expand

title	Data export failed

This XML file does not appear to have any style information associated with it. The document tree is shown below.

<Error>

<Code>NoSuchKey</Code>

<Message>The specified key does not exist.</Message>

<HostId>tzFaRcMEHCtq7ZuHDxueqpq2uE+w6W7S2jGSoRSRNskejfvcI4T3azSiy786C26Z/Io/oOjSgzA=</HostId>

</Error>

Methodology/Approach

All tests were run using the JMeter script. Data has been pre-generated for all tenants

Infrastructure

PTF -environment mcpt

11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIBv CPUsmax_connections
db.r6g.4xlarge 128 GiB; 16 vCPU
MSK tenant
- 2 m5.2xlarge brokers in 1 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB

The table contains modules, memory, and CPU parameters

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mcpt-pvt	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mod-consortia	11	mod-consortia:1.0.3	2	2048	1802	128	768	88	128	FALSE
mod-organizations	6	mod-organizations:1.8.0	2	1024	896	128	700	88	128	FALSE
mod-data-import	7	mod-data-import:3.0.7	1	2048	1844	256	1292	384	512	FALSE
mod-search	12	mod-search:3.0.5	2	2592	2480	2048	1440	512	1024	FALSE
mod-configuration	6	mod-configuration:5.9.2	2	1024	896	128	768	88	128	FALSE
mod-permissions	14	mod-permissions:6.4.0	2	1684	1544	512	1024	384	512	FALSE
mod-entities-links	10	mod-entities-links:2.0.4	2	2592	2480	400	1440	0	1024	FALSE
mod-inventory-storage	10	mod-inventory-storage:27.0.4	2	4096	3690	2048	3076	384	512	FALSE
mod-data-export-worker	9	mod-data-export-worker:3.1.2	2	3072	2800	1024	2048	384	512	FALSE
mod-source-record-storage	10	mod-source-record-storage:5.7.5	2	5600	5000	2048	3500	384	512	FALSE
mod-inventory	10	mod-inventory:20.1.6	2	2880	2592	1024	1814	384	512	FALSE
mod-di-converter-storage	9	mod-di-converter-storage:2.1.5	2	1024	896	128	768	88	128	FALSE
mod-users	11	mod-users:19.2.2	2	1024	896	128	768	88	128	FALSE
mod-source-record-manager	10	mod-source-record-manager:3.7.7	2	5600	5000	2048	3500	384	512	FALSE
mod-quick-marc	10	mod-quick-marc:5.0.1	1	2288	2176	128	1664	384	512	FALSE
nginx-okapi	2	nginx-okapi:2023.06.14	2	1024	896	512	0	0	0	FALSE
okapi-b	1	okapi:5.1.2	3	1684	1440	1024	922	384	512	FALSE
mod-data-export	10	mod-data-export:4.8.7	1	1024	896	1024	768	88	128	FALSE

Versions Compared

Old Version 1

New Version Current

Key

Overview

Summary

Recommendations & Jiras

Test Results

Resource Utilization

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

DB Load

SQL queries

Additional information

Errors

Methodology/Approach

Infrastructure

Page Comparison

Versions Compared

Old Version 1

New Version Current

Key

Overview

Summary

Recommendations & Jiras

Test Results

Resource Utilization

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

DB Load

SQL queries

Additional information

Errors

Methodology/Approach

Infrastructure