Skip to end of banner
Go to start of banner

Data Export Test Report (Mobius)

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Overview

This document contains the results of testing Data Export tests on 61 tenants (excluding the central tenant):

  • DE_instances T1(500-1K-2k). (MARC BIB) on Mobius environment with Data Export tests for 500, 1k, 2k. Three CSV files were prepared to run Data Export with Default instances export profile.

  • DE_instances T2(Max). (MARC BIB) on Mobius environment with Data Export tests for maximal number of instances on each member tenant(The exact number of instances is in the section “Errors / Additional information”). Three CSV files were prepared to run Data Export with Default instances export profile.

Ticket: PERF-777 - Getting issue details... STATUS

Summary

Recommendations & Jiras

Test Results

DE_instances T1(500-1K-2k). This table contains durations for Data-export for jobs with “Default instances export profile” on 61 tenants with 3 sets of data(500 records; 1000 records; 2000 records). 

Transaction name

DE 500, sec

DE 1K, sec

DE 2K, sec

DE_Exporting MARC Bib records workflow Transaction cs00000001_0001

12

32

127

DE_Exporting MARC Bib records workflow Transaction cs00000001_0002

16

51

207

DE_Exporting MARC Bib records workflow Transaction cs00000001_0003

23

73

330

DE_Exporting MARC Bib records workflow Transaction cs00000001_0004

22

100

494

DE_Exporting MARC Bib records workflow Transaction cs00000001_0005

27

122

606

DE_Exporting MARC Bib records workflow Transaction cs00000001_0006

29

131

704

DE_Exporting MARC Bib records workflow Transaction cs00000001_0007

28

154

782

DE_Exporting MARC Bib records workflow Transaction cs00000001_0008

34

170

875

DE_Exporting MARC Bib records workflow Transaction cs00000001_0009

39

191

936

DE_Exporting MARC Bib records workflow Transaction cs00000001_0010

38

216

965

DE_Exporting MARC Bib records workflow Transaction cs00000001_0011

40

228

1022

DE_Exporting MARC Bib records workflow Transaction cs00000001_0012

45

261

1054

DE_Exporting MARC Bib records workflow Transaction cs00000001_0013

45

277

1082

DE_Exporting MARC Bib records workflow Transaction cs00000001_0014

45

293

1108

DE_Exporting MARC Bib records workflow Transaction cs00000001_0015

52

308

1163

DE_Exporting MARC Bib records workflow Transaction cs00000001_0016

51

331

1161

DE_Exporting MARC Bib records workflow Transaction cs00000001_0017

56

372

1183

DE_Exporting MARC Bib records workflow Transaction cs00000001_0018

56

374

1209

DE_Exporting MARC Bib records workflow Transaction cs00000001_0019

63

399

1232

DE_Exporting MARC Bib records workflow Transaction cs00000001_0020

63

419

1254

DE_Exporting MARC Bib records workflow Transaction cs00000001_0021

68

432

1275

DE_Exporting MARC Bib records workflow Transaction cs00000001_0022

69

457

1271

DE_Exporting MARC Bib records workflow Transaction cs00000001_0023

72

479

1290

DE_Exporting MARC Bib records workflow Transaction cs00000001_0024

74

475

1306

DE_Exporting MARC Bib records workflow Transaction cs00000001_0025

74

489

1300

DE_Exporting MARC Bib records workflow Transaction cs00000001_0026

75

506

1319

DE_Exporting MARC Bib records workflow Transaction cs00000001_0027

85

503

1336

DE_Exporting MARC Bib records workflow Transaction cs00000001_0028

80

502

1331

DE_Exporting MARC Bib records workflow Transaction cs00000001_0029

91

535

1347

DE_Exporting MARC Bib records workflow Transaction cs00000001_0030

87

532

1344

DE_Exporting MARC Bib records workflow Transaction cs00000001_0031

87

530

1357

DE_Exporting MARC Bib records workflow Transaction cs00000001_0032

93

539

1349

DE_Exporting MARC Bib records workflow Transaction cs00000001_0033

97

535

1363

DE_Exporting MARC Bib records workflow Transaction cs00000001_0034

97

547

1357

DE_Exporting MARC Bib records workflow Transaction cs00000001_0035

97

560

1373

DE_Exporting MARC Bib records workflow Transaction cs00000001_0036

107

554

1349

DE_Exporting MARC Bib records workflow Transaction cs00000001_0037

102

557

1359

DE_Exporting MARC Bib records workflow Transaction cs00000001_0038

103

563

1370

DE_Exporting MARC Bib records workflow Transaction cs00000001_0039

109

571

1363

DE_Exporting MARC Bib records workflow Transaction cs00000001_0040

104

548

1364

DE_Exporting MARC Bib records workflow Transaction cs00000001_0041

109

552

1365

DE_Exporting MARC Bib records workflow Transaction cs00000001_0042

115

553

1362

DE_Exporting MARC Bib records workflow Transaction cs00000001_0043

119

497

1363

DE_Exporting MARC Bib records workflow Transaction cs00000001_0044

126

555

1363

DE_Exporting MARC Bib records workflow Transaction cs00000001_0045

121

550

1356

DE_Exporting MARC Bib records workflow Transaction cs00000001_0046

126

553

1352

DE_Exporting MARC Bib records workflow Transaction cs00000001_0047

120

544

1352

DE_Exporting MARC Bib records workflow Transaction cs00000001_0048

131

551

1347

DE_Exporting MARC Bib records workflow Transaction cs00000001_0049

126

546

1340

DE_Exporting MARC Bib records workflow Transaction cs00000001_0050

130

538

1336

DE_Exporting MARC Bib records workflow Transaction cs00000001_0051

127

533

1341

DE_Exporting MARC Bib records workflow Transaction cs00000001_0052

128

533

1337

DE_Exporting MARC Bib records workflow Transaction cs00000001_0053

127

529

1334

DE_Exporting MARC Bib records workflow Transaction cs00000001_0054

123

527

1320

DE_Exporting MARC Bib records workflow Transaction cs00000001_0055

122

516

1320

DE_Exporting MARC Bib records workflow Transaction cs00000001_0056

119

516

1316

DE_Exporting MARC Bib records workflow Transaction cs00000001_0057

114

508

1315

DE_Exporting MARC Bib records workflow Transaction cs00000001_0058

110

503

1306

DE_Exporting MARC Bib records workflow Transaction cs00000001_0059

108

501

1302

DE_Exporting MARC Bib records workflow Transaction cs00000001_0060

101

493

1298

DE_Exporting MARC Bib records workflow Transaction cs00000001_0061

98

487

1289

DE_Exporting MARC Bib records workflow Transaction cs00000001_0062

16

31

36

image-20240206-125717.png

DE_instances T2(Max). The table below contains durations for Data-export for jobs with “Default instances export profile” on 61 tenants with 1 set of data(all of the instances on each tenant). 

Tenant

Number of exported instances

Duration

Shift of the start time

 cs00000001_0001

65359

08:34:11.924

0:00:00

 cs00000001_0002

22370

02:01:17.113

00:00:04.576

 cs00000001_0003

136291

15:38:26.969

00:20:55.783

 cs00000001_0004

77785

10:42:11.942

00:02:52.106

 cs00000001_0005

110093

13:57:20.399

00:12:33.693

 cs00000001_0006

293580

17:24:21.58

02:58:03.642

 cs00000001_0007

279493

17:15:17.802

02:55:41.421

 cs00000001_0008

160024

16:21:18.866

00:56:26.929

 cs00000001_0009

110918

14:01:42.358

00:25:51.269

 cs00000001_0010

83254

11:36:56.235

00:14:09.228

 cs00000001_0011

94171

12:43:36.487

00:20:24.292

 cs00000001_0012

227489

16:52:02.623

02:24:44.754

 cs00000001_0013

19131

01:58:23.547

00:01:37.648

 cs00000001_0014

20746

02:16:27.599

00:01:52.916

 cs00000001_0015

26899

03:24:08.545

00:03:01.837

 cs00000001_0016

153333

15:46:35.492

01:22:21.469

 cs00000001_0017

16492

01:41:43.743

00:02:40.122

 cs00000001_0018

22263

02:43:54.315

00:04:28.863

 cs00000001_0019

5312

00:14:46.338

00:02:05.943

 cs00000001_0020

44259

06:30:26.536

00:12:04.781

 cs00000001_0021

83988

11:46:23.477

00:33:39.156

 cs00000001_0022

86900

12:03:06.865

00:36:22.655

 cs00000001_0023

91517

12:27:03.606

00:40:56.469

 cs00000001_0024

385123

15:35:51.105

05:43:10.805

 cs00000001_0025

84215

11:47:31.81

00:36:33.173

 cs00000001_0026

153912

15:32:54.171

01:42:36.675

 cs00000001_0027

48176

07:11:08.729

00:16:44.682

 cs00000001_0028

64556

09:30:37.567

00:25:12.347

 cs00000001_0029

5140

00:16:41.394

00:03:41.85

 cs00000001_0030

97589

12:52:29.07

00:50:52.289

 cs00000001_0031

71895

10:26:16.656

00:31:10.105

 cs00000001_0032

80074

11:22:00.789

00:37:31.626

 cs00000001_0033

45886

06:52:06.447

00:17:36.951

 cs00000001_0034

18240

02:14:18.252

00:07:47.49

 cs00000001_0035

135704

14:55:28.661

01:29:32.543

 cs00000001_0036

114170

13:56:14.427

01:08:43.67

 cs00000001_0037

24226

03:18:25.02

00:09:57.63

 cs00000001_0038

225563

16:13:32.965

03:06:16.602

 cs00000001_0039

128523

14:37:45.055

01:24:25.093

 cs00000001_0040

111992

13:47:45.898

01:08:36.465

 cs00000001_0041

61091

09:03:29.08

00:27:37.435

 cs00000001_0042

634675

12:24:24.995

09:48:38.838

 cs00000001_0043

2831

00:02:12.583

00:04:22.427

 cs00000001_0044

70871

10:19:26.438

00:35:14.19

 cs00000001_0045

78320

11:10:12.277

00:41:17.857

 cs00000001_0046

109677

13:38:14.993

01:08:58.069

 cs00000001_0047

48531

07:19:11.872

00:22:04.318

 cs00000001_0048

53346

08:01:00.977

00:24:42.919

 cs00000001_0049

6952

00:31:27.638

00:06:36.51

 cs00000001_0050

190736

15:54:19.153

02:34:40.768

 cs00000001_0051

351325

15:41:17.61

05:23:26.089

 cs00000001_0052

137940

14:54:13.405

01:39:56.924

 cs00000001_0053

61834

09:11:33.182

00:31:14.104

 cs00000001_0054

16297

02:00:46.997

00:09:40.899

 cs00000001_0055

176009

15:42:52.69

02:21:13.276

 cs00000001_0056

160976

15:28:32.362

02:06:05.583

 cs00000001_0057

22596

03:06:34.087

00:12:13.641

 cs00000001_0058

123819

14:19:45.503

01:27:29.477

 cs00000001_0059

37715

05:38:43.984

00:18:51.151

 cs00000001_0060

38395

05:45:32.811

00:19:17.503

 cs00000001_0061

87117

12:00:00.92

00:53:34.119

 cs00000001_0062

1

Service CPU Utilization

DE_instances T1(500-1K-2k). Service CPU usage during the test was quite low, only the mod-data-export-b okapi-b mod-permissions-b services had a slight increase up to 13%.

image-20240206-132843.png

2024-02-05 19:52 Local time Top 10 CPU-consuming services
mod-consortia-b 19.4813457932
mod-data-export-b 13.0495252091
mod-permissions-b 11.8486188099
mod-pubsub-b 11.3424816303
okapi-b 11.0208864888
mod-inventory-b 8.95271167462
mod-users-b 8.38095871043
mod-configuration-b 8.00652350057
mod-quick-marc-b 6.94299935374

DE_instances T2(Max).


Memory Utilization

DE_instances T1(500-1K-2k). During testing, the memory usage does not increase

image-20240206-134017.png

DE_instances T2(Max).

DB CPU Utilization

DE_instances T1(500-1K-2k). Except for the database spikes that started at 18:15 and repeated every 30 minutes. CPU usage was about 2.5%, during all Data Export of 500, 1000, and 2000 instances, CPU usage increased up to 4%;

image-20240206-134338.png

DB Connections

DE_instances T1(500-1K-2k). Maximal number of DB connections was 1190 for 500 DE; 1270 for 1000DE and 1340 for 2000 DE.

image-20240206-134832.png

DB Load

DE_instances T1(500-1K-2k).

image-20240206-135444.png

SQL queries

DE_instances T1(500-1K-2k).

image-20240206-135513.png

Errors / Additional information

Number of instances on each of the 62 tenant

 Number of instances on each of the 62 tenant

 cs00000001_0001

65359

 cs00000001_0002

22370

 cs00000001_0003

136291

 cs00000001_0004

77785

 cs00000001_0005

110093

 cs00000001_0006

293580

 cs00000001_0007

279493

 cs00000001_0008

160024

 cs00000001_0009

110918

 cs00000001_0010

83254

 cs00000001_0011

94171

 cs00000001_0012

227489

 cs00000001_0013

19131

 cs00000001_0014

20746

 cs00000001_0015

26899

 cs00000001_0016

153333

 cs00000001_0017

16492

 cs00000001_0018

22263

 cs00000001_0019

5312

 cs00000001_0020

44259

 cs00000001_0021

83988

 cs00000001_0022

86900

 cs00000001_0023

86900

 cs00000001_0024

385123

 cs00000001_0025

84215

 cs00000001_0026

153912

 cs00000001_0027

48176

 cs00000001_0028

64556

 cs00000001_0029

5140

 cs00000001_0030

97589

 cs00000001_0031

71895

 cs00000001_0032

80074

 cs00000001_0033

45886

 cs00000001_0034

18240

 cs00000001_0035

135704

 cs00000001_0036

114170

 cs00000001_0037

24226

 cs00000001_0038

225563

 cs00000001_0039

128523

 cs00000001_0040

111992

 cs00000001_0041

61091

 cs00000001_0042

634675

 cs00000001_0043

2831

 cs00000001_0044

70871

 cs00000001_0045

78320

 cs00000001_0046

109677

 cs00000001_0047

48531

 cs00000001_0048

53346

 cs00000001_0049

6952

 cs00000001_0050

190736

 cs00000001_0051

351325

 cs00000001_0052

137940

 cs00000001_0053

61834

 cs00000001_0054

16297

 cs00000001_0055

176009

 cs00000001_0056

160976

 cs00000001_0057

22596

 cs00000001_0058

123819

 cs00000001_0059

37715

 cs00000001_0060

38395

 cs00000001_0061

87117

 cs00000001_0062

1

Methodology/Approach

Infrastructure

PTF -environment mcpt

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 2 database  instances, writer/reader

    NameMemory GIBvCPUsmax_connections

    db.r6g.xlarge

    32 GiB4 vCPUs2731

  • MSK tenant

    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

Table contains modules, memory and CPU parameters

  • No labels