/
Data Export Test Report (Mobius) part 2

Data Export Test Report (Mobius) part 2

Overview

This document contains the results of Data Export tests on 61 tenants (excluding the central tenant) on the MCPT environment, this report is the second part of the DE testing, the previous report is available Data Export Test Report (Mobius) | Errors / Additional information :

  • T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).

The member tenant with the largest data set runs export as described in Scenario 2 using the default export profile(DEP). The remaining member tenants run exports as described in Scenario 1, also using the default job profile. All exports are run concurrently (or almost concurrently) on all tenants.
Scenario 1:

  1. Member tenants: 500-2K instance records;

  2. Profiles: Default instances export profile

  • T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP).

The member tenant with the largest data set runs export as described in Scenario 2 using the default export profile. The remaining member tenants run exports as described in Scenario 1, also using the custom job profile(CJP). All exports are run concurrently on all tenants.

Scenario 1:

  1. Member tenants: 500-2K instance records;

  2. Profiles: Default instances export profile

  • T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)

The central tenant runs export as described in Scenario 5, all member tenants run exports as described in Scenario 1 using the default export profile

Scenario 1:

  1. Member tenants: 500-2K instance records;

  2. Profiles: Default instances export profile

Scenario 5:

  1. Central tenant: all available authority records;

  2. Profiles: Default authority export profile

  • T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)

One member tenant - cs00000001_0003 runs concurrently exports as described in scenarios 1, 3, 4, 5 (all default export profiles)

  • T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

One member tenant - cs00000001_0003 runs concurrently exports as described in scenario 1(custom export profile) and scenario 2 (default export profile)

Ticket:PERF-803: MOBIUS baseline tests: Data Export - part 2Closed

Summary

  • The tested Data-export jobs are quite light for the system and cause an increase in CPU on the services: mod-data-export, okapi and mod-inventory-storage up to 17%;

  • No memory leaks, memory consumption was stable during all of the tests;

  • Database CPU usage varied up to a maximum of 8%;

  • The number of failed DE jobs during concurrent testing was about 7% all with the same errors, but the duration of failed jobs was like the duration of completed, error messages are at the end of the report

  • The behavior of the export date was the same as in the first part of the report, with the latest processes starting with a delay. On the first tenants, where the file with instance ids was downloaded the data-export process is completed the fastest, but the data-export on the last tenants lasts the longest, as can be seen from the graphs below the tables

  • General metrics can be found in the tables below

Data-export duration statistics:

 

Data export

T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).

T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP).

T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)

500,

sec

1000 ,

sec

2000 ,

sec

500 ,

sec

1000 ,

sec

2000,

sec

500 ,

sec

1000 ,

sec

2000,

sec

Average duration

80

256

830

956

2110

4441

215

533

1281

Minimal duration

21

19

74

384

982

1863

45

14

12

Maximal duration

1028

360

987

1075

2257

4624

254

558

1374

75t percentile

77

339

968

1029

2223

4603

244

657

1362

90t percentile

108

356

982

1037

2241

4618

249

661

1368

CS42(MAX RECORDS)

1 hrs 56 min

1 hrs 49 min

2hrs 18 min

2hrs 10 min

2hrs 30 min

1 hrs 49 min

32 min 7 sec

44 min 3 sec

55 min 31 sec

  • T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)

 

500 records

1000 records

2000 records

MARC

10 seconds

35 seconds

1 minutes 45 seconds

Holdings

8,3 seconds

26 seconds

1 minutes 43 seconds

Holdings MAX

43 minutes 26 seconds

41 minutes 16 seconds

40 minutes 55 seconds

authorityCS00

36 minutes 2 seconds

35 minutes 37 seconds

36 minutes 39 seconds

  • T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)

Tenant cs00000001_0003

Starterted time

Finished time

Duration

Test 3. Data-export . Max records. Default export profile

"2024-02-23T11:02:11.934+00:00"

"2024-02-23T11:29:32.025+00:00"

00:27:20.091

Test 3. Data-export . 2000 records. Custom job profile

"2024-02-23T11:02:00.802+00:00"

"2024-02-23T11:03:30.687+00:00"

00:01:29.885

Test 2. Data-export . Max records. Default export profile

"2024-02-23T10:26:59.739+00:00"

"2024-02-23T10:53:33.241+00:00"

00:26:33.502

Test 2. Data-export . 1000 records. Custom job profile

"2024-02-23T10:26:43.425+00:00"

"2024-02-23T10:27:25.347+00:00"

00:00:41.922

Test 1. Data-export . Max records. Default export profile

"2024-02-23T09:56:41.692+00:00"

"2024-02-23T10:23:11.426+00:00"

00:26:29.734

Test 1. Data-export . 500 records. Custom job profile

"2024-02-23T09:56:20.276+00:00"

"2024-02-23T09:56:39.402+00:00"

00:00:19.126

Recommendations & Jiras

  • Investigate Data-exports job that have “FAILED“ status and <Code>NoSuchKey</Code> <Message>The specified key does not exist.</Message>

  • Run data-import and data-export of authority records with default profiles to reproduce 5xx errors on okapi and mod-entity-link high memory utilization.


Test Results

  • T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). This table contains durations for Data-export for jobs with “Default instances export profile” on 61 tenants with 3 sets of data(500 records; 1000 records; 2000 records) except tenant 42, on this tenant data-export run with the largest number of instances and “Default instances export profile”. Additional information about failed jobs is in the Additional Information/Error section

Tenant

500

1000

2000

Status

Started Date

Compl Date

Duration

Status

Started Date

Compl Date

Duration

Status

Started Date

Compl Date

Duration

 cs00000001_0001

COMPLETED

08:41:18.306

08:41:42.679

00:00:24.373

COMPLETED

16:12:56.192

16:13:15.479

00:00:19.287

COMPLETED

20:43:32.753

20:44:47.318

00:01:14.565

 cs00000001_0002

COMPLETED

08:41:25.000

08:41:47.885

00:00:22.885

COMPLETED

16:13:01.591

16:13:34.742

00:00:33.151

COMPLETED

20:43:37.932

20:45:58.223

00:02:20.291

 cs00000001_0003

COMPLETED

08:41:31.646

08:41:53.671

00:00:22.025

COMPLETED

16:13:09.258

16:13:52.747

00:00:43.489

COMPLETED

20:43:46.351

20:47:26.980

00:03:40.629

 cs00000001_0004

COMPLETED

08:41:36.937

08:41:58.192

00:00:21.255

COMPLETED

16:13:17.224

16:14:11.263

00:00:54.039

COMPLETED

20:43:56.513

20:49:11.379

00:05:14.866

 cs00000001_0005

COMPLETED

08:41:42.698

08:42:03.951

00:00:21.253

COMPLETED

16:13:24.351

16:14:36.431

00:01:12.08

COMPLETED

20:44:03.713

20:50:33.894

00:06:30.181

 cs00000001_0006

COMPLETED

08:41:47.885

08:42:08.914

00:00:21.029

COMPLETED

16:13:30.177

16:14:50.495

00:01:20.318

COMPLETED

20:44:12.660

20:51:55.779

00:07:43.119

 cs00000001_0007

FAIL

08:41:53.167

08:42:15.317

00:00:22.15

COMPLETED

16:13:38.634

16:15:12.011

00:01:33.377

COMPLETED

20:44:21.387

20:53:27.125

00:09:05.738

 cs00000001_0008

COMPLETED

08:41:59.692

08:42:25.276

00:00:25.584

COMPLETED

16:13:43.573

16:15:26.623

00:01:43.05

COMPLETED

20:44:29.600

20:54:27.147

00:09:57.547

 cs00000001_0009

COMPLETED

08:42:04.965

08:42:33.644

00:00:28.679

COMPLETED

16:13:52.774

16:15:50.168

00:01:57.394

COMPLETED

20:44:37.979

20:55:25.725

00:10:47.746

 cs00000001_0010

COMPLETED

08:42:10.940

08:42:42.604

00:00:31.664

COMPLETED