/
Master Script normal load test - NLA report

Master Script normal load test - NLA report

Jun 24, 2023

 

Overview

 

In the scope of PERF-524 it's needed to run tests to answer questions: 

  • Can the current System accommodate an average load? Load model described here NLA load model investigation and creation.

  • What happened at peak times when all workflows are running at once? 

  • Typical KPIs:

    • Service CPU

    • Service Memory

    • DB CPU

    • DB Memory

  • Response times

  • Duration of long workflows

  • Recommendations to improve on scaling up/out modules to accommodate peak times

Summary

  • The current system can accommodate an average load only without Data Import. With data import, we will have 'HTTP 500 Internal Server Error. If the issue persists, please report it to EBSCO Connect.' for several workflows in the source-storage/records/{id}/formatted requests PERF-582: Response 500 for NLA normal load test with Data Import job (2 SP)Blocked, and the general response time will be longer up to 2 times for all other workflows.

  • According to the ticketPERF-525: NLA: Improve performance of Fixed load testsClosed, were performed a set of tests with changing parameters of DI modules. Results in the table show that for the configuration of mod-source-record-manager with 500 database connections count and mod-source-record-storage with 500 database connections count and 60-sec timeout it was not observed "HTTP 500 Internal Server Error." issue, but the response time for requests like GET /source-storage/records/id/formatted greatly increases during DI (up to 75 sec).

  • Increasing the number of database connections to DB_MAXPOOLSIZE = 200 for mod-source-record-manager and mod-source-record-storage did not give any positive effect on 'HTTP 500 Internal Server Error. If the issue persists, please report it to EBSCO Connect.' but duration of Data Import job was decreased twice.

  • Service CPU utilization did not exceed 36% even at the beginning of all processes together. Without spikes at the beginning and start of DI jobs average CPU usage was about 15%. Instance CPU Utilization did not exceed 22%.

  • Service memory utilization was stable, and no memory leaks were suspected during tests.

  • Each FYR (Fiscal close - end of FY rollover) job is consuming a lot of DB CPU (each spike here corresponds to each FYR job). Approximately DB CPU usage was up to 92%

Recommendations & Jiras (Optional)

Jiras

PERF-582: Response 500 for NLA normal load test with Data Import job (2 SP)Blocked

Test Runs & Results

Test #

# configuration

Test duration

comments

Test #

# configuration

Test duration

comments

1

All workflows started at the same time

1 hour

Res 500 ERROR during Data Import (DI) process

2

Data import and data export started with 15 min delay

1 hour

Res 500 ERROR during Data Import process

3

Data import and data export started with 1 min delay

1 hour

Res 500 ERROR during Data Import process

4

Data import started with 1 min delay and data export started with 20 min delay

35 min

Res 500 ERROR during Data Import process

5

Same as Test# 4 Jenkins configuration with increased DB_MAXPOOLSIZE = 200 for mod_srs and mod_srm

30 min

Res 500 ERROR during Data Import process. Data import duration decreased 2 times.

6

Test without Data import

1 hour

NO errors

 Test results from 1st test run (1st, 2nd and 3rd test run results are similar):

Test #

Workflow name 

Total time it takes to complete workflow

After environment improvement, no server errors were observed (23-06-2023)***

Time-consuming requests for each workflow during DI for default configuration, finished with Response body: HTTP 500 Internal Server Error. If the issue persists, please report it to EBSCO Connect.

 

 

Avg With DI (sec)

95th pct with DI (sec)

Avg no DI (sec)

95th pct no DI (sec)

With DI Avg

Without DI

1

CICO_Checkin

1.238

1.506

1.054

1.591

1.188

1.089

 

2

CICO_Checkout

2.156

2.829

1.650

1.948

2.040

1.933

 

3

IO_View invoices

0.907

1.305

0.763

0.913

1.178

1.015

 

4

IO_Create invoices

1.433

1.815

1.174

1.370

1.689

1.795

 

5

IO_Edit invoices

1.983

2.422

1.581

1.897

2.276

1.015

 

6

IO_Delete invoices

1.070

1.196

0.804

0.927

1.248

1.175

 

7

AIE_Approving Invoices

1.752

2.211

1.453

1.940

3.208

2.740

 

8

VAR_View Authority records

22.037

30.604

0.289

0.381

49.268

0.295

VAR_GET /source-storage/records/marc_id/formatted

9

VTT_View MARC tag table

41.272

61.935

0.987

1.284

75.006

1.052

VTT_GET source-storage/records/{id}/formatted *2

10

VH_View holdings records

27.328

33.579

1.526

1.922

34.645

1.291

VH_GET source-storage/records/{id}/formatted

11

VB_View Bib

22.851

31.634

0.841

1.168

42.175

0.900

VB_GET source-storage/records/{id}/formatted

12

PRO_View patron records

0.672

1.118

0.566

0.883

0.359

0.638

 

13

PRO_Delete patron records

0.892

1.336

0.638

1.070

0.728

0.763

 

14

PRO_Update patron records

1.386

2.097

1.043

1.625

1.166

1.157

 

15

PRO_Create patron records

1.547

1.979

1.098

1.261

1.571

1.286

 

16

LO_View Ledger

0.122

0.458

0.050

0.088

0.125

0.047

 

17

LO_Create ledger

0.684

0.840

0.616

0.761

0.542

0.629

 

18

LO_Edit ledger

0.076

0.094

0.054

0.085

0.698

0.047

 

19

LO_Delete a ledger

0.080

0.129

0.046

0.080

0.073

0.050

 

20

DE_Export bib "Default instances export job profile"

11 sec  (5000 records)

-

5 sec  (5000 records)

-

6 sec (5000 records)

5 sec (5000 records)

 

21

DE_Export holdings "Default holdings export job profile"

3 min 16 sec  (5000 records)

-

26 sec  (5000 records)

-

28 sec (5000 records)

27 sec (5000 records)

 

22

DE_Export authority records "Default authority export job profile"

8 sec (5000 records)

-

3 sec  (5000 records)

-

3 sec (5000 records)

3 sec (5000 records)

 

23

DI "DISC HRID match"

1sec (1 record)

-

-

-

1 sec

-

 

24

DI "DS LA edeposit records update"

17 min 5 sec

-

-

-

3 min 9 sec

-

 

25

DI "DISC New edeposit records"

-

-

-

-

13 sec

-

 

26

DI "DISC New NON edeposit records"

4 sec (5 records)

-

-

-

3 sec

-

 

27

IRO_View item records

24.771

32.327

1.289

1.649

37.804

1.449

IRO_GET source-storage/records/{id}/formatted 

28

IRO_update item records

17.226

33.950

0.998

1.250

-

1.033

IRO_GET source-storage/records/{id}/formatted 

29

IRO_delete item records

23.686