/
Check-in-check-out Test Report (Orchid)

Check-in-check-out Test Report (Orchid)

Apr 11, 2023

Overview

The test's target is to investigate the behaviour of the Check-In/Check-Out activities during fixed load tests.

PERF-470: [Orchid] [Regular performance testing] CI/CO RetestClosed

 

Summary

  • The essential difference (even without absolute comparing): ~20% of degradation for average response time between current test results and:

  • 8 Container instances for ncp5 instead 9 for ncp3 could have a negative impact but not critical

  • No memory leaks were found during the series of tests

  • Based on the longevity test results, the response time for 30 virtual users during the long period of time was degraded by ~ 75% in comparison with the fixed load test for 8 virtual users of the same configuration. Moreover, 'mod-users' CPU Utilization was more than 100% during all test execution. After ~16 hours of the longevity test run, significant degradation of CPU Utilization took place by services involved in Check-in/Check-out operations.

  • Unexpected behaviour of 'mod-search' CPU utilization during fixed load retests with 20 virtual users

  • ERROR LOG Messages have not been found, but there were a lot of WARN LOG Messages (see Additional) connected to the queries usage without an index 

Recommendations

  • Discuss & analyze the situation with WARN LOG Messages (see Additional):

    • Potentially create indexes based on WARN LOG messages

    • Clarify action items for 'The configuration' WARN LOG Messages

  • Discuss & analyze the worst API calls and potential root causes with a development team

  • Decrease the number of virtual users for longevity testing Orchid release candidate (down to 20 or even 8)

  • Discuss & analyze the involvement of 'mod-search' in Check-in/Check-out operations, and pay attention to 'mod-search' CPU utilization during future tests

 

Test Runs 

The following table contains information about a test model and related Grafana snapshots:

Test #

Test Conditions

vUsers

Ramp-up, sec

Duration, sec 

Grafana

1.

circulation_checkInCheckOut_orchid

8

80

2'780

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1679388600000&to=1679391720000&var-percentile=95&var-test_type=fix_load&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

2.

circulation_checkInCheckOut_orchid

20

200

2'900

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1679392500000&to=1679395560000&var-percentile=95&var-test_type=fix_load&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

3.

circulation_checkInCheckOut_orchid

30

300

3'000

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1679396220000&to=1679399400000&var-percentile=95&var-test_type=fix_load&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

4.

circulation_checkInCheckOut_orchid

75

750

3'450

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1679400360000&to=1679404020000&var-percentile=95&var-test_type=fix_load&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

5.

circulation_checkInCheckOut_orchid

30

300

86'700

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1679070600000&to=1679157300000&var-percentile=95&var-test_type=fix_load&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

Based on previous results, the decision of additional execution was made:

Test #

Test Conditions

vUsers

Ramp-up, sec

Duration, sec 

Grafana

1*

circulation_checkInCheckOut_orchid

8

80

2'780

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&var-percentile=95&var-test_type=fix_load&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All&from=1679932800000&to=1679935800000

2*

circulation_checkInCheckOut_orchid

8

80

2'780

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&var-percentile=95&var-test_type=fix_load&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All&from=1679940840000&to=1679943720000

3*

circulation_checkInCheckOut_orchid

20

200

2'900

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&var-percentile=95&var-test_type=fix_load&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All&from=1679944440000&to=1679947500000

4*

circulation_checkInCheckOut_orchid

20

200

2'900

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&var-percentile=95&var-test_type=fix_load&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All&from=1679947980000&to=1679951220000

Results

The following table contains information about test results and their comparison with the baseline (circulation_checkInCheckOut_orchid 8 VU).

Response Times 

Test #

Test Conditions

vUsers

Action

Request count

MIN, ms

AVG, ms

Median, ms

95pct, ms

MAX, ms

Std.Dev

MoE

Confidence 95% 

CoV

1.

circulation_checkInCheckOut_orchid

Baseline

8

Check-in

1'435

379

534

456

1'041

2'770

231

11.95

43.26%

Check-out

1'845

605

909

 

 

797

1'462

3'390

286

13.05

31.46%

2.

circulation_checkInCheckOut_orchid

20

Check-in

3'513

318

600

+12.36%

482

+5.70%

1'196

+14.89%

2'580

291

9.62

48.50%

Check-out

4'682

599

1'110

+22.11%

1'010

+26.73%

1'807

+23.60%

3'670

396

11.34

35.68%

3.

circulation_checkInCheckOut_orchid

30

Check-in

5'109

341

834

+56.18%

662

+45.18%

1'708

+64.07%

4'810

469

12.86

56.24%

Check-out

6'783

609

1'590

+12.36%

1'511

+89.59%

2'777

+89.95%

6'470

635

15.11

39.94%

4.

circulation_checkInCheckOut_orchid

75

Check-in

13'378

337

825

+54.49%

688

+50.88%

1'566

+50.43%

4'230

411

6.96

49.82%

Check-out

17'744

644

1'960

+115.62%

1'904

+138.90%

3'142

+114.91%

6'370

667

9.81

34.03%

5.

circulation_checkInCheckOut_orchid

30

Check-in

208'809

311

550

+3.00%

455

-0.22%

1'061

+1.92%

16'108

257

1.10

46.73%

Check-out

277'292

111

1'597

+75.69%

1'306

+63.86%

3'233

+121.14%

133'698

852

3.17

53.35%

Comparisons

The following table contains information about additional test executions, their results and a comparison with Nolana's results:

Test #

Test Conditions

vUsers

Action

AVG, ms

Median, ms

95pct, ms

Std.Dev

B

circulation_checkInCheckOut_nolana

8

Check-in

456

441

534

 

Check-out

698

672

839

 

1-Retest

circulation_checkInCheckOut_orchid

8

Check-in

534

+17.11%

474

+7.48%

1,027

+92.32%

239

Check-out

811

+16.19%

749

+11.46%

1,201

+43.15%

252

2-Retest

circulation_checkInCheckOut_orchid

8

Check-in

538

+17.98%

475

+7.71%

1,428

+167.42%

241

Check-out

799

+14.47%

747

+11.16%

1,008

+20.14%

214

B

circulation_checkInCheckOut_nolana

20

Check-in

425

414

489

 

Check-out

658

638

758

 

3-Retest

circulation_checkInCheckOut_orchid

20

Check-in

513

+20.71%

460

+11.11%

812

+66.05%

229

Check-out

798

+21.28%

737