Versions Compared

Old Version 2

changes.mady.by.user Mykhailo Petryshyn

Saved on

compared with

New Version Current

changes.mady.by.user Mykhailo Petryshyn

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Test Run 1: Running without New Relic.

  • Test Run 2: Running with the New Relic monitoring tool enabled.

  • Test Run 3: Running with the New Relic monitoring tool enabled and changed JVM mod-oa parameter from -XX:MaxMetaspaceSize=128m to -XX:MaxMetaspaceSize=256m

These tests were performed using 20 virtual users to simulate the load on the system. The environment was used for these tests QCP1.

The primary goal of these tests is to assess whether the New Relic monitoring tool impacts the system’s performance.
Ticket:

Jira Legacy
serverSystem Jira
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-920

Summary

Test Run 1 (Without New Relic). Generate 3159 Check-out transactions and 2420 Check-in transactions. The error rate during the test was 0.04 %. Median Response time for CO = 1.585 960 seconds and median Response time for CI = 1.214 640 seconds.

Test Run 2 (With New Relic). Generate 3180 Check-out transactions and 2381 Check-in transactions. The error rate during the test was 0.04 %. Median Response time for CO = 1.594 968 seconds and median Response time for CI = 1,193 seconds.
Conclusion. Two 594 seconds.
Test Run 3 (With New Relic + mod-oa(256m)). Generate 3212 Check-out transactions and 2402 Check-in transactions. The error rate during the test was 0.04 %. Median Response time for CO = 948 seconds and median Response time for CI = 588 seconds.

Conclusion. Two separate tests were conducted in order to determine the impact of enabling New Relic monitoring tool on the Check-in/Check-Out process. The aim of the test was to identify any significant variations in key performance metrics that could potentially be attributed to New Relic.

...

In conclusion, the results of the tests demonstrate that utilizing the New Relic monitoring tool does not adversely affect the check-in/check-out process and the overall system performance remains unaffected.

...

, but after deployment of New RElic we observed high CPU consumption and OOM errors on mod-oa service.

  1. After increasing mod-oa parameter from -XX:MaxMetaspaceSize=128m to -XX:MaxMetaspaceSize=256m memory CPU utilization decrease to 80%.

Recommendations & Jiras (Optional)

  • While the current tests provided insights into how New Relic impacts

...

  • the CICO process, it's recommended for future analysis to cover a more extensive range of operations to fully evaluate the impact of New Relic on the system performance during data-import, data-export, or other workflows.

  • Investigate errors described in the Error section, high CPU consumption and OutOfMemoryError on mod-oa module

Test Runs 

  • Test Run 1: Running without New Relic.

    • Test setup: 20 Virtual users, 4000 seconds, 200 seconds of rump up period.

  • Test Run 2: Running with the New Relic monitoring tool enabled.

    • Test

    Run 2

    • setup: 20 Virtual users, 4000 seconds, 200 seconds of rump up period

  • Test Run 3: Running with the New Relic monitoring tool enabled and mod-oa(256m).

    • Test setup: 20 Virtual users, 4000 seconds, 200 seconds of rump up period

Results

NAME

without

Test 1. Without New Relic

Test 2. With New Relic

with

Test 3. With New Relic + mod-oa(256m)

CO

CI

CO

CI

CO

CI

TTL REQ, COUNT

3159

2420

3180

2381

3212

2402

THRGHPT, REQ/SEC

1.585

1.214

1.594

1.193

1.638

1.239

ERRORS, COUNT

36

23

43

16

36

24

MIN, MS

24

39

25

46

28

35

MEDIAN, MS

959

640

968

594

947

588

PCT95, MS

1,226

759

1,247

740

740

1,150

682

MAX, MS

6,504

20,712

14,834

12,165

12,694

17,582

Resource utilization

CPU utilization

The slight variation in CPU usage within the tolerance limit of 5% between the test runs poses no significant concern and is well within acceptable boundaries.

Test Run 1:

...

Test Run 2:

...


Test Run 3:

...


Top 16 CPU-consuming services

Module

Avrg CPU Utlz (Test 1) %

Avrg CPU Utlz (Test 2)%

Avrg CPU Utlz (Test 3)%

mod-users-b

46.04

45.43

47

nginx-okapi

33.77

27.58

32

mod-configuration-b

29.59

27.53

31

mod-feesfines-b

27.13

27.68

27

okapi-b

25.76

30.62

28

mod-authtoken-b

22.81

19.87

17

mod-permissions-b

18.43

19.49

21

mod-consortia-b

20.87

22.13

20

mod-dcb-b

17.56

17.94

17.5

mod-audit-b

15.98

11.50

13.2

mod-inventory-b

14.23

15.14

14.4

mod-agreements-b

11.36

12.43

11.7

mod-pubsub-b

10.64

11.49

12.1

mod-notes-b

8

10.19

11

mod-calendar-b

9.69

9.31

9

pub-okapi

9.02

10.12

10

Memory utilization

Memory Usage: No changes in memory usage patterns were observed upon enabling New Relic. No memory leaks in any module.

Test Run 1:

...

Test Run 2:

...

Test 3:

...


Top Memory-consuming services

Module

Avrg Memory Utlz (Test 1) %

Avrg Memory Utlz (Test 2)%

mod-data-export-b

-

104.04

mod-users-b

46.04

45.43

nginx-okapi

33.77

27.58

mod-configuration-b

29.59

27.53

mod-feesfines-b

27.13

27.68

okapi-b

25.76

30.62

mod-authtoken-b

22.81

19.87

mod-permissions-b

18.23

19.49

mod-consortia-b

20.87

-

mod-dcb-b

17.56

17.94

mod-audit-b

15.98

18.568

mod-inventory-b

14.23

15.14

Errors

After the deployment of New Relic on QCP1, two services were identified as having high CPU consumption: mod-oa-b and mod-data-export. These two services had new task definitions with new relic variables.

...

mod-oa was throwing error messages “OutOfMemoryError“

...



Infrastructure


PTF -environment qcp1

...