/
PTF-Performance retesting of Eureka with native sidecars 3.0.10 (CSP1)

PTF-Performance retesting of Eureka with native sidecars 3.0.10 (CSP1)

Overview

PTF plan to test performance using the upcoming sidecar version 3.0.10-nb (native binaries), which will be included in CSP1.

The goal is to compare the performance of the native binary sidecars v3.0.10 against Sunflower CSP1.

Note! Testing were conducted sidecar version 3.0.10-nb and Light weight token

For the evaluation we will run:

  • CICO (Check-In / Check-Out) tests: 8, 20, 30, and 75 users, duration - 45 minutes;

  • CICO (Check-In / Check-Out) Longivity test: 30-user longevity test for 10 hours;

  • Data Import tests (Create and Update) with 5K, 10K, 25K, 50K, and 100K records to validate sidecar performance under different data volumes.

Ticket:

https://folio-org.atlassian.net/browse/PERF-1209

 

Summary

  • All tests finished successfully without significant errors and module/sidecars crashes.

  • Data-import

    • For Create operations Sidecars 3.0.10-nb demonstrates consistent ~20–30 % improvement for 5k-100k records set

    • For Update operations, 3.0.10-nb has higher update durations (≈ 20–40 % slower) for 10k-100k records set.

  • Check-In and Check-Out. Average response times for both Check-In and Check-Out increased sharply.
    As the number of virtual users grew from 8 to 75, average response times increased from 35% to 17 times

  • Sidecars resource utiization

    a) CPU utilization graph demonstrates a correlation between the increase in workload (load) and the CPU resource consumption across the containers. No anomalies detected;
    b) Memory utilization graph illustrates stable memory usage across the containers throughout the entire testing period, no memory leaks.

  • The stability of Sidecar version 3.0.10-nb has been verified through CICO (Check-In / Check-Out) Longevity Test.

    The test, which simulated a high-load environment by running a 30-user longevity scenario for 10 hours, was completed without any issues. The system showed stability throughout the entire test , with no crashes, performance degradation.

Test Runs 

CICO (Check-In / Check-Out)

Test #

vUsers

Ramp-up, sec

Duration, sec 

Comment

Test #

vUsers

Ramp-up, sec

Duration, sec 

Comment

1

8

80

2700

native binaries sidecar

2

20

200

2700

native binaries sidecar

3

30

300

2700

native binaries sidecar

4

75

750

2700

native binaries sidecar

5

30

300

36000

native binaries sidecar

longevity test

 

Data Import

Test #

Size

Profile

Test #

Size

Profile

1

5K

PTF-Create-2

2

10K

PTF-Create-2

3

25K

PTF-Create-2

4

50K

PTF-Create-2

5

100K

PTF-Create-2

6

5K

PTF-UpdateSuccess-6

7

10K

PTF-UpdateSuccess-6

8

25K

PTF-UpdateSuccess-6

9

50K

PTF-UpdateSuccess-6

10

100K

PTF-UpdateSuccess-6

Results/Comparisons

CICO comparison

S CSP1 - Sunflower CSP1 release original testing results

S(version-nb) - Sunflower version results with native binaries sidecars

 

Response Times, milliseconds

8 vUsers   (test #1)

20 vUsers   (test #2)

30 vUsers   (test #3)

75 vUsers   (test #4)

Requests

S
CSP1

S
(3.0.4-nb)

S
(3.0.10-nb)

S

CSP1

S
(3.0.4-nb)

S
(3.0.10-nb)

S

CSP1

S
(3.0.4-nb)

S
(3.0.10-nb)

S

CSP1

S
(3.0.4-nb)

S
(3.0.10-nb)

Label

Average

Average

Average

Average

Check-Out Controller

905

835

1225

875

835

1456

889

835

1409

1456

835

6607

Check-In Controller

414

377

618

396

377

766

403

377

2141

653

377

10913

CSP1 Check-in-check-out Test Report Sunflower [non-ECS]. https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/1209040897/Check-in-check-out+Test+Report+Sunflower+non-ECS+CSP1?themeState=dark%3Adark+light%3Alight+spacing%3Aspacing+typography%3Atypography-refreshed+colorMode%3Alight

Data Import Comparison

This set of test were performed to check sidecars durability during high load and compare durations of data import.

Test #

Data-import test

Profile

Duration

Sunflower CSP1

 

Duration

Sunflower

3.0.4-nb

Duration

Sunflower

3.0.10-nb

Delta Duration (CSP1 - 3.0.10-nb)

Direction

Status

Test #

Data-import test

Profile

Duration

Sunflower CSP1

 

Duration

Sunflower

3.0.4-nb

Duration

Sunflower

3.0.10-nb

Delta Duration (CSP1 - 3.0.10-nb)

Direction

Status

1.1

5k MARC BIB Create

PTF - Create 2

3 min 4 s

2 min 24 s

2 min 01 s

1 min 3 s

faster

Completed

1.2 

10k MARC BIB Create

PTF - Create 2

5 min 25 s

4 min 56 s

4 min 14 s

1 min 11 s

faster

Completed

1.3

25k MARC BIB Create

PTF - Create 2

11 min 27 s

11 min 44 s

10 min 09 s

1 min 18 s

faster

Completed 

1.4

50k MARC BIB Create

PTF - Create 2

24 min 33 s

20 min 54 s

20 min 14 s

4 min 19 s

faster

Completed 

1.5

100k MARC BIB Create

PTF - Create 2

54 min 31 s

42 min 39 s

44 min 16 s

10 min 15 s

faster

Completed

2.1 

5k MARC BIB Update

PTF - Updates Success - 6

5 min 7 s

3 min 41 s

2 min 37 s

2 min 30 s

faster

Completed

2.2

10k MARC BIB Update

PTF - Updates Success - 6

8 min 13

8 min 7 s

6 min 45 s

1 min 28 s

faster

Completed

2.3

25k MARC BIB Update

PTF - Updates Success - 6

19 min 27 s

18 min 11 s

17 min 04 s

2 min 23 s

faster

Completed

2.4

50k MARC BIB Update

PTF - Updates Success - 6

37 min 11 s

35 min 59 s

35 min 20 s

1 min 51 s

faster

Completed

2.5

100k MARC BIB Update

PTF - Updates Success - 6

1 hr 29 min

1 hr 17 min

1 hr 09 min

20 min

faster

Completed with erros*

*org.folio.inventory.dataimport.exceptions.DataImportException: handle:: Failed to update Instance with id = 39ca526c-0598-4faf-a8eb-7a25e707a46b. Instance with source=LINKED_DATA cannot be updated using Data Import. Please use Linked Data Editor
This errors related due to the data preparation for Linked Data performance testing. 1 or 2 records failed during DI

CSP1 Data-import Test Report Sunflower [non-ECS].https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/1200652289

Resource Utilization

Below charts represents resouce usage during tests.

CICO Test 1-4

CPU utilization during CICO tests

image-20251008-104430.png

Memory utilization during CICO tests

image-20251008-104824.png

Database CPU utilization during CICO tests. Before each test you can observe CPU spikes - data preparation for CICO

image-20251008-105032.png

Database Connections

image-20251008-105210.png

RDS Performace metrics

image-20251008-105407.png
image-20251008-105422.png

Sidecar resources utilization

Sidecars CPU utiization

The provided graph clearly demonstrates a direct correlation between the increase in workload (load) and the CPU resource consumption across the containers.

image-20251009-145355.png

Sidecars Memory utiization

The provided graph illustrates stable memory usage across the containers throughout the entire testing period, demonstrating resilience despite fluctuations in the workload (load).

image-20251009-145626.png

CICO Test 5. Longevity test

CPU utilization during CICO longivity test

image-20251007-105239.png

Memory utilization during CICO longivity test

image-20251008-114641.png

Database CPU utilization

image-20251008-115019.png

Database Connections

image-20251008-115242.png

 

RDS Performace metrics

image-20251008-114840.png
image-20251008-114853.png

Data Import

Test 1-5 with MarcBib create profile


CPU utilization during Data-import tests

image-20251008-120611.png
image-20251008-120356.png

Memory utilization during Data-import tests

image-20251008-122440.png

Database CPU utilization

image-20251008-122719.png

Database Connections

image-20251008-123120.png


RDS Performace metrics

image-20251008-145409.png

Test 6-10 with MarcBib update profile

CPU utilization during Data-import tests

image-20251008-120904.png

Memory utilization during Data-import tests

image-20251008-122250.png

Database CPU Utilization

image-20251008-122841.png

Database Connections

image-20251008-123208.png

RDS Performace metrics

image-20251008-145521.png
image-20251008-145532.png

Appendix

Infrastructure

PTF -environment secp1

PTF -environment secp1