PTF-Performance testing of Eureka with native sidecars
Overview
Sidecars built with native binaries may perform better than with the JVM in the middle. This page contains results and comparisons for performance testing of sidecars with native binaries vs original JVM sidecars.
Tested with 2 different sidecar versions 3.0.4-nb (native binaries) and 3.0.5 (original JVM based sidecars).
For testing was chosen:
CICO (Check-In Check-Out) set of tests with 8, 20, 30, 75 users and 30 user longevity test for 10 hours.
Data Import set Creates and Updates 5K, 10K, 25K, 50K, 100K to check how sidecars will perform.
Summary
All tests finished successfully without significant errors and module/sidecars crashes.
There’s visible improvements in response times for CICO comparing with initial testing set. With latest module versions and 3.0.5 sidecars response times are about the same as with native binaries.
Comparison of resources usage between native binaries sidecars (snapshot from 3.0.4)and 3.0.5 sidecars:
CPU usage - 5-15% higher on native binaries sidecars
Significant memory usage reduce on native binaries sidecars: 6%-10% memory usage during test versus 40-50% memory usage on v3.0.5 sidecar (with JVM based sidecars)
Test Runs
CICO
Test # | vUsers | Ramp-up, sec | Duration, sec | Comment |
|---|---|---|---|---|
1 | 8 | 80 | 2700 | native binaries sidecar |
2 | 20 | 200 | 2700 | native binaries sidecar |
3 | 30 | 300 | 2700 | native binaries sidecar |
4 | 75 | 750 | 2700 | native binaries sidecar |
5 | 30 | 300 | 36000 | native binaries sidecar longevity test |
1 | 8 | 80 | 2700 | JVM based sidecars |
2 | 20 | 200 | 2700 | JVM based sidecars |
3 | 30 | 300 | 2700 | JVM based sidecars |
4 | 75 | 750 | 2700 | JVM based sidecars |
5 | 30 | 300 | 36000 | JVM based sidecars longevity test |
Data Import
Test # | Size | Profile |
|---|---|---|
1 | 5K | PTF-Create-2 |
2 | 10K | PTF-Create-2 |
3 | 25K | PTF-Create-2 |
4 | 50K | PTF-Create-2 |
5 | 100K | PTF-Create-2 |
6 | 5K | PTF-UpdateSuccess-6 |
7 | 10K | PTF-UpdateSuccess-6 |
8 | 25K | PTF-UpdateSuccess-6 |
9 | 50K | PTF-UpdateSuccess-6 |
10 | 100K | PTF-UpdateSuccess-6 |
Results/Comparisons
CICO comparison
*S - Sunflower release original testing results
**S(native) - Sunflower CICO results with native binaries sidecars
All tests finished successfully with 15-20% better response times than they were during initial sunflower testing.
| Response Times, milliseconds | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
8 vUsers (test #1) | 20 vUsers (test #2) | 30 vUsers (test #3) | 75 vUsers (test #4) | |||||||||||||
Requests | S | S(native) | Delta | S | S(native) | Delta | S | S(native) | Delta | S | S(native) | Delta | ||||
Label | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | Average | Delta,ms | Delta,% | ||||
Check-Out Controller | 995 | 835 | -160 | -16.08% | 1000 | 841 | -159 | -15.90% | 1040 | 854 | -186 | -17.88% | 1629 | 1220 | -409 | -25.11% |
Check-In Controller | 450 | 377 | -73 | -16.22% | 456 | 376 | -80 | -17.54% | 464 | 380 | -84 | -18.10% | 782 | 551 | -231 | -29.54% |
CICO Comparison vs Ramsons Okapi
Below comparison of native sidecars vs Ramsons Okapi. most of results are the same.
| Response Times, milliseconds | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
8 vUsers (test #1) | 20 vUsers (test #2) | 30 vUsers (test #3) | 75 vUsers (test #4) | |||||||||||||
Requests | Okapi | S(native) | Delta | Okapi | S(native) | Delta | Okapi | S(native) | Delta | Okapi | S(native) | Delta | ||||
Label | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | ||||
Check-Out Controller | 811 | 835 | +24 | 2.9% | 835.63 | 841 | 6 | +0.6 | 822 | 854 | 32 | +3 | 1100.28 | 1220 | 119 | +10 |
Check-In Controller | 470 | 377 | -93 | -19.7% | 482.65 | 376 | -106 | -22% | 455 | 380 | -75 | -16% | 606.04 | 551 | -55 | -9 |
CICO vs latest modules and 3.0.5 sidecars
*S+ - Sunflower release with latest modules versions and sidecar 3.0.5 version
**S(native) - Sunflower CICO results with native binaries sidecars
There’s no significant differences between native binaries and 3.0.5 sidecars on response times.
| Response Times, milliseconds | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
8 vUsers (test #1) | 20 vUsers (test #2) | 30 vUsers (test #3) | 75 vUsers (test #4) | |||||||||||||
Requests | S+ | S(native) | Delta | S+ | S(native) | Delta | S+ | S(native) | Delta | S+ | S(native) | Delta | ||||
Label | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | Average | Delta,ms | Delta,% | ||||
Check-Out Controller | 884 | 835 | -49 | -5.54% | 819 | 841 | +22 | +2% | 839 | 854 | +15 | 1.7% | 1340 | 1220 | -120 | -9% |
Check-In Controller | 420 | 377 | -43 | -10% | 370 | 376 | +6 | +1.6% | 380 | 380 | 0 | 0 | 609 | 551 | -58 | -10% |
Data Import Comparison
This set of test were performed to check sidecars durability during high load and compare durations of data import.
Test # | Data-import test | Profile |
|---|