[Poppy] Check-out API with enabled check-out lock feature
Overview
Test goal is to assess performance impact of Check-out lock feature on Check-out functionality.
Ticket: 
Summary
Load tests showed that there is no significant influence of check-out lock on performance of the application.
Average response time of check-out request is about 0.5s (level of load - 6.6 requests/minute).
Recomendations and Jiras
It would be usefull to implement some additional monitoring for the check-out lock feature. It can include such metrics as number of locks accross the users, locking history and rate of locking to closely monitor it for performance implications.
Test runs
Scenario | Level of load | Configuration | Response time average | Response time 95perc |
|---|---|---|---|---|
Check-out API | 10 virtual users, 20 requests each (during 30 minutes). Overal load - 6.6 requests/minute | Check-out lock enabled* | 0.558 | 2.152 |
Check-out lock disabled | 0.513 | 0.556 | ||
100 virtual** users, 20 requests each (during 30 minutes). Overal load - 66 requests/minute | Check-out lock enabled* | 0.379 | 0.476 | |
Check-out lock disabled | 0.336 | 0.391 |
*Check-out lock feature was enabled with the following parameters:
"noOfRetryAttempts" - 30
"retryInterval" - 250
"lockTtl" - 2000
**Tests were repeated with higher level of load (66 requests/minute) to ensure that there are enough data points for statistics and results are accurate.
Test results
Check-out lock enabled
Check-out lock disabled
Check-out lock enabled
Check-out lock disabled
Service CPU Utilization
Memory Utilization
DB CPU Utilization
DB Connections
DB Load
Top SQL
Check-out lock enabled
Check-out lock disabled
Additional Information
In order to make sure that check-out lock work correctly, next requests were found in logs during load tests with check-out lock enabled:
At the screenshot we can see that lock is being created in check-out-lock-storage and then deleted almost at the same moment. Deletion of lock happens when check-out for item is done.
Appendix
Infrastructure
PTF -environment pcp1
10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instances, writer
MSK ptf-kakfa-3
4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
Module | Task Def. Revision | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize | R/W split enabled |
|---|---|---|---|---|---|---|---|---|---|
7 | 1 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | false | |
17 | 1 | 1024 | 896 | 128 | 768 | 88 | 128 | false | |
8 | 1 | 1024 | 896 | 128 | 768 | 88 | 128 | false | |
7 | 1 | 2880 | 2592 | 1536 | 1814 | 384 | 512 | false | |
7 | 1 | 1536 | 1440 | 1024 | 922 | 384 | 512 | false | |
7 | 1 | 1024 | 896 | 128 | 768 | 88 | 128 | false | |
7 | 1 | 1684 | 1440 | 1024 | 922 | 384 | 512 | false | |
7 | 1 | 2880 | 2592 | 1024 | 1814 | 384 | 512 | false | |
8 | 1 | 2880 | 2592 | 1536 | 768 | 88 | 128 | false | |
7 | 1 | 1024 | 896 | 1024 | 768 | 88 | 128 | false | |
7 | 1 | 1440 | 1152 | 512 | 922 | 88 | 128 | false |
Methodology
Conduct baseline tests with check-out lock disabled.
Conduct verification tests with check-out lock enabled. It can be done directly in the database (table [tenant]_mod_settings.settings).
Value used for the testing:
{ "value": "{\"checkOutLockFeatureEnabled\":true,\"noOfRetryAttempts\":30,\"retryInterval\":250,\"lockTtl\":2000}" }