MaxRAMPercentage vs Xmx with Data Import and Check-ins Check-outs (Poppy)
Overview
This document contains the results of testing different configurations of modules with changes in JAVA_OPTS replacing XMX with MaxRAMPercentage. The main KPI for these tests are Memory consumption of modules, Data import durations and Check-in / Check-out response times.
Ticket: https://folio-org.atlassian.net/browse/PERF-893
Summary
Replacing XMX with MaxRAMPercentage do not affect main KPI's. Comparing DI's durations in tests #1 and #2 deltas do not exceed 4 minutes that is acceptable.
After changing MaxRAMPercentage from default values to recommended 66% Check-in / Check-out response times increased during data import for CI - 15%, for CO - 7%, and it also affected Data import durations - 8 minutes difference in DI test #6.
Comparing DI durations between configuration #2 and #3 (with limited MaxRAMPercentage) we see that DI slower in configuration #3. Previously known issue with mod-inventory (container stop because of CPU growing) appear much faster in configuration #3 with limited MaxRAMPercentage.
CPU utilization top three modules:
Service | Configuration #1 | Configuration #2 | Configuration #3 |
mod-inventory-b | 254.27 | 305.56 | 216.98 |
mod-quick-marc-b | 120.83 | 58.49 | 87.44 |
nginx-okapi | 91.73 | 56.1 | 95.49 |
Memory consumption top three modules:
Service | Configuration #1 | Configuration #2 | Configuration #3 |
mod-inventory-b | 99.4 | 105.84 | 102.32 |
mod-search-b | 96.43 | 89.1 | 96.81 |
mod-users-b | 63.08 | 61.58 | 60.63 |
Service CPU utilization changed between tests because of the cluster restarts
RDS CPU utilization was close to 96% during CI/CO + Data import
RDS DB Connections were close to 600 during CI/CO + Data import
Test Runs
Test / Configuration # | Scenario | Load level |
|
|---|---|---|---|
1 | 20 vUsers longevity CI/CO + DI MARC Bib Create | 12 Data Import tests with 100K sequentially (with 5 min pause) |
|
2 | 20 vUsers longevity CI/CO + DI MARC Bib Create | 12 Data Import tests with 100K sequentially (with 5 min pause) |
|
3 | 20 vUsers longevity CI/CO + DI MARC Bib Create | 12 Data Import tests with 100K sequentially (with 5 min pause) |
|
Test Results
Comparison
The following table compares test results
Configuration #1 BASELINE, XMX | Configuration #2 default MaxRAMPercentage | Configuration #3 recommended MaxRAMPercentage, 66% | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
16 hours CICO | Test # | hh:mm:ss | 24 hours CI/CO | Test # | hh:mm:ss | delta, configuration #2 - #1 | 24 hours CI/CO | Test # | hh:mm:ss | delta, configuration #3 - #1 |
|
|
|
|
|
|
|
| DI 100k Create #12 | 00:56:32 | These results after 1 container for mod-inventory stopped. |
|
|
|
| DI 100k Create #11 | 01:39:09 | 4 1k jobs stopped |
| DI 100k Create #11 | 00:55:58 | |
|
|
|
| DI 100k Create #10 | 01:03:19 |
|
| DI 100k Create #10 | 00:57:00 | |
|
|
|
| DI 100k Create #9 | 00:59:40 |
|
| DI 100k Create #9 | 00:56:10 | |
| DI 100k Create #8 | 00:55:01 |
| DI 100k Create #8 | 00:58:47 | 00:03:46 |
| DI 100k Create #8 | 00:55:23 | |
| DI 100k Create #7 | 00:54:43 |
| DI 100k Create #7 | 00:58:08 | 00:03:26 |
| DI 100k Create #7 | 02:46:27 | 3 1k jobs stopped |
| DI 100k Create #6 | 00:55:25 |
| DI 100k Create #6 | 00:56:09 | 00:00:44 |
| DI 100k Create #6 | 01:03:07 | 00:07:42 |
| DI 100k Create #5 | 00:54:45 |
| DI 100k Create #5 | 00:58:19 | 00:03:33 |
| DI 100k Create #5 | 01:03:37 | 00:08:52 |
| DI 100k Create #4 | 00:55:38 |
| DI 100k Create #4 | 00:54:27 | 00:01:11 |
| DI 100k Create #4 | 00:59:11 | 00:03:33 |
| DI 100k Create #3 | 00:53:30 |
| DI 100k Create #3 | 00:54:36 | 00:01:06 |
| DI 100k Create #3 | 00:58:34 | 00:05:04 |
| DI 100k Create #2 | 00:53:16 |
| DI 100k Create #2 | 00:53:03 | 00:00:13 |
| DI 100k Create #2 | 00:57:59 | 00:04:43 |
| DI 100k Create #1 | 00:52:51 |
| DI 100k Create #1 | 01:05:10 | 00:12:18 |
| DI 100k Create #1 | 01:05:42 | 00:12:50 |
|
|
|
|
|
|
|
|
|
|
|
Results from test #8 | DI + | DI - |
| DI + | DI - |
|
| DI + | DI - | Comparison CI/CO + DI |
CI | 1220 | 538 | CI | 1078 | 486 |
| CI | 1240 | 470 | 15% |
CO | 2311 | 978 | CO | 1967 | 1041 |
| CO | 2121 | 965 | 7% |
Detailed CICO response time for CICO with DI
Request* | Response time (avg, sec) | |
|---|---|---|
Pure CICO | CICO + 100K MARC BIB Create | |
Check-Out Controller | 0.965 | 2.121 |
Check-In Controller | 0.47 | 1.24 |
POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 0.314 | 0.879 |
POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 0.208 | 0.73 |
GET_circulation/loans (Submit_barcode_checkout) | 0.142 | 0.259 |
GET_circulation/loans (Submit_patron_barcode) | 0.1 | 0.203 |
GET_inventory/items (Submit_barcode_checkin) | 0.055 | 0.182 |
*Top requests were taken where response times >= 100ms.