Versions Compared

Version Old Version 5 New Version Current
Changes made by

Leonid Kolesnykov

Leonid Kolesnykov

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

...

...

...

...

...

...

...

...

...

Table of Contents
Overview

...

This document contains the results of testing different configurations of modules  with changes in JAVA_OPTS replacing XMX with MaxRAMPercentage. The main KPI for these tests are Memory consumption of modules, Data import durations and Check-in / Check-out response times.

...

  • Replacing XMX with MaxRAMPercentage do not affect main KPI's. Comparing DI's durations in tests #1 and #2 deltas do not exceed 4 minutes that is acceptable.
  • After changing MaxRAMPercentage from default values to recommended 66%  Check-in / Check-out response times increased during data import for CI - 15%, for CO - 7%, and it also affected Data import durations - 8 minutes difference in DI test #6.
  • Comparing DI durations between configuration #2 and #3 (with limited MaxRAMPercentage) we see 4 minutes differencethat DI slower in configuration #3. Previously known issue with mod-inventory (container stop because of CPU growing) appear much faster in configuration #3 with limited MaxRAMPercentage.
  • CPU utilization top three modules:
ServiceConfiguration #1Configuration #2Configuration #3
mod-inventory-b254.27305.56216.98
mod-quick-marc-b120.8358.4987.44
nginx-okapi91.7356.195.49
  • Memory consumption top three modules:
ServiceConfiguration #1Configuration #2Configuration #3
mod-inventory-b99.4105.84102.32
mod-search-b96.4389.196.81
mod-users-b63.0861.5860.63
  • Service CPU utilization changed between tests because of the cluster restarts
  • RDS CPU utilization was close to 96% during CI/CO + Data import 
  • RDS DB Connections were close to 600 during CI/CO + Data import 

...

Request*Response time (avg, sec)
Pure CICOCICO + 100K MARC BIB Create
Check-Out Controller0.9652.121
Check-In Controller0.471.24
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)0.3140.879
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)0.2080.73
GET_circulation/loans (Submit_barcode_checkout)0.1420.259
GET_circulation/loans (Submit_patron_barcode)0.10.203
GET_inventory/items (Submit_barcode_checkin)0.0550.182Authorize_simple0.178POST_bl-users/login0.178GET_inventory/items (Submit_barcode_checkout)0.142GET_automated-patron-blocks (Submit_patron_barcode)0.124

*Top requests were taken where response times >= 100ms.

Service CPU Utilization

*Top requests were taken where response times >= 100ms.

Service CPU Utilization

Expand
titleCPU utilization all tests


AVERAGES


ServiceConfiguration #1Configuration #2Configuration #3
mod-inventory-b254.27305.56216.98
mod-quick-marc-b120.8358.4987.44
nginx-okapi91.7356.195.49
mod-di-converter-storage-b7433.0880.35
okapi-b45.6932.344.08
mod-source-record-storage-b39.419.1448.36
mod-users-b38.2733.133.79
mod-source-record-manager-b29.5415.2332.39
mod-inventory-storage-b21.9313.1720.09
mod-authtoken-b19.6715.9919.56
mod-configuration-b17.7216.5514.97
mod-feesfines-b17.1216.7115.97
mod-pubsub-b10.199.529.74
mod-permissions-b8.074.736.74
pub-okapi7.748.416.47
mod-circulation-storage-b3.723.083.28
mod-circulation-b3.033.052.95
mod-password-validator-b2.712.42.65
mod-data-import-b1.812.951.98


...

Infrastructure

PTF -environment pcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

    NameMemory GIBvCPUsmax_connections

    db.r6g.xlarge

    		32 GiB4 vCPUs2731


  • MSK tenant
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3
ModuleTask Def. RevisionTask Def. Revision with MaxRAMPercentage = by defaultTask Def. Revision with MaxRAMPercentage = 66.0Module VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabledMaxRAMPercentage
Tests 		MaxRAMPercentage = recommended (66%)
pcp1-pvt












Thu Apr 04 09:40:28 UTC 2024












mod-data-import202324mod-data-import:3.0.71204818442561292384512FALSE70.166.0
mod-authtoken161718mod-authtoken:2.14.121440115251292288128FALSE80.066.0
mod-configuration101213mod-configuration:5.9.22102489612876888128FALSE85.766.0
mod-users-bl101112mod-users-bl:7.6.021440115251292288128FALSE80.066.0
mod-inventory-storage151718mod-inventory-storage:27.0.424096369020483076384512FALSE83.466.0
mod-circulation-storage141516mod-circulation-storage:17.1.722880259215361814384512FALSE70.066.0
mod-source-record-storage181920mod-source-record-storage:5.7.525600500020483500384512FALSE70.066.0
mod-inventory151617mod-inventory:20.1.822880259210241814384512FALSE70.066.0
mod-di-converter-storage181920mod-di-converter-storage:2.1.52102489612876888128FALSE85.766.0
mod-circulation141516mod-circulation:24.0.1122880259215361814384512FALSE70.066.0
mod-pubsub131415mod-pubsub:2.11.32153614401024922384512FALSE64.066.0
mod-users343536mod-users:19.3.0-SNAPSHOT.6772102489612876888128FALSE85.766.0
mod-patron-blocks101112mod-patron-blocks:1.9.021024896102476888128FALSE85.766.0
mod-source-record-manager171920mod-source-record-manager:3.7.825600500020483500384512FALSE70.066.0
nginx-okapi9

nginx-okapi:2023.06.1421024896128000FALSE0.0
okapi-b111213okapi:5.1.23168414401024922384512FALSE64.066.0
mod-feesfines111213mod-feesfines:19.0.02102489612876888128FALSE85.766.0

Methodology/Approach

  1. Run CI/CO for 24 hours.
  2. Run DI script with 12 DI's sequentially after ramp-up period for CI/CO.
  3. After getting baseline, repeat CI/CO with changed revisions of modules with needed configuration of JAVA_OPTS - replaced XMX with MaxRAMPercentage. 
  4. Before tests restart the cluster to get consistent results.

Additional

...

Screenshots of graphs or charts

Spreadsheet with notes

View file
nameResults_pcp1_PERF-893_RAM_vs_XMX.xlsx
height250

...