/
PTF - Performance testing of CPU=0 and tasks placement strategy for Services (QCP1)

PTF - Performance testing of CPU=0 and tasks placement strategy for Services (QCP1)

Feb 25, 2025

Overview

  • In this report, PTF investigates the impact of setting CPU=0 and disabling the distinctInstance task placement strategy on FOLIO’s performance, with and without New Relic/OpenTelemetry enabled. Previous observations suggested that this configuration negatively affects system performance. The goal of these tests was to reproduce the issue and evaluate whether performance could be improved by adjusting CPU allocations and enabling distinctInstance placement. Through a series of experiments, we analyzed how these settings influence system behavior, ensuring that CPU allocation and task distribution strategies are optimized for stability and efficiency.

StoryPERF-1071: Test with various scenarios for CPU and task placement strategyClosed 

Summary

  • The performance tests conducted in this report indicate that neither the distinctInstance placement strategy nor the use of New Relic/OpenTelemetry had a significant impact on system performance. Across all tests, performance variations remained within a 5% margin. The slight difference observed in Test №3 was attributed to a lower instance count rather than the placement strategy itself. Further tests (№4 and №5) confirmed these findings, showing consistent results regardless of the number of instances used. Additionally, disabling New Relic did not affect performance, suggesting that monitoring overhead was negligible. Overall, the experiments demonstrate that the tested configurations do not introduce meaningful performance differences.

  • It appears that increasing the number of virtual users in our test led to contention in the database, specifically due to a high volume of concurrent updates on the same row in the auth_attempts table by mod-login. The query:

    UPDATE fs09000000_mod_login.auth_attempts SET jsonb = $1::jsonb WHERE id='9883ca16-ef27-41f7-81d7-6693b79cddad'
    suggests that multiple sessions are attempting to modify the same record simultaneously, leading to row-level locking. As a result, transactions are waiting in the background for the lock to be released, potentially causing performance degradation or request timeouts.

Test Runs

Test #

Description

Status

Test #

Description

Status

Test 1

New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned ON, CPU values set for list of modules

Completed

Test 2

New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned OFF, CPU values set for list of modules

Completed

Test 3

New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned OFF, CPU values set 0 for all services

Completed

Test 4

New Relic/OpenTelemetry enabled, OTEL value for mod-inventory and mod-circulation set TRUE, distinctInstance placement strategy turned OFF, CPU values set 0 for all services

Completed

Test 5

New Relic/OpenTelemetry enabled, OTEL value for mod-inventory and mod-circulation set TRUE, distinctInstance placement strategy turned OFF, CPU values set for list of modules

Completed

Test 6

New Relic/OpenTelemetry disabled, distinctInstance placement strategy turned OFF, CPU values set 0 for all services

Completed

Test 7

New Relic/OpenTelemetry disabled, distinctInstance placement strategy turned OFF, CPU values set for list of modules

Completed

Test Results

This table contains response time for Check In\Check Out and RTAC tests. 

Requests

Test №1
Response time, ms

Test №2
Response time, ms

Test №3
Response time, ms

Test №4
Response time, ms

Test №5
Response time, ms

Test №6
Response time, ms

Test №7
Response time, ms

 

Average

Average

Average

Average

Average

Average

Average

Check-In Controller

995

965

1057

886

922

929

917

Check-Out Controller

1518

1502

1637

1352

1392

1431

1447

RTAC

1174

1136

1258

1087

1069

1085

1091

 

 

Test №1-2-3

Test №1: Modules have CPU = 0, and distinctInstance placement strategy is OFF.
Goal: Establish baseline performance metrics for comparison with subsequent configurations.

Test №2: Modules are assigned specific CPU values, with distinctInstance placement strategy still OFF.
Goal: Evaluate whether performance improves compared to the baseline (Test №1).

Test №3: Modules are assigned specific CPU values, and distinctInstance placement strategy is ON.
Goal: Assess whether enabling distinctInstance further enhances performance over the previous tests.

Results: Performance remained nearly the same across all tests, with differences of less than 5%. Test №3 showed a slight variance due to having 5 instances instead of 6, as used in Tests №1 and №2, indicating no negative impact from disabling the distinctInstance placement strategy.

Service CPU Utilization

Here we can see Test №1 with CPU=VALUE  and that mod-rtac module used 112% CPU power.

Here we can see Test №2 with CPU=VALUE  and that mod-rtac module used 109% CPU power.

Here we can see Test №3 with CPU=0 and that mod-nginx-okapi and okapi modules used 12% Instances CPU power.

Service Memory Utilization

Here we can see that all modules show a stable trend.

 

Kafka metrics

OpenSearch Data Nodes metrics

DB CPU Utilization

DB CPU was 85% average.

DB Connections

Max number of DB connections was 950.

DB load

Top SQL-queries

Test №4-5

Goal: Repeat Test №2 and Test №3 to validate previous results.

Results: The results remained the same, despite Test №4 running with 5 instances and Test №5 with 6 instances, confirmed no negative impact from disabling the distinctInstance placement strategy.

Service CPU Utilization

Here we can see Test №4 with CPU=0 and that mod-nginx-okapi and okapi modules used 10% Instances CPU power.

Here we can see Test №5 with CPU=VALUE  and that mod-rtac module used 131% CPU power.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

 

Kafka metrics

 

OpenSearch Data Nodes metrics

DB CPU Utilization

DB CPU was 85%

DB Connections

Max number of DB connections was 1600.

DB load

Top SQL-queries

 

Test №6-7

Goal: Disable New Relic and repeat Test №4 and Test №5 to observe any effects.

Results: The results remained the same compared to all previous tests, showing no impact from using or not using New Relic.

Service CPU Utilization

Here we can see Test №6 with CPU=0 and that mod-nginx-okapi and okapi modules used 12% Instances CPU power.

Here we can see Test №7 with CPU=VALUE  and that mod-rtac module used 101% CPU power.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

 

Kafka metrics

OpenSearch Data Nodes metrics

DB CPU Utilization

DB CPU was 85% maximum.

DB Connections

Max number of DB connections was 1050.

DB load

Top SQL-queries

Appendix

Infrastructure

PTF - QCP1 environment configuration (was changed during testing)

  • 5-6 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

    Name

    Memory GIB

    vCPUs

    db.r6g.xlarge

    32 GB

    4 vCPUs

  • Open Search ptf-test 

    • Data nodes

      • Instance type - r6g.2xlarge.search

      • Number of nodes - 4

      • Version: OpenSearch_2_7_R20240502

    • Dedicated master nodes

      • Instance type - r6g.large.search

      • Number of nodes - 3

  • MSK fse-tenant

    • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x 

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

 

Cluster Resources qcp1-pvt (Thu Feb 20 09:58:24 UTC 2025) Module Task Definition Revision Module Version Task Count Mem Hard Limit Mem Soft Limit CPU Units Xmx Metaspace Size Max Metaspace Size mod-remote-storage 13 5mod-remote-storage:3.2.0 2 4920 4472 0 3960 512 512 mod-finance-storage 13 5mod-finance-storage:8.6.1 2 1024 896 0 700 88 128 mod-sudoc 5 5mod-sudoc:1.0 2 1024 896 0 768 88 128 mod-ebsconet 13 5mod-ebsconet:2.2.0 2 1248 1024 0 700 128 256 edge-sip2 11 5edge-sip2:3.2.7 2 1024 896 0 768 88 128 mod-tags 13 5mod-tags:2.2.0 2 1024 896 0 768 88 128 edge-courses 13 5edge-courses:1.4.5 2 1024 896 0 768 88 128 mod-authtoken 18 5mod-authtoken:2.15.2 2 1440 1152 0 922 88 128 mod-inventory-update 13 5mod-inventory-update:3.3.1 2 1024 896 0 768 88 128 mod-notify 13 5mod-notify:3.2.0 2 1024 896 0 768 88 128 mod-configuration 13 5mod-configuration:5.10.0 2 1024 896 0 768 88 128 edge-caiasoft 13 5edge-caiasoft:2.2.5 2 1024 896 0 768 88 128 mod-login-saml 13 5mod-login-saml:2.8.4 2 1024 896 0 768 88 128 mod-licenses 13 5mod-licenses:6.0.5 2 2480 2312 0 1792 384 512 mod-gobi 13 5mod-gobi:2.8.1 2 1024 896 0 700 88 128 mod-graphql 15 5mod-graphql:1.12.2 2 1024 896 0 768 88 128 mod-erm-usage 13 5mod-erm-usage:4.8.0 2 2800 2550 0 1800 384 512 mod-batch-print 14 5mod-batch-print:1.2.0 2 1024 896 0 768 88 128 mod-copycat 13 5mod-copycat:1.6.0 2 1024 512 0 768 88 128 mod-entities-links 14 5mod-entities-links:3.0.2 2 2592 2480 0 1440 0 1024 pub-edge 8 5pub-edge:2023.06.14 2 1024 896 0 768 0 0 mod-orders 14 5mod-orders:12.8.10 2 2048 1740 0 1024 384 512 edge-patron 14 5edge-patron:5.1.2 2 1024 896 0 768 88 128 edge-ncip 13 5edge-ncip:1.10.1 2 1024 896 0 768 88 128 edge-inn-reach 9 5edge-inn-reach:3.1.1-SNAPSHOT.45 2 1024 896 0 768 88 128 mod-users-bl 13 5mod-users-bl:7.7.5 2 1440 1152 0 922 88 128 mod-oa 11 5mod-oa:2.1.0-SNAPSHOT.62 2 1024 896 0 768 88 128 mod-inventory-storage 17 5mod-inventory-storage:27.1.5 4 4096 3690 0 3076 384 512 mod-invoice 15 5mod-invoice:5.8.3 2 1440 1152 0 922 88 128 mod-user-import 13 5mod-user-import:3.8.1 2 1024 896 0 768 88 128 mod-sender 14 5mod-sender:1.12.0 2 1024 896 0 768 88 128 mod-data-export-worker 13 5mod-data-export-worker:3.2.4 2 3072 2800 0 2048 384 512 mod-circulation-storage 15 5mod-circulation-storage:17.2.6 4 2880 2592 0 1814 384 512 mod-calendar 13 5mod-calendar:3.1.0 2 2048 1536 0 768 88 128 mod-event-config 13 5mod-event-config:2.7.1 2 1024 896 0 768 88 128 mod-courses 13 5mod-courses:1.4.11 2 1024 896 0 768 88 128 mod-circulation-item 13 5mod-circulation-item:1.0.0 2 1024 896 0 0 0 0 mod-email 13 5mod-email:1.17.0 2 2800 2550 0 1800 384 512 mod-pubsub 13 5mod-pubsub:2.13.1 2 1536 1440 0 922 384 512 mod-circulation 19 5mod-circulation:24.2.6 4 2880 2592 0 1814 384 512 mod-di-converter-storage 13 5mod-di-converter-storage:2.2.3 2 1024 896 0 768 88 128 edge-rtac 13 5edge-rtac:2.7.3 2 1024 896 0 768 88 128 edge-orders 13 5edge-orders:3.0.3 2 1024 896 0 768 88 128 mod-users 14 5mod-users:19.3.2 2 1024 896 0 768 88 128 mod-template-engine 13 5mod-template-engine:1.20.0 2 1024 896 0 768 88 128 mod-audit 13 5mod-audit:2.9.0 2 1024 896 0 768 88 128 mod-source-record-manager 15 5mod-source-record-manager:3.8.7 2 5600 5000 0 3500 384 512 mod-quick-marc 15 5mod-quick-marc:5.1.1 1 2288 2176 0 1664 384 512 nginx-okapi 8 5nginx-okapi:2023.06.14 2 1024 896 0 0 0 0 mod-feesfines 14 5mod-feesfines:19.1.0 2 1024 896 0 768 88 128 edge-users 3 5edge-users:1.5.0 2 1024 896 0 768 88 128 mod-dcb 12 5mod-dcb:1.1.0 2 1024 896 0 768 88 128 mod-service-interaction 13 5mod-service-interaction:4.0.2 2 2048 1844 0 1290 384 512 mod-patron 15 5mod-patron:6.1.0 2 1024 896 0 768 88 128 edge-connexion 13 5edge-connexion:1.3.1 2 1024 896 0 768 88 128 mod-data-export-spring 13 5mod-data-export-spring:3.2.2 1 2048 1844 0 1536 384 512 mod-organizations-storage 13 5mod-organizations-storage:4.7.0 2 1024 896 0 700 88 128 mod-login 14 5mod-login:7.11.2 2 1440 1298 0 768 384 512 edge-erm 3 5edge-erm:1.2.1 2 1024 896 0 768 88 128 mod-ncip 13 5mod-ncip:1.14.5 2 1024 896 0 768 88 128 mod-agreements 13 5mod-agreements:7.0.10 2 1592 1488 0 0 0 0 mod-organizations 13 5mod-organizations:1.9.2 2 1024 896 0 700 88 128 mod-consortia 9 5mod-consortia:1.1.0 2 5136 4776 0 4416 512 1024 mod-serials-management 13 5mod-serials-management:1.0.4 2 2480 2312 0 1792 384 512 mod-settings 13 5mod-settings:1.0.3 2 1024 896 0 768 88 128 mod-data-import 19 5mod-data-import:3.1.1 1 2048 1844 0 1292 384 512 edge-dematic 13 5edge-dematic:2.2.5 1 1024 896 0 768 88 128 mod-search 13 5mod-search:3.2.7 2 2592 2480 0 1440 512 1024 mod-inn-reach 9 5mod-inn-reach:3.2.0-SNAPSHOT.86 2 3600 3240 0 2880 512 1024 edge-inventory 3 5edge-inventory:1.6.1 2 1024 896 0 768 88 128 mod-orders-storage 14 5mod-orders-storage:13.7.4 2 1024 896 0 700 88 128 mod-erm-usage-harvester 13 5mod-erm-usage-harvester:4.5.0 2 1024 896 0 768 88 128 mod-password-validator 13 5mod-password-validator:3.2.1 2 1440 1298 0 768 384 512 mod-bulk-operations 13 5mod-bulk-operations:2.0.2 2 3072 2600 0 1536 384 512 mod-fqm-manager 13 5mod-fqm-manager:2.0.5 2 1024 896 0 768 88 128 edge-dcb 11 5edge-dcb:1.1.0 2 1024 896 0 768 88 128 mod-finance 13 5mod-finance:4.9.1 2 1024 896 0 700 88 128 mod-lists 13 5mod-lists:2.0.6 2 1024 896 0 768 88 128 mod-permissions 23 5mod-permissions:6.5.0 2 1684 1544 0 1024 384 512 mod-marc-migrations 10 5mod-marc-migrations:1.0.0-SNAPSHOT.5 2 1024 896 0 768 88 128 edge-ea-data-export 3 5edge-ea-data-export:4.2.0 2 1024 896 0 768 88 128 edge-oai-pmh 13 5edge-oai-pmh:2.9.2 2 1512 1360 0 1440 384 512 mod-rtac 15 5mod-rtac:3.6.1 2 1024 896 0 768 88 128 mod-task-list 3 5mod-task-list:1.9.4 2 1024 896 0 768 88 128 mod-source-record-storage 28 5mod-source-record-storage:5.8.10 2 5600 5000 0 3500 384 512 mod-inventory 19 5mod-inventory:20.2.11 4 2880 2592 0 1814 384 512 mod-patron-blocks 13 5mod-patron-blocks:1.10.0 2 1024 896 0 768 88 128 edge-fqm 13 5edge-fqm:2.0.3 2 1024 896 0 768 88 128 nginx-edge 8 5nginx-edge:2023.06.14 2 1024 896 0 0 0 0 okapi-b 11 5okapi:5.3.0 3 1684 1440 0 922 384 512 mod-invoice-storage 13 5mod-invoice-storage:5.8.2 2 1872 1536 0 1024 384 512 mod-data-export 28 5mod-data-export:5.0.4 1 2592 2480 0 1440 88 1024 mod-oai-pmh 13 5mod-oai-pmh:3.13.2 2 4096 3690 0 3076 384 512 mod-kb-ebsco-java 13 5mod-kb-ebsco-java:4.0.0 2 1024 896 0 768 88 128 mod-notes 13 5mod-notes:5.2.0 2 1024 896 0 952 384 512 pub-okapi 8 5pub-okapi:2023.06.14 2 1024 896 0 768 0 0 mod-eusage-reports 11 5mod-eusage-reports:2.1.1 2 1024 896 0 768 88 128

 

Methodology/Approach

CICO Tests scenarios were started for 100 users and concurrently RTAC for 10 users by JMeter script from load generator. 

QCP1 configuration

and steps to configure ECS infrastructure:

  • Instance type r7g.2xlarge

  • When New Relic/Open Telementry enabled OTEL value in mod-inventory's task definition set to true (OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_ENABLED=true)

  • mod-oa-b service was turn off for all tests

  • during testing CPU value was applied for list of modules:

    • mod-inventory: 3072

    • mod-inventory-storage: 2048

    • mod-circulation: 1536

    • mod-circulation-storage: 1536

    • mod-feesfines: 256

    • mod-orders: 1024

    • mod-orders-storage: 512

    • mod-login: 1024

    • mod-source-record-storage: 2048

    • mod-rtac: 128

    • mod-patron: 128

 

Test 1:
The QCP1 environment was configured with 6 Instances, New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned ON, CPU values set for list of modules, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 2:
The QCP1 environment was configured with with 6 Instances, New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned OFF, CPU values set for list of modules, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 3:
The QCP1 environment was configured with with 5 Instances, New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned OFF, CPU values set 0 for all services, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 4:
The QCP1 environment was configured with with 5 Instances, New Relic/OpenTelemetry enabled, OTEL value for mod-inventory and mod-circulation set TRUE, distinctInstance placement strategy turned OFF, CPU values set 0 for all services, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 5:
The QCP1 environment was configured with with 6 Instances, New Relic/OpenTelemetry enabled, OTEL value for mod-inventory and mod-circulation set TRUE, distinctInstance placement strategy turned OFF, CPU values set for list of modules, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 6:
The QCP1 environment was configured with with 5 Instances, New Relic/OpenTelemetry disabled, distinctInstance placement strategy turned OFF, CPU values set 0, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 7:
The QCP1 environment was configured with with 6 Instances, New Relic/OpenTelemetry disabled, distinctInstance placement strategy turned OFF, CPU values set for list of modules, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test artifacts: