Overview

In this report, PTF investigates the impact of setting CPU=0 and disabling the distinctInstance task placement strategy on FOLIO’s performance, with and without New Relic/OpenTelemetry enabled. Previous observations suggested that this configuration negatively affects system performance. The goal of these tests was to reproduce the issue and evaluate whether performance could be improved by adjusting CPU allocations and enabling distinctInstance placement. Through a series of experiments, we analyzed how these settings influence system behavior, ensuring that CPU allocation and task distribution strategies are optimized for stability and efficiency.

PERF-1071 - Getting issue details... STATUS

Summary

The performance tests conducted in this report indicate that neither the distinctInstance placement strategy nor the use of New Relic/OpenTelemetry had a significant impact on system performance. Across all tests, performance variations remained within a 5% margin. The slight difference observed in Test №3 was attributed to a lower instance count rather than the placement strategy itself. Further tests (№4 and №5) confirmed these findings, showing consistent results regardless of the number of instances used. Additionally, disabling New Relic did not affect performance, suggesting that monitoring overhead was negligible. Overall, the experiments demonstrate that the tested configurations do not introduce meaningful performance differences.

It appears that increasing the number of virtual users in our test led to contention in the database, specifically due to a high volume of concurrent updates on the same row in the auth_attempts table by mod-login. The query:

UPDATE fs09000000_mod_login.auth_attempts SET jsonb = $1::jsonb WHERE id='9883ca16-ef27-41f7-81d7-6693b79cddad'
suggests that multiple sessions are attempting to modify the same record simultaneously, leading to row-level locking. As a result, transactions are waiting in the background for the lock to be released, potentially causing performance degradation or request timeouts.

Test Runs

Test #	Description	Status
Test 1	New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned ON, CPU values set for list of modules	Completed
Test 2	New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned OFF, CPU values set for list of modules	Completed
Test 3	New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned OFF, CPU values set 0 for all services	Completed
Test 4	New Relic/OpenTelemetry enabled, OTEL value for mod-inventory and mod-circulation set TRUE, distinctInstance placement strategy turned OFF, CPU values set 0 for all services	Completed
Test 5	New Relic/OpenTelemetry enabled, OTEL value for mod-inventory and mod-circulation set TRUE, distinctInstance placement strategy turned OFF, CPU values set for list of modules	Completed
Test 6	New Relic/OpenTelemetry disabled, distinctInstance placement strategy turned OFF, CPU values set 0 for all services	Completed
Test 7	New Relic/OpenTelemetry disabled, distinctInstance placement strategy turned OFF, CPU values set for list of modules	Completed

Test Results

This table contains response time for Check In\Check Out and RTAC tests.

Requests	Test №1 Response time, ms	Test №2 Response time, ms	Test №3 Response time, ms	Test №4 Response time, ms	Test №5 Response time, ms	Test №6 Response time, ms	Test №7 Response time, ms
	Average	Average	Average	Average	Average	Average	Average
Check-In Controller	995	965	1057	886	922	929	917
Check-Out Controller	1518	1502	1637	1352	1392	1431	1447
RTAC	1174	1136	1258	1087	1069	1085	1091

Test №1-2-3

Test №1: Modules have CPU = 0, and distinctInstance placement strategy is OFF.
Goal: Establish baseline performance metrics for comparison with subsequent configurations.

Test №2: Modules are assigned specific CPU values, with distinctInstance placement strategy still OFF.
Goal: Evaluate whether performance improves compared to the baseline (Test №1).

Test №3: Modules are assigned specific CPU values, and distinctInstance placement strategy is ON.
Goal: Assess whether enabling distinctInstance further enhances performance over the previous tests.

Results: Performance remained nearly the same across all tests, with differences of less than 5%. Test №3 showed a slight variance due to having 5 instances instead of 6, as used in Tests №1 and №2, indicating no negative impact from disabling the distinctInstance placement strategy.

Service CPU Utilization

Here we can see Test №1 with CPU=VALUE and that mod-rtac module used 112% CPU power.

Here we can see Test №2 with CPU=VALUE and that mod-rtac module used 109% CPU power.

Here we can see Test №3 with CPU=0 and that mod-nginx-okapi and okapi modules used 12% Instances CPU power.

Service Memory Utilization

Here we can see that all modules show a stable trend.

Kafka metrics

OpenSearch Data Nodes metrics

DB CPU Utilization

DB CPU was 85% average.

DB Connections

Max number of DB connections was 950.

DB load

Top SQL-queries

Test №4-5

Goal: Repeat Test №2 and Test №3 to validate previous results.

Results: The results remained the same, despite Test №4 running with 5 instances and Test №5 with 6 instances, confirmed no negative impact from disabling the distinctInstance placement strategy.

Service CPU Utilization

Here we can see Test №4 with CPU=0 and that mod-nginx-okapi and okapi modules used 10% Instances CPU power.

Here we can see Test №5 with CPU=VALUE and that mod-rtac module used 131% CPU power.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

OpenSearch Data Nodes metrics

DB CPU Utilization

DB CPU was 85%

DB Connections

Max number of DB connections was 1600.

DB load

Top SQL-queries

Test №6-7

Goal: Disable New Relic and repeat Test №4 and Test №5 to observe any effects.

Results: The results remained the same compared to all previous tests, showing no impact from using or not using New Relic.

Service CPU Utilization

Here we can see Test №6 with CPU=0 and that mod-nginx-okapi and okapi modules used 12% Instances CPU power.

Here we can see Test №7 with CPU=VALUE and that mod-rtac module used 101% CPU power.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

OpenSearch Data Nodes metrics

DB CPU Utilization

DB CPU was 85% maximum.

DB Connections

Max number of DB connections was 1050.

DB load

Top SQL-queries

Appendix

Infrastructure

PTF - QCP1 environment configuration (was changed during testing)

5-6 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs
db.r6g.xlarge
32 GB 4 vCPUs
Open Search ptf-test
- Data nodes
  - Instance type - r6g.2xlarge.search
  - Number of nodes - 4
  - Version: OpenSearch_2_7_R20240502
- Dedicated master nodes
  - Instance type - r6g.large.search
  - Number of nodes - 3
MSK fse-tenant
- 2 brokers, kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

qcp1 modules memory and CPU parameters

Cluster Resources

qcp1-pvt (Thu Feb 20 09:58:24 UTC 2025)

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	13	5mod-remote-storage:3.2.0	2	4920	4472	3960	512	512
mod-finance-storage	13	5mod-finance-storage:8.6.1	2	1024	896	700	88	128
mod-sudoc	5	5mod-sudoc:1.0	2	1024	896	768	88	128
mod-ebsconet	13	5mod-ebsconet:2.2.0	2	1248	1024	700	128	256
edge-sip2	11	5edge-sip2:3.2.7	2	1024	896	768	88	128
mod-tags	13	5mod-tags:2.2.0	2	1024	896	768	88	128
edge-courses	13	5edge-courses:1.4.5	2	1024	896	768	88	128
mod-authtoken	18	5mod-authtoken:2.15.2	2	1440	1152	922	88	128
mod-inventory-update	13	5mod-inventory-update:3.3.1	2	1024	896	768	88	128
mod-notify	13	5mod-notify:3.2.0	2	1024	896	768	88	128
mod-configuration	13	5mod-configuration:5.10.0	2	1024	896	768	88	128
edge-caiasoft	13	5edge-caiasoft:2.2.5	2	1024	896	768	88	128
mod-login-saml	13	5mod-login-saml:2.8.4	2	1024	896	768	88	128
mod-licenses	13	5mod-licenses:6.0.5	2	2480	2312	1792	384	512
mod-gobi	13	5mod-gobi:2.8.1	2	1024	896	700	88	128
mod-graphql	15	5mod-graphql:1.12.2	2	1024	896	768	88	128
mod-erm-usage	13	5mod-erm-usage:4.8.0	2	2800	2550	1800	384	512
mod-batch-print	14	5mod-batch-print:1.2.0	2	1024	896	768	88	128
mod-copycat	13	5mod-copycat:1.6.0	2	1024	512	768	88	128
mod-entities-links	14	5mod-entities-links:3.0.2	2	2592	2480	1440	0	1024
pub-edge	8	5pub-edge:2023.06.14	2	1024	896	768	0	0
mod-orders	14	5mod-orders:12.8.10	2	2048	1740	1024	384	512
edge-patron	14	5edge-patron:5.1.2	2	1024	896	768	88	128
edge-ncip	13	5edge-ncip:1.10.1	2	1024	896	768	88	128
edge-inn-reach	9	5edge-inn-reach:3.1.1-SNAPSHOT.45	2	1024	896	768	88	128
mod-users-bl	13	5mod-users-bl:7.7.5	2	1440	1152	922	88	128
mod-oa	11	5mod-oa:2.1.0-SNAPSHOT.62	2	1024	896	768	88	128
mod-inventory-storage	17	5mod-inventory-storage:27.1.5	4	4096	3690	3076	384	512
mod-invoice	15	5mod-invoice:5.8.3	2	1440	1152	922	88	128
mod-user-import	13	5mod-user-import:3.8.1	2	1024	896	768	88	128
mod-sender	14	5mod-sender:1.12.0	2	1024	896	768	88	128
mod-data-export-worker	13	5mod-data-export-worker:3.2.4	2	3072	2800	2048	384	512
mod-circulation-storage	15	5mod-circulation-storage:17.2.6	4	2880	2592	1814	384	512
mod-calendar	13	5mod-calendar:3.1.0	2	2048	1536	768	88	128
mod-event-config	13	5mod-event-config:2.7.1	2	1024	896	768	88	128
mod-courses	13	5mod-courses:1.4.11	2	1024	896	768	88	128
mod-circulation-item	13	5mod-circulation-item:1.0.0	2	1024	896	0	0	0
mod-email	13	5mod-email:1.17.0	2	2800	2550	1800	384	512
mod-pubsub	13	5mod-pubsub:2.13.1	2	1536	1440	922	384	512
mod-circulation	19	5mod-circulation:24.2.6	4	2880	2592	1814	384	512
mod-di-converter-storage	13	5mod-di-converter-storage:2.2.3	2	1024	896	768	88	128
edge-rtac	13	5edge-rtac:2.7.3	2	1024	896	768	88	128
edge-orders	13	5edge-orders:3.0.3	2	1024	896	768	88	128
mod-users	14	5mod-users:19.3.2	2	1024	896	768	88	128
mod-template-engine	13	5mod-template-engine:1.20.0	2	1024	896	768	88	128
mod-audit	13	5mod-audit:2.9.0	2	1024	896	768	88	128
mod-source-record-manager	15	5mod-source-record-manager:3.8.7	2	5600	5000	3500	384	512
mod-quick-marc	15	5mod-quick-marc:5.1.1	1	2288	2176	1664	384	512
nginx-okapi	8	5nginx-okapi:2023.06.14	2	1024	896	0	0	0
mod-feesfines	14	5mod-feesfines:19.1.0	2	1024	896	768	88	128
edge-users	3	5edge-users:1.5.0	2	1024	896	768	88	128
mod-dcb	12	5mod-dcb:1.1.0	2	1024	896	768	88	128
mod-service-interaction	13	5mod-service-interaction:4.0.2	2	2048	1844	1290	384	512
mod-patron	15	5mod-patron:6.1.0	2	1024	896	768	88	128
edge-connexion	13	5edge-connexion:1.3.1	2	1024	896	768	88	128
mod-data-export-spring	13	5mod-data-export-spring:3.2.2	1	2048	1844	1536	384	512
mod-organizations-storage	13	5mod-organizations-storage:4.7.0	2	1024	896	700	88	128
mod-login	14	5mod-login:7.11.2	2	1440	1298	768	384	512
edge-erm	3	5edge-erm:1.2.1	2	1024	896	768	88	128
mod-ncip	13	5mod-ncip:1.14.5	2	1024	896	768	88	128
mod-agreements	13	5mod-agreements:7.0.10	2	1592	1488	0	0	0
mod-organizations	13	5mod-organizations:1.9.2	2	1024	896	700	88	128
mod-consortia	9	5mod-consortia:1.1.0	2	5136	4776	4416	512	1024
mod-serials-management	13	5mod-serials-management:1.0.4	2	2480	2312	1792	384	512
mod-settings	13	5mod-settings:1.0.3	2	1024	896	768	88	128
mod-data-import	19	5mod-data-import:3.1.1	1	2048	1844	1292	384	512
edge-dematic	13	5edge-dematic:2.2.5	1	1024	896	768	88	128
mod-search	13	5mod-search:3.2.7	2	2592	2480	1440	512	1024
mod-inn-reach	9	5mod-inn-reach:3.2.0-SNAPSHOT.86	2	3600	3240	2880	512	1024
edge-inventory	3	5edge-inventory:1.6.1	2	1024	896	768	88	128
mod-orders-storage	14	5mod-orders-storage:13.7.4	2	1024	896	700	88	128
mod-erm-usage-harvester	13	5mod-erm-usage-harvester:4.5.0	2	1024	896	768	88	128
mod-password-validator	13	5mod-password-validator:3.2.1	2	1440	1298	768	384	512
mod-bulk-operations	13	5mod-bulk-operations:2.0.2	2	3072	2600	1536	384	512
mod-fqm-manager	13	5mod-fqm-manager:2.0.5	2	1024	896	768	88	128
edge-dcb	11	5edge-dcb:1.1.0	2	1024	896	768	88	128
mod-finance	13	5mod-finance:4.9.1	2	1024	896	700	88	128
mod-lists	13	5mod-lists:2.0.6	2	1024	896	768	88	128
mod-permissions	23	5mod-permissions:6.5.0	2	1684	1544	1024	384	512
mod-marc-migrations	10	5mod-marc-migrations:1.0.0-SNAPSHOT.5	2	1024	896	768	88	128
edge-ea-data-export	3	5edge-ea-data-export:4.2.0	2	1024	896	768	88	128
edge-oai-pmh	13	5edge-oai-pmh:2.9.2	2	1512	1360	1440	384	512
mod-rtac	15	5mod-rtac:3.6.1	2	1024	896	768	88	128
mod-task-list	3	5mod-task-list:1.9.4	2	1024	896	768	88	128
mod-source-record-storage	28	5mod-source-record-storage:5.8.10	2	5600	5000	3500	384	512
mod-inventory	19	5mod-inventory:20.2.11	4	2880	2592	1814	384	512
mod-patron-blocks	13	5mod-patron-blocks:1.10.0	2	1024	896	768	88	128
edge-fqm	13	5edge-fqm:2.0.3	2	1024	896	768	88	128
nginx-edge	8	5nginx-edge:2023.06.14	2	1024	896	0	0	0
okapi-b	11	5okapi:5.3.0	3	1684	1440	922	384	512
mod-invoice-storage	13	5mod-invoice-storage:5.8.2	2	1872	1536	1024	384	512
mod-data-export	28	5mod-data-export:5.0.4	1	2592	2480	1440	88	1024
mod-oai-pmh	13	5mod-oai-pmh:3.13.2	2	4096	3690	3076	384	512
mod-kb-ebsco-java	13	5mod-kb-ebsco-java:4.0.0	2	1024	896	768	88	128
mod-notes	13	5mod-notes:5.2.0	2	1024	896	952	384	512
pub-okapi	8	5pub-okapi:2023.06.14	2	1024	896	768	0	0
mod-eusage-reports	11	5mod-eusage-reports:2.1.1	2	1024	896	768	88	128

Methodology/Approach

CICO Tests scenarios were started for 100 users and concurrently RTAC for 10 users by JMeter script from load generator.

QCP1 configuration and steps to configure ECS infrastructure:

Instance type r7g.2xlarge
When New Relic/Open Telementry enabled OTEL value in mod-inventory's task definition set to true (OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_ENABLED=true)
mod-oa-b service was turn off for all tests
during testing CPU value was applied for list of modules:
- mod-inventory: 3072
- mod-inventory-storage: 2048
- mod-circulation: 1536
- mod-circulation-storage: 1536
- mod-feesfines: 256
- mod-orders: 1024
- mod-orders-storage: 512
- mod-login: 1024
- mod-source-record-storage: 2048
- mod-rtac: 128
- mod-patron: 128

Test 1:
The QCP1 environment was configured with 6 Instances, New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned ON, CPU values set for list of modules, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 2:
The QCP1 environment was configured with with 6 Instances, New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned OFF, CPU values set for list of modules, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 3:
The QCP1 environment was configured with with 5 Instances, New Relic/OpenTelemetry enabled, OTEL value for mod-inventory set TRUE, distinctInstance placement strategy turned OFF, CPU values set 0 for all services, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 4:
The QCP1 environment was configured with with 5 Instances, New Relic/OpenTelemetry enabled, OTEL value for mod-inventory and mod-circulation set TRUE, distinctInstance placement strategy turned OFF, CPU values set 0 for all services, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 5:
The QCP1 environment was configured with with 6 Instances, New Relic/OpenTelemetry enabled, OTEL value for mod-inventory and mod-circulation set TRUE, distinctInstance placement strategy turned OFF, CPU values set for list of modules, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 6:
The QCP1 environment was configured with with 5 Instances, New Relic/OpenTelemetry disabled, distinctInstance placement strategy turned OFF, CPU values set 0, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test 7:
The QCP1 environment was configured with with 6 Instances, New Relic/OpenTelemetry disabled, distinctInstance placement strategy turned OFF, CPU values set for list of modules, 4 tasks assigned to mod-inventory, mod-inventory-storage, mod-circulation, and mod-circulation-storage, with Check-In/Check-Out tests running 100 users and RTAC tests running 10 users concurrently on a main tenant for 35 minutes.

Test artifacts:

Folio Development Teams

PTF - Performance testing of CPU=0 and tasks placement strategy for Services (QCP1)

Overview

Summary

Test Runs

Test Results

Test №1-2-3

Service CPU Utilization

Service Memory Utilization

Kafka metrics

OpenSearch Data Nodes metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Test №4-5

Service CPU Utilization

Service Memory Utilization

Kafka metrics

OpenSearch Data Nodes metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Test №6-7

Service CPU Utilization

Service Memory Utilization

Kafka metrics

OpenSearch Data Nodes metrics

DB CPU Utilization

DB Connections

DB load

Top SQL-queries

Appendix

Infrastructure

Cluster Resources

Methodology/Approach

Related content

Name	Memory GIB	vCPUs
db.r6g.xlarge	32 GB	4 vCPUs