...

• The performance tests conducted in this report indicate that neither the distinctInstance placement strategy nor the use of New Relic/OpenTelemetry had a significant impact on system performance. Across all tests, performance variations remained within a 5% margin. The slight difference observed in Test №3 was attributed to a lower instance count rather than the placement strategy itself. Further tests (№4 and №5) confirmed these findings, showing consistent results regardless of the number of instances used. Additionally, disabling New Relic did not affect performance, suggesting that monitoring overhead was negligible. Overall, the experiments demonstrate that the tested configurations do not introduce meaningful performance differences.
• It appears that increasing the number of virtual users in our test led to contention in the database, specifically due to a high volume of concurrent updates on the same row in the auth_attempts table by mod-login. The query:

  UPDATE fs09000000_mod_login.auth_attempts SET jsonb = $1::jsonb WHERE id='9883ca16-ef27-41f7-81d7-6693b79cddad'
  suggests that multiple sessions are attempting to modify the same record simultaneously, leading to row-level locking. As a result, transactions are waiting in the background for the lock to be released, potentially causing performance degradation or request timeouts.

Test Runs

Test Results

This table contains response time for Check In\Check Out and RTAC tests. 

...

Here we can see Test №1 with CPU=VALUE  and that mod-rtac module used 112% CPU power.

...

Here we can see Test №2 with CPU=VALUE  and that mod-rtac module used 109% CPU power.

...

Here we can see Test №3 with CPU=0 and that mod-nginx-okapi and okapi modules used 12% Instances CPU power.

...

Here we can see that all modules show a stable trend.

Kafka metrics

OpenSearch Data Nodes metrics

...

Here we can see Test №4 with CPU=0 and that mod-nginx-okapi and okapi modules used 10% Instances CPU power.

...

Here we can see Test №5 with CPU=VALUE  and that mod-rtac module used 131% CPU power.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

OpenSearch Data Nodes metrics

...

Here we can see Test №6 with CPU=0 and that mod-nginx-okapi and okapi modules used 12% Instances CPU power.

...

Here we can see Test №7 with CPU=VALUE  and that mod-rtac module used 101% CPU power.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics

OpenSearch Data Nodes metrics

...

PTF - QCP1 environment configuration (was changed during testing)

• 5-6 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  
  

  Name	Memory GIB	vCPUs
  db.r6g.xlarge	32 GB	4 vCPUs

  
  

• Open Search ptf-test 
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Version: OpenSearch_2_7_R20240502
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3
• MSK fse-tenant
  
  • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

  • Apache Kafka version 3.7.x 

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

...

QCP1 configuration

• Instance type r7g.2xlarge

• When New Relic/Open Telementry enabled OTEL value in mod-inventory's task definition set to true (OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_ENABLED=true)

• mod-oa-b service was turn off for all tests
• during testing CPU value was applied for list of modules:
  • mod-inventory: 3072
  • mod-inventory-storage: 2048
  • mod-circulation: 1536
  • mod-circulation-storage: 1536
  • mod-feesfines: 256
  • mod-orders: 1024
  • mod-orders-storage: 512
  • mod-login: 1024
  • mod-source-record-storage: 2048
  • mod-rtac: 128
  • mod-patron: 128

...