Overview

This document contains the results of testing Sharing local instances(SLI) for MARC Source records.

Summary

• Duration on three tenants cs00000int_0001-cs00000int_0003 for 1 SLI process is about the same and the average value is about 16 seconds.  Duration on the last tenant during test execution cs00000int_0004 is about 6 seconds. For 2 parallel SLIs duration is about 16 seconds on first cs00000int_0001 tenant and 6,5 seconds on second tenant cs00000int_0002. For 3 parallel SLIs and for 4 parallel SLIs is about 16 seconds for all except the last tenant - 6 seconds.
• Duration increased significantly if to compare with approximate 2 seconds in Poppy release.
• After adding parameter Dinventory.sharing.di.status.poll.interval.seconds=2 the duration decreased from 6 seconds to 2,9 seconds for SLI.
• No memory leak is suspected for SLI modules. Memory consumption grew for mod-inventory to 56% during 3-rd test and didn't change.
• Maximum CPU utilization was about 15% on mod-inventory and mod-consortia-b 1VU-23%, 2VU-25%, 3VU-28%, 4VU-35%.
• RDS CPU utilization were from 8% for 1VU and up to 10% for 4VU. That is twice less than in Poppy.
• All SLI finished successfully(without erros). Error rate = 0

Recommendations and Jiras

• It's important to add parameters to task definition to reduce local instance sharing time

  • "name": "JAVA_OPTS",
    "value": " -Dinventory.sharing.di.status.poll.interval.seconds=2"

Test results

Test 1. Virtual user working sequentially on each of the 4 tenants.

Test 2. Virtual users working parallel on 2 tenants.

Test 3. Virtual users working parallel on 3 tenants

Test 4. Virtual users working parallel on 4 tenants

Results after mod-inventory module task definition update ("value": " -Dinventory.sharing.di.status.poll.interval.seconds=2")

Test 1. Virtual user working sequentially on each of the 4 tenants.

Test 2. Virtual users working parallel on 2 tenants.

Test 3. Virtual users working parallel on 3 tenants

Test 4. Virtual users working parallel on 4 tenants

Resource Utilization Test 1,2,3,4. 

Below are the resource utilization graphs for all tests.

Service CPU Utilization 

CPU utilization increased only during the SLI process and all modules came back to default numbers after all SLI was finished.

mod-consortia-b

Test 1 - 25%, test 2 - 27%, test 3 - 30%, test 4 - 35%

Memory Utilization

Memory usage was stable over 4 tests no memory leak is suspected for all modules, on the graph there are 10 most memory-consuming services. Mod-consortia - 74%, mod-inventory - 56%.

RDS CPU Utilization 

For 1VU average RDS CPU Utilization was about 7% for all 4 tenants. 2VU ~ 8%;  3VU~9% and 4VU ~10%. 

RDS Database Connections

The average number of DB connections before the test was about 500.  During tests the connections spiked to 620.

Average active sessions (AAS)

Database load sliced by SQL

Errors

"error": "Sharing instance with InstanceId=[UUID] to the target tenant cs00000int. Error: Failed sending record data."

Fixed after deploy of mod-di-converter-storage:2.2.2 

Appendix

Record parameters, on each of the Тenants

 PTF -environment: QCON

• 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
• 1 instance of db.r6.xlarge database instance: Writer instance
• MSK - tenant
  
  • 4 kafka.m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• OpenSearch 2.7 domain: ptf-test 
  • Dedicated master nodes enabled 
    • Instance type r6g.large.search
    • Number of nodes 3

  • Data nodes

    • Instance type r6g.2xlarge.search
    • Number of nodes 4
• Topics: 
  • { "name": "KAFKA_PRODUCER_TENANT_COLLECTION", "value": "ALL" }

Infrastructure


Additional Files

• source files for SLI CPU and Memory dashboard creation 
  

Methodology

jMeter request to trigger instance sharing:

• CSI_POST /consortia/{consortiumId}/sharing/instances

  BODY: {"sourceTenantId":"${sourceTenantId}","instanceIdentifier":"${list_id1}","targetTenantId":"${targetTenantId}","id":"${__UUID()}"}
  
  

Test 1. The shared local instances process was started from the Jmeter script for Tenant cs00000int_0001, cs00000int_0002, cs00000int_0003, and cs00000int_0004 one after another. 

Test 2. The shared local instances process was started from the Jmeter script in parallel for Tenant cs00000int_0001 and cs00000int_0002. 

Test 3. The shared local instances process was started from the Jmeter script in parallel for Tenant cs00000int_0001,  cs00000int_0002, and cs00000int_0003. 

Test 4. The shared local instances process was started from the Jmeter script in parallel for Tenant cs00000int_0001,  cs00000int_0002, cs00000int_0003, and cs00000int_0004.