Overview

This document contains the results of testing Sharing local instances(SLI) for MARC Source records. PERF-755 - Getting issue details... STATUS
After improvement, the fixed version  mod-inventory:20.1.7-SNAPSHOT.487 was deployed on the Pcon cluster. MODINV-950 - Getting issue details... STATUS

Summary

• Duration on all of the tenants cs00000int_0001-cs00000int_0004 for 1 SLI process is about the same and the average value is about 2 seconds.  For 2 parallel SLIs duration is about 2.2 Seconds, for 3 parallel SLIs 2.15 seconds and for 4 parallel SLIs is 2.1.
• No memory leak is suspected for SLI modules. Memory consumption was quite low in comparison to the "before-test" state.
• Maximal CPU utilization was about 17% on mod-inventory and mod-quick-marc. 
• RDS CPU utilization were from 15% for 1VU and up to 20% for 4VU;

Recommendations and Jiras

• Test SLI in parallel with other workflows

Test results

Test parameters inventory.sharing.di.status.poll.interval.seconds= 1 and inventory.sharing.di.status.poll.number = 5(default)
Test 1. 1 Virtual user working sequentially on each of the 4 tenants.

Test 2. 2 Virtual users working parallel on 2 tenants.

Test 3. 3 Virtual users working parallel on 3 tenants

Test 4. 4 Virtual users working parallel on 4 tenants

Additional tests were performed to check the duration of SLI with the next parameters inventory.sharing.di.status.poll.interval.seconds= 2 and inventory.sharing.di.status.poll.number = 5(default)

Test 1.a 1 Virtual user working sequentially on each of the 4 tenants.

Test 2.a 2 Virtual users working parallel on 2 tenants.

Test 3.a  3 Virtual users working parallel on 2 tenants.

Resource Utilization Test 1,2,3,4. 

Below are the resource utilization graphs for all tests.

Memory Utilization

Memory usage was stable over 4 tests no memory leak is suspected for all modules, on the graph there are 10 most memory-consuming services.

Service CPU Utilization 

CPU utilization increased only during the SLI process and all modules came back to default numbers after all SLI was finished.

RDS CPU Utilization 

For 1VU avarage RDS CPU Utilization was asbout 14% for all 4 tenants. 2VU ~ 15%;  3VU~17% and 4VU ~20%.

RDS Database Connections

The average number of DB connections before the test was 450. SLI for 1VU~510; SLI for 2VU~520; SLI for 3VU~530; SLI for 4VU~540; 

Average active sessions (AAS)

Database load sliced by SQL

Errors

Failed request response.  All of the failed requests have the same errors.
{"errors":[{"message":"ERROR: duplicate key value violates unique constraint \"uq_instance_id_source_tenant_id_target_tenant_id\"\n  Detail: Key (instance_id, source_tenant_id, target_tenant_id)=(cf2a6947-e1bb-4e8f-ad43-f3ecd600b8f4, cs00000int_0004, cs00000int) already exists.","type":"-1","code":"VALIDATION_ERROR"}]}

Appendix

Environment: PCON

Record parameters, on each of the Тenants

• Tenant cs00000int_0001: Number of shared instances  1695139 and not shared 606035. Source MARC = 2185072 and FOLIO = 116102;
• Tenant cs00000int_0002: Number of shared instances  1695139 and not shared 1009666. Source MARC = 2559671 and FOLIO= 145134; 
• Tenant cs00000int_0003: Number of shared instances  1695139 and not shared 800515. Source MARC = 2380417 and FOLIO= 115237; 
• Tenant cs00000int_0004: Number of shared instances  1695139 and not shared 787757. Source MARC = 2367659 and FOLIO= 115237; 

PTF -environment ncp5 

• 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
• 1 database instance, writer Instance class db.r6g.xlarge vCPU 4 RAM 32 GB

Infrastructure

Mod-inventory:20.1.7-SNAPSHOT is a release of mod-inventory:20.1.6 + changes to add two configuration options. Parameters could be changes from task definition 

"name": "JAVA_OPTS", -Dinventory.sharing.di.status.poll.interval.seconds=2

Methodology

Test 1. The shared local instances process was started from the Jmeter script for Tenant cs00000int_0001, cs00000int_0002, cs00000int_0003, and cs00000int_0004 one after another. 

Test 2. The shared local instances process was started from the Jmeter script in parallel for Tenant cs00000int_0001 and cs00000int_0002. 

Test 3. The shared local instances process was started from the Jmeter script in parallel for Tenant cs00000int_0001,  cs00000int_0002, and cs00000int_0003. 

Test 4. The shared local instances process was started from the Jmeter script in parallel for Tenant cs00000int_0001,  cs00000int_0002, cs00000int_0003, and cs00000int_0004.