Member tenants sharing local instances (Quesnelia)

Overview

This document contains the results of testing Sharing local instances(SLI) for MARC Source records. PERF-904 - Getting issue details... STATUS

Summary

  • Duration on three tenants cs00000int_0001-cs00000int_0003 for 1 SLI process is about the same and the average value is about 16 seconds.  Duration on the last tenant during test execution cs00000int_0004 is about 6 seconds. For 2 parallel SLIs duration is about 16 seconds on first cs00000int_0001 tenant and 6,5 seconds on second tenant cs00000int_0002. For 3 parallel SLIs and for 4 parallel SLIs is about 16 seconds for all except the last tenant - 6 seconds.
  • Duration increased significantly if to compare with approximate 2 seconds in Poppy release.
  • After adding parameter Dinventory.sharing.di.status.poll.interval.seconds=2 the duration decreased from 6 seconds to 2,9 seconds for SLI.
  • No memory leak is suspected for SLI modules. Memory consumption grew for mod-inventory to 56% during 3-rd test and didn't change.
  • Maximum CPU utilization was about 15% on mod-inventory and mod-consortia-b 1VU-23%, 2VU-25%, 3VU-28%, 4VU-35%.
  • RDS CPU utilization were from 8% for 1VU and up to 10% for 4VU. That is twice less than in Poppy.
  • All SLI finished successfully(without erros). Error rate = 0

Recommendations and Jiras

  • It's important to add parameters to task definition to reduce local instance sharing time
    • "name": "JAVA_OPTS",
      "value": " -Dinventory.sharing.di.status.poll.interval.seconds=2"

Test results

Test 1. Virtual user working sequentially on each of the 4 tenants.

Tenant 

TTL REQ, COUNT

Average, MS

THRGHPT, REQ/SEC

ERRORS, COUNT

MIN, MS

MEDIAN, MS

PCT95, MS

MAX, MS

cs00000int_0001

100159740.06015560159671636916482
cs00000int_0002100160040.06015594160161635116377
cs00000int_0003100155230.06015281155031576015852
cs00000int_000410059980.1605714597562856941

Test 2. Virtual users working parallel on 2 tenants.

Tenant 

TTL REQ, COUNT

Average, MS

THRGHPT, REQ/SEC

ERRORS, COUNT

MIN, MS

MEDIAN, MS

PCT95, MS

MAX, MS

cs00000int_0001

100159580.06015571159451631116488
cs00000int_000210064360.1505982641268028082

Test 3. Virtual users working parallel on 3 tenants

Tenant 

TTL REQ, COUNT

Average, MS

THRGHPT, REQ/SEC

ERRORS, COUNT

MIN, MS

MEDIAN, MS

PCT95, MS

MAX, MS

cs00000int_0001

100159840.06015554159841626916464
cs00000int_0002100159760.06015613159671632316457
cs00000int_000310059850.1605655595163087414

Test 4. Virtual users working parallel on 4 tenants

Tenant 

TTL REQ, COUNT

Average, MS

THRGHPT, REQ/SEC

ERRORS, COUNT

MIN, MS

MEDIAN, MS

PCT95, MS

MAX, MS

cs00000int_0001

100159930.06015630159941630216548
cs00000int_0002100159200.06015554158751631516405
cs00000int_0003100155150.06015281155071572915755
cs00000int_000410059560.1605656593062887321


Results after mod-inventory module task definition update ("value": " -Dinventory.sharing.di.status.poll.interval.seconds=2")

Test 1. Virtual user working sequentially on each of the 4 tenants.

Tenant 

TTL REQ, COUNT

Average, MS

THRGHPT, REQ/SEC

ERRORS, COUNT

MIN, MS

MEDIAN, MS

PCT95, MS

MAX, MS

cs00000int_0001

100159800.06015613159671633216457
cs00000int_0002100159510.06015583159771632116354
cs00000int_0003100155220.06015263155041575515978
cs00000int_000410028750.3302603286031303215

Test 2. Virtual users working parallel on 2 tenants.

Tenant 

TTL REQ, COUNT

Average, MS

THRGHPT, REQ/SEC

ERRORS, COUNT

MIN, MS

MEDIAN, MS

PCT95, MS

MAX, MS

cs00000int_0001

100159720.06015592159941629616405
cs00000int_000210033110.2801035344837394692

Test 3. Virtual users working parallel on 3 tenants

Tenant 

TTL REQ, COUNT

Average, MS

THRGHPT, REQ/SEC

ERRORS, COUNT

MIN, MS

MEDIAN, MS

PCT95, MS

MAX, MS

cs00000int_0001

100159860.06015643159781630516416
cs00000int_0002100159570.06015573159491626416472
cs00000int_000310028500.330257928143095.24225

Test 4. Virtual users working parallel on 4 tenants

Tenant 

TTL REQ, COUNT

Average, MS

THRGHPT, REQ/SEC

ERRORS, COUNT

MIN, MS

MEDIAN, MS

PCT95, MS

MAX, MS

cs00000int_0001

100159650.06015580159641634616473
cs00000int_0002100159590.06015617159621628616434
cs00000int_0003100155290.06015263155381570415889
cs00000int_000410028740.1602601285231014433

Resource Utilization Test 1,2,3,4. 

Below are the resource utilization graphs for all tests.

Service CPU Utilization 

CPU utilization increased only during the SLI process and all modules came back to default numbers after all SLI was finished.

mod-consortia-b

Test 1 - 25%, test 2 - 27%, test 3 - 30%, test 4 - 35%

 CPU utilization table over the tests


CPUTest 1Test 2Test 3Test 4
mod-consortia-b23.6425.4128.3335.09
mod-inventory-b13.9514.713.7113.9
mod-pubsub-b6.626.656.546.62
mod-quick-marc-b66.035.537.02
mod-di-converter-storage-b2.944.774.785.17
mod-source-record-manager-b2.42.682.853.01
mod-tags-b2.252.232.282.27
mod-organizations-b2.152.092.092.17
mod-search-b1.61.691.811.85
mod-source-record-storage-b1.571.631.641.72
okapi-b1.532.473.193.59
nginx-okapi1.082.343.153.81
mod-inventory-storage-b0.460.851.021.19
mod-circulation-b0.330.370.350.34


Memory Utilization

Memory usage was stable over 4 tests no memory leak is suspected for all modules, on the graph there are 10 most memory-consuming services. Mod-consortia - 74%, mod-inventory - 56%.

RDS CPU Utilization 

For 1VU average RDS CPU Utilization was about 7% for all 4 tenants. 2VU ~ 8%;  3VU~9% and 4VU ~10%. 


RDS Database Connections

The average number of DB connections before the test was about 500.  During tests the connections spiked to 620.

Average active sessions (AAS)

Database load sliced by SQL

Errors

"error": "Sharing instance with InstanceId=[UUID] to the target tenant cs00000int. Error: Failed sending record data."

Fixed after deploy of mod-di-converter-storage:2.2.2  MODDICONV-379 - Getting issue details... STATUS

Appendix

Record parameters, on each of the Тenants





QCONInstancesSource:Records:
cs00000int1,540,307MARC1436417


FOLIO103890




cs00000int_00013,147,248MARC3042890


FOLIO104358




cs00000int_00022,306,678MARC2180113


FOLIO126565




cs00000int_00031,540,307MARC1436417


FOLIO103890




cs00000int_00041,540,307MARC1436417


FOLIO103890




cs00000int_00051,540,307MARC1436417


FOLIO103890


 PTF -environment: QCON

  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 1 instance of db.r6.xlarge database instance: Writer instance
  • MSK - tenant
    • 4 kafka.m5.2xlarge brokers in 2 zones
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3
  • OpenSearch 2.7 domain: ptf-test 
    • Dedicated master nodes enabled 
      • Instance type r6g.large.search
      • Number of nodes 3
    • Data nodes

      • Instance type r6g.2xlarge.search
      • Number of nodes 4
  • Topics: 
    • { "name": "KAFKA_PRODUCER_TENANT_COLLECTION", "value": "ALL" }

Infrastructure

 Share local instance modules
ModuleTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSize
qcon-pvt








Fri May 10 18:49:16 UTC 2024








mod-consortia1mod-consortia:1.1.023072204812820485121024
mod-organizations1mod-organizations:1.9.02102489612870088128
mod-tags1mod-tags:2.2.02102489612876888128
mod-inventory-storage1mod-inventory-storage:27.1.024096369020483076384512
mod-source-record-storage1mod-source-record-storage:5.8.025600500020483500384512
mod-inventory1mod-inventory:20.2.022880259210241814384512
mod-circulation2mod-circulation:24.2.122880259215361814384512
mod-pubsub1mod-pubsub:2.13.02153614401024922384512
mod-di-converter-storage2mod-di-converter-storage:2.2.22102489612876888128
mod-source-record-manager1mod-source-record-manager:3.8.025600500020483500384512
mod-quick-marc1mod-quick-marc:5.1.01228821761281664384512
nginx-okapi1nginx-okapi:2023.06.1421024896128000
okapi-b1okapi:5.3.03168414401024922384512


 QCON all modules
ModuleTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSize
qcon-pvt








Fri May 10 18:49:16 UTC 2024








mod-remote-storage1mod-remote-storage:3.2.024920447210243960512512
mod-finance-storage1mod-finance-storage:8.6.021024896102470088128
mod-ncip1mod-ncip:1.14.42102489612876888128
mod-agreements1mod-agreements:7.0.0015921488128000
mod-ebsconet1mod-ebsconet:2.2.0212481024128700128256
mod-consortia1mod-consortia:1.1.023072204812820485121024
mod-organizations1mod-organizations:1.9.02102489612870088128
mod-serials-management1mod-serials-management:1.0.00248023121281792384512
mod-settings1mod-settings:1.0.32102489620076888128
mod-search4mod-search:3.2.0251844960204814405121024
edge-dematic1edge-dematic:2.2.01102489612876888128
mod-data-import1mod-data-import:3.1.01204818442561292384512
mod-tags1mod-tags:2.2.02102489612876888128
mod-authtoken3 non-expired tokenmod-authtoken:2.15.121440115251292288128
edge-courses1edge-courses:1.4.02102489612876888128
mod-inventory-update1mod-inventory-update:3.3.02102489612876888128
mod-notify1mod-notify:3.2.02102489612876888128
mod-configuration1mod-configuration:5.10.02102489612876888128
mod-orders-storage1mod-orders-storage:13.7.02102489651270088128
edge-caiasoft1edge-caiasoft:2.2.02102489612876888128
mod-login-saml1mod-login-saml:2.8.02102489612876888128
mod-erm-usage-harvester1mod-erm-usage-harvester:4.5.02102489612876888128
mod-password-validator1mod-password-validator:3.2.0214401298128768384512
mod-licenses1mod-licenses:6.0.02248023121281792384512
mod-gobi1mod-gobi:2.8.02102489612870088128
mod-bulk-operations1mod-bulk-operations:2.0.023072260010241536384512
mod-fqm-manager1mod-fqm-manager:2.0.12300026001282048384512
edge-dcb1edge-dcb:1.1.02102489612876888128
mod-graphql1mod-graphql:1.12.12102489612876888128
mod-finance1mod-finance:4.9.02102489612870088128
mod-erm-usage1mod-erm-usage:4.7.02102489612876888128
mod-batch-print1mod-batch-print:1.1.02102489612876888128
mod-copycat1mod-copycat:1.6.02102451212876888128
mod-lists1mod-lists:2.0.02300026001282048384512
mod-entities-links1mod-entities-links:3.0.0225922480400144001024
mod-permissions2mod-permissions:6.5.02168415445121024384512
pub-edge1pub-edge:2023.06.142102489612876800
mod-orders1mod-orders:12.8.022048144010241024384512
edge-patron1edge-patron:5.1.02102489625676888128
edge-ncip1edge-ncip:1.9.22102489612876888128
mod-users-bl1mod-users-bl:7.7.021440115251292288128
mod-invoice1mod-invoice:5.8.021440115251292288128
mod-inventory-storage1mod-inventory-storage:27.1.024096369020483076384512
mod-user-import1mod-user-import:3.8.02102489612876888128
mod-sender1mod-sender:1.12.02102489612876888128
edge-oai-pmh1edge-oai-pmh:2.9.021512136010241440384512
mod-data-export-worker1mod-data-export-worker:3.2.123072204810242048384512
mod-rtac1mod-rtac:3.6.02102489612876888128
mod-circulation-storage1mod-circulation-storage:17.2.022880259215361814384512
mod-source-record-storage1mod-source-record-storage:5.8.025600500020483500384512
mod-calendar1mod-calendar:3.1.02102489612876888128
mod-event-config1mod-event-config:2.7.02102489612876888128
mod-courses1mod-courses:1.4.102102489612876888128
mod-circulation-item1mod-circulation-item:1.0.021024896128000
mod-inventory1mod-inventory:20.2.022880259210241814384512
mod-email1mod-email:1.17.02102489612876888128
mod-circulation2mod-circulation:24.2.122880259215361814384512
mod-pubsub1mod-pubsub:2.13.02153614401024922384512
mod-di-converter-storage2mod-di-converter-storage:2.2.22102489612876888128
edge-orders1edge-orders:3.0.02102489612876888128
edge-rtac1edge-rtac:2.7.12102489612876888128
mod-users1mod-users:19.3.12102489612876888128
mod-template-engine1mod-template-engine:1.20.02102489612876888128
mod-patron-blocks1mod-patron-blocks:1.10.021024896102476888128
mod-audit1mod-audit:2.9.02102489612876888128
edge-fqm1edge-fqm:2.0.02102489612876888128
mod-source-record-manager1mod-source-record-manager:3.8.025600500020483500384512
nginx-edge1nginx-edge:2023.06.1421024896128000
mod-quick-marc1mod-quick-marc:5.1.01228821761281664384512
nginx-okapi1nginx-okapi:2023.06.1421024896128000
okapi-b1okapi:5.3.03168414401024922384512
mod-feesfines1mod-feesfines:19.1.02102489612876888128
mod-invoice-storage1mod-invoice-storage:5.8.021872153610241024384512
mod-service-interaction1mod-service-interaction:4.0.12204818442561290384512
mod-dcb1mod-dcb:1.1.02102489612876888128
mod-patron1mod-patron:6.1.02102489612876888128
mod-data-export1mod-data-export:5.0.01204815241024000
mod-oai-pmh1mod-oai-pmh:3.13.024096369020483076384512
edge-connexion1edge-connexion:1.2.02102489612876888128
mod-notes1mod-notes:5.2.021024896128952384512
mod-kb-ebsco-java1mod-kb-ebsco-java:4.0.02102489612876888128
mod-login1mod-login:7.11.02144012981024768384512
mod-organizations-storage1mod-organizations-storage:4.7.02102489612870088128
mod-data-export-spring1mod-data-export-spring:3.2.01204818442561536384512
pub-okapi1pub-okapi:2023.06.142102489612876800
mod-eusage-reports1mod-eusage-reports:2.1.12102489612876888128

Additional Files

  • source files for SLI CPU and Memory dashboard creation 

Methodology

jMeter request to trigger instance sharing:

  • CSI_POST /consortia/{consortiumId}/sharing/instances
    BODY: {"sourceTenantId":"${sourceTenantId}","instanceIdentifier":"${list_id1}","targetTenantId":"${targetTenantId}","id":"${__UUID()}"}

Test 1. The shared local instances process was started from the Jmeter script for Tenant cs00000int_0001, cs00000int_0002, cs00000int_0003, and cs00000int_0004 one after another. 

Test 2. The shared local instances process was started from the Jmeter script in parallel for Tenant cs00000int_0001 and cs00000int_0002. 

Test 3. The shared local instances process was started from the Jmeter script in parallel for Tenant cs00000int_0001,  cs00000int_0002, and cs00000int_0003. 

Test 4. The shared local instances process was started from the Jmeter script in parallel for Tenant cs00000int_0001,  cs00000int_0002, cs00000int_0003, and cs00000int_0004. 

Mykhailo Petryshyn
May 22, 2024

Error rate =0