PTF -RTAC Workflow Test Report (Quesnelia)


WIP

Work in progress...

Overview

PERF-932 - Getting issue details... STATUS  was created to re-run RTAC performance test in the Quesnelia environment which will verify performance issues reported by  PERF-895 - Getting issue details... STATUS

 We tested mod-rtac, edge-rtac with 1, 15, 20. 50 virtual users for 15, 30 minutes with different combinations of instance size.

Summary

  1. On average edge-rtac GET_/rtac?instanceIds=id1,id2,id3,idn API call takes 807 ms for 50 instance with 3,4,5,6,7 holdings. This is an improvement from 2.6 seconds in Honeysuckle.
  2. On average mod-rtac POST_/rtac-batch API call takes 487 ms for 50 instances with 3,4,5,6,7 holdings. This is an improvement from 2.6 seconds in Honeysuckle.
  3. On running mod-inventory-storage POST_inventory-hierarchy/items-and-holdings for 20 concurrent virtual users API call takes 500 ms for 50 instances with 3,4,5,6,7 holdings. This is an improvement from 9.3 seconds in Honeysuckle. 

Test Runs

Test

Concurrent Virtual Users

Duration

Holdings per instance

API call

1. 

1

30 min

greater than 300

mod-rtac POST https://[hostName]/rtac-batch
2.

1

30 mingreater than 300edge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true
3.130 min50 instances with a mix of 3,4,5,6,7 holdings per instanceedge-rtac GET https://edge-[hostName]/rtac?instanceIds=id1,id2,id3,id4,id5,id6,id7,id8,id9,id10&apikey={API_KEY}&fullPeriodicals=true
4.130 min50 instances with a mix of 3,4,5,6,7 holdings per instancemod-rtac POST https://[hostName]/rtac-batch
5.215 min50 instances with a mix of 3,4,5,6,7 holdings per instancemod-rtac POST https://[hostName]/rtac-batch
6.2015 min50 instances with a mix of 3,4,5,6,7 holdings per instancemod-rtac POST https://[hostName]/rtac-batch
7.1515 min50 instances with a mix of 3,4,5,6,7 holdings per instancemod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings 
8.2015 min50 instances with a mix of 3,4,5,6,7 holdings per instancemod-inventory-storage POST https://[hostName]/inventory-hierarchy/items-and-holdings 


Results

mod-rtac

A mod-rtac level request to get all open loans for greater than 300 holdings per instance

POST /rtac-batch for 30 minutesTotal InstancesTotal HoldingsAverage 95th %tileAverage (Honeysuckle)Delta
1 vUser1333790 ms854 ms529 ms+261 ms
1 vUser50mix of 3,4,5,6,7487 ms516 ms2.57 seconds-2.1 sec
1 vUser30>=10 (one has 5k) from  PERF-895 - Getting issue details... STATUS 2659 ms2933 ms--
1 vUser1>=10 from  PERF-895 - Getting issue details... STATUS 82 ms86 ms--

This test was run to exaggerate with 20 virtual users to recreate an error from mod-rtac. Out of 50 instance lookups, 3 lookups failed with facade error mapping.

POST /rtac-batch for 15 minutesTotal InstancesTotal HoldingsAverage 95th %tileAverage (Honeysuckle)Delta
2 vUsers50mix of 3,4,5,6,7538 ms627 ms2.97 seconds-2.4 sec
20 vUsers50mix of 3,4,5,6,73622 ms4194 ms21.75 seconds-18.1 sec

No errors for mod-rtac were found in CloudWatch logs

mod-rtac calls inventory-hierarchy.  API tests were run for 15 virtual users and 20 virtual users. No errors were reported.

POST inventory-hierarchy/items-and-holdings for 15 minutesTotal InstancesTotal HoldingsAverage 95th %tileAverage (Honeysuckle)Delta
15 vUsers50mix of 3,4,5,6,7371 ms513 ms7.11 seconds-6.7 sec
20 vUsers50mix of 3,4,5,6,7500 ms704 ms9.34 seconds-8.8 sec


edge-rtac

An edge-rtac level request to get all open loans for greater than 300 holdings per instance

GET /rtac for 30 minutesTotal InstancesTotal HoldingsAverage 95th %tileAverage (Honeysuckle)Delta
1 vUser1333800 ms972 ms721 ms+79 ms
1 vUser50mix of 3,4,5,6,7807 ms1349 ms2.58 seconds-1.7 sec
1 vUser50mix of 11,125796 ms10997 ms--
2 vUsers50mix of 11,126904 ms14619 ms--
20 vUsers50mix of 11,1234725 ms57423 ms--
1 vUser30>=10 (one has 5k) from   PERF-895 - Getting issue details... STATUS 4887 ms5165 ms--
50 vUsers30>=10 (one has 5k) from   PERF-895 - Getting issue details... STATUS 63966 ms10123 ms--
1 vUser1>=10 (one has 5k) from   PERF-895 - Getting issue details... STATUS 239 ms412 ms--

Slow Queries

1 instance with 333 holdings


Average Time, ms

Module

Query

1 vUser119 msmod_inventory_storage

select * from get_items_and_holdings_view($1,$2)

50 instances with a mix of 3,4,5,6,7 holdings per instance


Average Time, ms

Module

Query

20 vUsers1057 msmod_inventory_storage

select * from get_items_and_holdings_view($1,$2)


RDS CPU Utilization

CPU is around 99.5% for /inventory-hierarchy. For 1 user was about 18%

Service Memory Utilization

Service CPU Utilization

Appendix

  • 14 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 
  • 1 instance of db.r6.xlarge database instance: Writer instance
  • MSK tenant [ kafka configurations]
    • 4 kafka.m5.2xlarge brokers in 2 zones
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

Modules memory and CPU parameters

ModuleTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSize
qcp1-pvt

mod-rtac5
mod-rtac:3.6.0
2