Overview

• The purpose of the tests is to measure API calls in Poppy release in scope of  PERF-804 - Getting issue details... STATUS  ticket on PCP1 environment.

Summary

• API with S3 bucket storage work faster with big files. More information in Test Results section.
• API with S3 bucket storage handle concurrent requests better than API with DB storage. With DB response times increased twice or more, with S3 bucket increased 25% - 45%.
• Some growing trend detected for CPU utilization in mod-users module.

• Tests with 1 API request POST /users/profile-picture sequentially with 1 user and 2 API requests POST /users/profile-picture, GET /users/profile-picture/[profile] with 3 users concurrently
  • Test set #1 
    • 1000 requests x 100kb
    • 1000 requests x 3.6Mb
    • 1000 requests x 5Mb
  • Test set #2
    • 100 requests x 100kb
    • 100 requests x 3.6Mb
    • 100 requests x 5Mb
    • 100 requests x 10Mb

Results

• Test set #1
  • DB storage
    • Test #2 1000 request
      • Total duration: 46 minutes
        • Duration 100kb  - < 1 minute, Response times - 24 ms
        • Duration 3.6Mb  - 19 minutes, Response times - 1115 ms
        • Duration 5Mb    - 26 minutes,  Response times - 1588 ms
  • S3 bucket storage
    • Test #2 1000 requests
      • Total duration: 19 minutes
        • Duration 100kb  - 1 minute 50 sec, Response times - 111 ms
        • Duration 3.6Mb  - 6 minutes,          Response times - 380 ms
        • Duration 5Mb     - 11 minutes,        Response times - 642 ms
• Test set #2
  
  • DB storage
    
    • 1 user
      • 100kb  - Response times - 25 ms
      • 3.6Mb  - Response times - 1074 ms
      • 5Mb     - Response times - 1559 ms
      • 10Mb    - Response times - 2889 ms
    • 3 users
      • 100kb  - Response times - 29 ms
      • 3.6Mb  - Response times - 2301 ms
      • 5Mb     - Response times - 3523 ms
      • 10Mb    - Response times - 5851 ms
  • S3 bucket storage
    • 1 user
      • 100kb  - Response times - 128 ms
      • 3.6Mb  - Response times - 394 ms
      • 5Mb     - Response times - 709 ms
      • 10Mb    - Response times - 1216 ms
    • 3 users
      • 100kb  - Response times - 150ms
      • 3.6Mb  - Response times - 532 ms
      • 5Mb     - Response times - 926 ms
      • 10Mb    - Response times - 1858 ms

Resource utilization

• Test set #1
  
  • CPU utilization mod-users module
    
    • Test with DB storage - 63%
    • Test with S3 bucket   - 170%
  • Memory consumption mod-users module
    • Test with DB storage - 54%
    • Test with S3 bucket   - 54%

• Test set #2
  • CPU utilization mod-users module
    
    • Test with DB storage - 1 user - 148%, 3 users - 250%
    • Test with S3 bucket   - 1 user - 254%, 3 users - 547%
  • Memory consumption mod-users module
    • Test with DB storage - 73%
    • Test with S3 bucket   - 77%

Recommendations & Jiras

• For S3 bucket storage tests a bucket was created with name fs09000000 (tenant Id) and AWS_URL created in task definition to point S3 service. It's recommended to add cluster name to the link as well to exclude possibility of cross requests from different clusters to the same bucket.
• It's impossible to detect file name or file size or createdDate for the records in DB [tenant]_mod_users.profile_picture table. Consider adding this to get more info about uploaded profile pictures.
• Allocate more CPU for mod-users as it grew up to 550% during 3 concurrent requests.
• Before allocating more CPU resources investigate mod-users behaviour because during Test set#2 of tests it started with 144% with 1 request on DB storage. But previously we saw that it started with 63%.

Test Runs & Results

Test set #1 - 1000 requests (POST)

Test set #2 - 100 requests (POST, GET)

Test set #2

Service CPU Utilization

Service Memory Consumption

During 3 concurrent profile picture uploading to S3 bucket mod-users used 77%. Uploading to DB mod-users utilized 73%.

Test set #1 Test #2 with DB storage

Service CPU Utilization

mod-users module CPU Utilization was 63% during tests

Service Memory Consumption

mod-users module memory consumption was 54%

RDS CPU Utilization

RDS Database Connections

RDS connections level before and during test - 385

Test set #1 Test #2 with S3 bucket storage source

Service CPU Utilization

mod-users module CPU Utilization was 170% during tests

Service Memory Consumption

mod-users module memory consumption was 54%

Response Times Over Time

Test set #1 Test #2 with DB storage 1 user 1000 requests

Test set #1 Test #2 with S3 bucket storage 1 user 1000 requests

Test set #2 Test #2 with DB storage 1 user 100 requests

Test set #2 Test #2 with S3 bucket storage 1 user 100 requests

Test set #2 Test #2 with DB storage 3 user 100 requests

Test set #2 Test #2 with S3 bucket storage 3 user 100 requests

Appendix

Methodology/Approach

• To get config Id and change storage source use API

  • GET /users/configurations/entry

  • PUT /users/configurations/entry/{configId}

• To upload picture to DB use API

  • POST /users/profile-picture

• To get or to check uploaded profile picture use API where profileId the record in DB [tenant]_mod_users.profile_picture

  • GET /users/profile-picture/{profileId}

•  To connect AWS S3 bucket storage add to task definition of mod-users module and restart it with changes in revision

  • {
    "name": "AWS_URL",
    "value": "https://s3.amazonaws.com"
    }

Infrastructure

Environment: PCP1
Release: Poppy (2023 R2)

• 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)
• 2 instances of db.r6.xlarge database instances, one reader, and one writer 
• MSK tenant
  • 4 brokers

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • og.retention.minutes=480
  • default.replication.factor=3

Modules