Uploading Patron profile picture

Overview

  • The purpose of the tests is to measure API calls in Poppy release in scope of  PERF-804 - Getting issue details... STATUS  ticket on PCP1 environment.

Summary

  • API with S3 bucket storage work faster with big files. More information in Test Results section.
  • API with S3 bucket storage handle concurrent requests better than API with DB storage. With DB response times increased twice or more, with S3 bucket increased 25% - 45%.
  • Some growing trend detected for CPU utilization in mod-users module.
  • Tests with 1 API request POST /users/profile-picture sequentially with 1 user and 2 API requests POST /users/profile-picture, GET /users/profile-picture/[profile] with 3 users concurrently
    • Test set #1 
      • 1000 requests x 100kb
      • 1000 requests x 3.6Mb
      • 1000 requests x 5Mb
    • Test set #2
      • 100 requests x 100kb
      • 100 requests x 3.6Mb
      • 100 requests x 5Mb
      • 100 requests x 10Mb

Results

  • Test set #1
    • DB storage
      • Test #2 1000 request
        • Total duration: 46 minutes
          • Duration 100kb  - < 1 minute, Response times - 24 ms
          • Duration 3.6Mb  - 19 minutes, Response times - 1115 ms
          • Duration 5Mb    - 26 minutes,  Response times - 1588 ms
    • S3 bucket storage
      • Test #2 1000 requests
        • Total duration: 19 minutes
          • Duration 100kb  - 1 minute 50 sec, Response times - 111 ms
          • Duration 3.6Mb  - 6 minutes,          Response times - 380 ms
          • Duration 5Mb     - 11 minutes,        Response times - 642 ms
  • Test set #2
    • DB storage
      • 1 user
        • 100kb  - Response times - 25 ms
        • 3.6Mb  - Response times - 1074 ms
        • 5Mb     - Response times - 1559 ms
        • 10Mb    - Response times - 2889 ms
      • 3 users
        • 100kb  - Response times - 29 ms
        • 3.6Mb  - Response times - 2301 ms
        • 5Mb     - Response times - 3523 ms
        • 10Mb    - Response times - 5851 ms
    • S3 bucket storage
      • 1 user
        • 100kb  - Response times - 128 ms
        • 3.6Mb  - Response times - 394 ms
        • 5Mb     - Response times - 709 ms
        • 10Mb    - Response times - 1216 ms
      • 3 users
        • 100kb  - Response times - 150ms
        • 3.6Mb  - Response times - 532 ms
        • 5Mb     - Response times - 926 ms
        • 10Mb    - Response times - 1858 ms

Resource utilization

  • Test set #1
    • CPU utilization mod-users module
      • Test with DB storage - 63%
      • Test with S3 bucket   - 170%
    • Memory consumption mod-users module
      • Test with DB storage - 54%
      • Test with S3 bucket   - 54%
  • Test set #2
    • CPU utilization mod-users module
      • Test with DB storage - 1 user - 148%, 3 users - 250%
      • Test with S3 bucket   - 1 user - 254%, 3 users - 547%
    • Memory consumption mod-users module
      • Test with DB storage - 73%
      • Test with S3 bucket   - 77%

Recommendations & Jiras

  • For S3 bucket storage tests a bucket was created with name fs09000000 (tenant Id) and AWS_URL created in task definition to point S3 service. It's recommended to add cluster name to the link as well to exclude possibility of cross requests from different clusters to the same bucket.
  • It's impossible to detect file name or file size or createdDate for the records in DB [tenant]_mod_users.profile_picture table. Consider adding this to get more info about uploaded profile pictures.
  • Allocate more CPU for mod-users as it grew up to 550% during 3 concurrent requests.
  • Before allocating more CPU resources investigate mod-users behaviour because during Test set#2 of tests it started with 144% with 1 request on DB storage. But previously we saw that it started with 63%.

Test Runs & Results

Test set #1 - 1000 requests (POST)



DB Storage

S3 bucket Storage     

Requests

Executions

Response Times (ms)

Executions

Response Times (ms)


Label

#Samples

Error %

Average

#Samples

Error %

Average

1 request tests  POST /users/profile-picture 3.6MB10.00%67410.00%870
POST /users/profile-picture 5MB10.00%76510.00%1244
POST /users/profile-picture 100kB10.00%25110.00%721








1000 request tests  POST /users/profile-picture 3.6MB10000.00%1115.4410000.00%380.64
POST /users/profile-picture 5MB10000.00%1588.0110000.00%642.39
POST /users/profile-picture 100kB10000.00%23.8510000.00%111.38

Test set #2 - 100 requests (POST, GET)



DB Storage     S3 bucket Storage     


1 user  3 users  1 user  3 users  

Requests

Executions

 

Response Times (ms)

Executions

 

Response Times (ms)

Executions

 

Response Times (ms)

Executions

 

Response Times (ms)


Label

#Samples

Error %

Average

#Samples

Error %

Average

#Samples

Error %

Average

#Samples

Error %

Average

100 /POSTPOST /users/profile-picture 100kB1000.00%25.083000.00%29.121000.00%128.93000.00%150.73
POST /users/profile-picture 3.6MB1000.00%1074.093000.00%2301.681000.00%394.643000.00%532.75
POST /users/profile-picture 5MB1000.00%1555.853000.00%3523.361000.00%709.253000.00%926.81
POST /users/profile-picture 10MB1000.00%2889.333000.00%5851.311000.00%1216.043000.00%1858.78














100 /GETGET /users/profile-picture/[ppid] 100kB1000.00%23.853000.00%29.471000.00%52.373000.00%85.17
GET /users/profile-picture/[ppid] 3.6MB1000.00%382.763000.00%391.921000.00%401.893000.00%449.69
GET /users/profile-picture/[ppid] 5MB1000.00%532.763000.00%568.91000.00%569.523000.00%773.59
GET /users/profile-picture/[ppid] 10MB1000.00%962.613000.00%1014.881000.00%1014.963000.00%1166.17

Test set #2

Service CPU Utilization

Service Memory Consumption

During 3 concurrent profile picture uploading to S3 bucket mod-users used 77%. Uploading to DB mod-users utilized 73%.


Test set #1 Test #2 with DB storage

Service CPU Utilization

mod-users module CPU Utilization was 63% during tests

Service Memory Consumption

mod-users module memory consumption was 54%

RDS CPU Utilization

RDS Database Connections

RDS connections level before and during test - 385

Test set #1 Test #2 with S3 bucket storage source

Service CPU Utilization

mod-users module CPU Utilization was 170% during tests

Service Memory Consumption

mod-users module memory consumption was 54%

Response Times Over Time

Test set #1 Test #2 with DB storage 1 user 1000 requests

Test set #1 Test #2 with S3 bucket storage 1 user 1000 requests

Test set #2 Test #2 with DB storage 1 user 100 requests

Test set #2 Test #2 with S3 bucket storage 1 user 100 requests

Test set #2 Test #2 with DB storage 3 user 100 requests

Test set #2 Test #2 with S3 bucket storage 3 user 100 requests

Appendix

Methodology/Approach

Infrastructure

Environment: PCP1
Release: Poppy (2023 R2)

  • 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)
  • 2 instances of db.r6.xlarge database instances, one reader, and one writer 
  • MSK tenant
    • 4 brokers
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • og.retention.minutes=480
    • default.replication.factor=3

Modules

 All modules
ModuleTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabled
pcp1-pvt









Thu Feb 29 11:11:48 UTC 2024









mod-remote-storage11mod-remote-storage:3.0.024920447210243960512512FALSE
mod-ncip10mod-ncip:1.14.42102489612876888128FALSE
mod-finance-storage10mod-finance-storage:8.5.021024896102470088128FALSE
mod-agreements10mod-agreements:6.0.2215921488128000FALSE
mod-ebsconet10mod-ebsconet:2.1.1212481024128700128256FALSE
edge-sip28edge-sip2:3.1.12102489612876888128FALSE
mod-organizations10mod-organizations:1.8.02102489612870088128FALSE
mod-settings11mod-settings:1.0.22102489620076888128FALSE
edge-dematic10edge-dematic:2.1.01102489612876888128FALSE
mod-data-import20mod-data-import:3.0.71204818442561292384512FALSE
mod-search20mod-search:3.0.5225922480204814405121024FALSE
mod-tags10mod-tags:2.1.02102489612876888128FALSE
mod-authtoken16mod-authtoken:2.14.121440115251292288128FALSE
edge-courses2edge-courses:1.3.02102489612876888128FALSE
mod-notify10mod-notify:3.1.02102489612876888128FALSE
mod-inventory-update10mod-inventory-update:3.2.12102489612876888128FALSE
mod-configuration10mod-configuration:5.9.22102489612876888128FALSE
mod-orders-storage10mod-orders-storage:13.6.02102489651270088128FALSE
edge-caiasoft10edge-caiasoft:2.1.02102489612876888128FALSE
mod-login-saml18mod-login-saml:2.7.12102489612876888128FALSE
mod-erm-usage-harvester11mod-erm-usage-harvester:4.4.12102489612876888128FALSE
mod-password-validator10mod-password-validator:3.1.0214401298128768384512FALSE
mod-licenses10mod-licenses:5.0.22248023121281792384512FALSE
mod-gobi10mod-gobi:2.7.12102489612870088128FALSE
mod-fqm-manager7mod-fqm-manager:1.0.32102489612876888128FALSE
mod-bulk-operations9mod-bulk-operations:1.1.723072260010241536384512FALSE
mod-graphql16mod-graphql:1.12.02102489612876888128FALSE
mod-finance10mod-finance:4.8.02102489612870088128FALSE
mod-erm-usage13mod-erm-usage:4.6.02102489612876888128FALSE
mod-copycat10mod-copycat:1.5.02102489612876888128FALSE
mod-lists6mod-lists:1.0.52300026001282048384512FALSE
mod-entities-links15mod-entities-links:2.0.4225922480400144001024FALSE
mod-permissions43mod-permissions:6.3.22168415445121024384512FALSE
pub-edge9pub-edge:2023.06.142102489612876800FALSE
mod-orders10mod-orders:12.7.122048144010241024384512FALSE
edge-patron10edge-patron:5.0.02102489625676888128FALSE
edge-ncip11edge-ncip:1.9.22102489612876888128FALSE
mod-users-bl10mod-users-bl:7.6.021440115251292288128FALSE
mod-inventory-storage15mod-inventory-storage:27.0.424096369020483076384512FALSE
mod-invoice10mod-invoice:5.7.221440115251292288128FALSE
mod-user-import10mod-user-import:3.8.02102489612876888128FALSE
mod-sender10mod-sender:1.11.02102489612876888128FALSE
edge-oai-pmh8edge-oai-pmh:2.7.121512136010241440384512FALSE
mod-data-export-worker10mod-data-export-worker:3.1.223072280010242048384512FALSE
mod-rtac10mod-rtac:3.5.02102489612876888128FALSE
mod-circulation-storage14mod-circulation-storage:17.1.722880259215361814384512FALSE
mod-calendar10mod-calendar:2.5.02102489612876888128FALSE
mod-source-record-storage18mod-source-record-storage:5.7.525600500020483500