Versions Compared

Old Version 39

changes.mady.by.user Stanislav Nehrii

Saved on

compared with

New Version 40

changes.mady.by.user Stanislav Nehrii

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

...

This table contains durations for all Workflows. 

WorkflowsTest 1
Start Time "8/21/24, 12:42 PM"
End Time "8/21/24, 1:04 PM"		Test 2
Start Time "8/21/24, 2:35 PM"
End Time "8/21/24, 2:57 PM"		Test 3 (repeat 2)
Start Time "8/22/24, 1:01 PM"
End Time "8/22/24, 1:23 PM"		Test 4
Start Time "8/23/24, 9:05 AM"
End Time "8/23/24, 9:27 AM"		Test 5  (repeat 4)
Start Time "8/23/24, 11:38 AM"
End Time "8/23/24, 12:00 PM"		Test 6
Start Time "8/23/24, 3:27 PM"
End Time "8/23/24, 3:49 PM"		Test 7 (repeat 6 )
Start Time "8/26/24, 8:48 AM"
End Time "8/26/24, 9:09 AM"		Test 8
Start Time "8/26/24, 9:18 AM"
End Time "8/26/24, 9:40 AM"		Test 9 (repeat 7 after Terminate)
Start Time "8/26/24, 11:39 AM"
End Time "8/26/24, 12:01 PM"		Test 10( repeat 8 after Terminate)
Start Time "8/26/24, 12:37 PM"
End Time "8/26/24, 12:59 PM"		Test 11
Start Time "8/28/24, 12:57 PM"
End Time "8/28/24, 1:19 PM"		Test 12
Start Time "8/31/24, 10:16 AM"
End Time "8/31/24, 10:38 AM"		Test 13
Start Time "8/31/24, 1:30 PM"
End Time "8/31/24, 1:51 PM"		Test 14 (terminate)
Start Time "8/31/24, 3:21 PM"
End Time "8/31/24, 3:43 PM"
Test 15
Start Time "9/2/24, 9:16 AM"
End Time "9/2/24, 9:38 AM"Test 2

Test 3

(repeat 2)

Test 4

Test 5

(repeat 4)

Test 6
 

Test 7

(repeat 6 )

Test 8

Test 9

(repeat 7 after Terminate)

Test 10

(repeat 8 after Terminate)

Test 11

 		Test 12
 		Test 13
 

Test 14

(Terminate)

Test 15Test 16 Test 17 

Average response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)ErrorsAverage response time (milliseconds)Errors
AIE_TC: Create Invoices10227100%7425100%7645100.00%8563100.00%8132100.00%7383100.00%7575100.00%7359100.00%10061100.00%11269100.00%8607100%11352100%8293100%9397100%10687100%8138100%7063100%
AIE_TC: Invoices Approve4649100%3281100%3252100.00%3442100.00%3175100.00%3081100.00%2966100.00%3100100.00%5464100.00%7132100.00%3438100%4792100%3506100%4052100%5297100%3054100%2873100%
AIE_TC: Paying Invoices5686100%3375100%3481100.00%3439100.00%3250100.00%3079100.00%2948100.00%3115100.00%4504100.00%4245100.00%3580100%4635100%3548100%4387100%6265100%3093100%2811100%
CICO_TC_Check-In Controller38280%23500%24460.00%23250.00%22590.00%20920.00%20460.00%21220.00%31130.00%29650.00%25080%31620%23960%28960%38170%21540%19020%
CICO_TC_Check-Out Controller59600%40850%42730.00%41290.00%39050.00%37690.00%35490.00%38160.00%51670.00%52240.00%43070%59541%44060%51000%67880%37210%34360%
CSI_TC:Share local instance1304519%160810%160910.00%1310518.63%1304618.98%160870.00%1311918.66%160710.00%160760.00%160580.00%43070%1326118%160540%1312319%160710%1307619%160050%
DE_Exporting MARC Bib records custom workflow534840%1013240%468600.00%971690.00%1600220.00%



442920.00%

723720.00%495150%17913638%648270%1273730%632060%849000%694460%
DE_Exporting MARC Bib records workflow427370%922130%648970.00%777470.00%2656510.00%



376220.00%

705570.00%456400%18790941%630850%1594530%842200%728050%699050%
EVA_TC: View Account8220%5780%5820.00%5780.00%5530.00%5230.00%5060.00%5340.00%35390.00%27090.00%6020%15132%5270%6700%9670%5030%4800%
ILR_TC: Create ILR19790%16050%16450.00%15950.00%15230.00%14360.00%13940.00%15280.00%31470.00%31220.00%15760%21240%17080%19280%26430%14960%12980%
MSF_TC: mod search by auth query222726%16720%8240.00%9332.15%6860.00%6860.00%6800.00%6920.00%6930.00%6220.00%7700%12240%6800%8100%10790%6660%6690%
MSF_TC: mod search by boolean query4482%4060%2010.00%2570.41%1720.00%1670.00%1670.00%1680.00%1670.00%1560.00%1930%2540%1630%2060%2850%1590%1630%
MSF_TC: mod search by contributors8820%9820%4950.00%5661.25%4070.00%4150.00%4020.00%4060.00%3940.00%3770.00%4410%5100%3993990%4800%6050%4803930%6054060%
MSF_TC: mod search by filter query6492%6030%3440.00%3580.23%2850.00%2860.00%2850.00%2910.00%2740.00%2670.00%3150%3430%2880%3310%4070%2850%2850%
MSF_TC: mod search by keyword query6040%5960%3450.00%3740.41%2900.00%2880.00%2860.00%2920.00%2740.00%2630.00%3160%3430%2830%3290%4020%2840%2870%
MSF_TC: mod search by subject query9301%9320%5150.00%5980.80%4200.00%4300.00%4190.00%4250.00%4080.00%3900.00%4560%5630%4160%4850%5880%4130%4170%
MSF_TC: mod search by title query23810%19650%12600.00%13620.49%10590.00%10610.00%10540.00%10640.00%10260.00%10120.00%10930%11570%10620%11350%12480%10640%10810%
OPIH_/oai/records50860%32480%32540.00%51930.00%30690.00%



41130.00%

24510.00%62670%48850%8820%30260%9430%29800%9400%
POO_TC: Add Order Lines601931%533410%545070.00%550080.00%546460.00%528060.00%521150.00%540830.00%494750.00%446150.00%565550%654362%592070%597320%621160%556610%530520%
POO_TC: Approve Order452940%417500%419350.00%424450.00%424860.00%415690.00%411940.00%420340.00%373100.00%339820.00%442550%461330%461890%452380%473870%425640%415070%
POO_TC Create Order343050%305830%309650.00%318530.00%314400.00%309700.00%303920.00%305750.00%280810.00%254510.00%326010%422671%330580%339960%228280%317450%304790%
RTAC_TC: edge-rtac40500%22280%21840.00%36650.00%37870.00%19780.00%38880.00%19800.00%92970.00%100410.00%38680%83480%21150%38540%22360%37290%20050%
SDIC_Single Record Import (Create)1478319%118820%114410.00%1386818.39%1334518.82%109640.00%1294918.23%110010.00%120150.00%115560.00%1384818%1623618%125380%1448218%337210%1335718%107020%
SDIU_Single Record Import (Update)223474%196850%192980.00%195040.00%37870.00%188690.00%187060.00%187480.00%217450.00%208290.00%197410%233763%207410%209330%228280%194200%183300%
TC: Receiving-an-Order-Line50466100%44954100%46684100.00%46396100.00%45548100.00%45629100.00%44356100.00%45106100.00%40982100.00%36920100.00%47570100%53302100%50146100%49516100%52266100%46404100%44880100%
Serials-Receiving-Workflow51572100%46047100%47460100.00%48142100.00%47661100.00%45645100.00%45404100.00%46299100.00%42105100.00%38241100.00%49262100%61545100%50285100%50943100%52493100%48123100%45827100%
Unreceiving-a-Piece9422100%8028100%8303100.00%8247100.00%7865100.00%7897100.00%7748100.00%7887100.00%7468100.00%6812100.00%8438100%9945100%8544100%8829100%9368100%8058100%7581100%
ULR_TC: Users loan Renewal Transaction40270%31980%32750.00%30600.00%29570.00%28140.00%28100.00%29690.00%446320.00%886020.00%31970%50801%34060%35830%49550%28650%26260%

Comparison

This graph shows the durations of all workflows compared between the best test result from Test №8, MCPT Basile configuration with 14 instances m6g.2xlarge and result from mcpt tunning report.

...

Service CPU Utilization

Here we can see that okapi modules used 42k% CPU power parameter CPU=2 for module.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics


DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics

DB CPU Utilization

DB CPU was 99% average with ERW: Exporting Receiving Information

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics


DB CPU Utilization

DB CPU was 99% maximum.

...

Service CPU Utilization

Here we can see that okapi used 43k% of the CPU power of parameter CPU=2.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics




DB CPU Utilization

DB CPU was 99%.

...

Service CPU Utilization

Here we can see that okapi used 46k% of the CPU power of parameter CPU=2.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics


DB CPU Utilization

DB CPU was 99%.

...

Service CPU Utilization

Here we can see that okapi used 46k% of the CPU power of parameter CPU=2.

...

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.

Kafka metrics


DB CPU Utilization

DB CPU was 99%.

...

Test №7 - 8

Introduction:  

  • Test 7: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow) MOBIUS test was run (Repeat Test 6).
  • Test 8: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run.

Objective: The objective of these tests was to validate the consistency of performance observed in Test 5 and Test 6. This was achieved by repeating the same configuration.

Results: We see performance improvements for Test 8.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi used 48k% of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics



DB CPU Utilization

DB CPU was 99%.

DB Connections

Max number of DB connections was 6010.

DB load

Top SQL-queries


Test №9 - 10

Introduction: 

  • Test 9: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow) MOBIUS test was run (Repeat Test 7 after Terminate Instances r7g.xlarge).
  • Test 10: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run (Repeat Test 8 after Terminate Instances r7g.xlarge).

Objective: The objective of these tests was to validate the consistency of performance observed in Test 7 and Test 8. This was achieved by repeating the same configuration and applying new random task locations per instance after terminating the r7g.xlarge instances.

Results: Performance result were worse on 50% for several workflows after terminating the r7g.xlarge instances and applying new random task locations per instance.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi used 48k% of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics




DB CPU Utilization

DB CPU was 99%.

DB Connections

Max number of DB connections was 5150.

DB load

Top SQL-queries

...

Test №11

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.

Objective: The goal of this test was to replace three c7g.large instances with one additional r7g.xlarge instance for all services, while retaining three c7g.large instances specifically for the okapi service.

Results: The performance was worse compared to when we used the additional three c7g.large instances for services with smaller workloads.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi used 45k% of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics



DB CPU Utilization

DB CPU was 99%.

DB Connections

Max number of DB connections was 5900.

DB load

Top SQL-queries


Test №12 - 13

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.

Objective: The objective of this test was to validate the degradation of performance observed in Test 11 by repeating the same configuration.

Results: We confirmed performance degradation with this configuration.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi used 43k% of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics

Image Modified





DB CPU Utilization

DB CPU was 99%.

DB Connections

Max number of DB connections was 6200.

DB load

Top SQL-queries


...

Test №14

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.

Objective: The objective of this test was to validate the degradation of performance observed in Test 11 by repeating the same configuration.

Results: We confirmed performance degradation with this configuration.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi used 45k% of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics



DB CPU Utilization

DB CPU was 99%.

DB Connections

Max number of DB connections was 5900.

DB load

Top SQL-queries

...

Test №15

Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi and mod-authtoken service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.

Objective: 

Results: We confirmed performance degradation with this configuration.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi used 38k% of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics



DB CPU Utilization

DB CPU was 99%.

DB Connections

Max number of DB connections was 5150.

DB load

Top SQL-queries

...

Test №16 - 17

Introduction: Test 16: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: r7g.2xlarge for others services. Fixed Load (average case) MOBIUS test was run. Test 17: Repeat Test 16 with the same environment configuration. 

Objective: The objective of this test was to reduce the number of instances in the main group from 10 to 5, while using higher capacity r7g.2xlarge instances instead of r7g.xlarge

Results: We observed performance improvements in Test 16 compared to Test 8, and by repeating the same configuration in Test 17, we confirmed these performance improvements.

Instance CPU Utilization

Service CPU Utilization

Here we can see that okapi used 45k% of the CPU power of parameter CPU=2.

Service Memory Utilization

Here we can't see any sign of memory leaks on every module. Memory shows stable trend.


Kafka metrics



DB CPU Utilization

DB CPU was 99%.

DB Connections

Max number of DB connections was 6100.

DB load

Top SQL-queries

...

PTF - Baseline MCPT environment configuration

  • 14 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer


    NameMemory GIBvCPUs

    db.r6g.4xlarge

    		128 GiB16 vCPUs


  • Open Search ptf-test 
    • Data nodes
      • Instance type - r6g.2xlarge.search
      • Number of nodes - 4
      • Version: OpenSearch_2_7_R20240502
    • Dedicated master nodes
      • Instance type - r6g.large.search
      • Number of nodes - 3
  • MSK ptf-KRaft-mode
    • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x 

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

...

Baseline MCPT Environment configuration according to tunning environment from previous report:  task count: 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instance type: m6g.2xlarge. Instances count: 14. Database r6g.4xlarge, Amazon OpenSearch Service  ptf-test: r6g.2хlarge.search (4 nodes).

...