PTF - LOC all workflows testing (RELC)
Overview
In this report, PTF conducts performance testing to identify the appropriate environment scale that will ensure the fulfillment of the NFRs outlined in ASA NFRs. The goal is also to identify potential bottlenecks in the system that may require code changes to optimize performance and meet the specified requirements.
PERF-1079: Test ASA workflowsClosed
Summary
During testing with different configurations, we observed that once OpenSearch exceeded 80% of its SPU capacity, performance was significantly impacted, leading to degraded ASA operations. As a result, we determined that the ASA load with additional traffic on the PTF-LOC environment should not exceed 80% of OpenSearch's SPU capacity. Therefore, we will use the r7g.4xlarge instance type to maintain optimal performance.
During ASA testing with extra load on the PTF-LOC environment, OpenSearch SPU utilization reached 62%, compared to 52% without the extra load. We expected that OpenSearch CPU utilization would increase to 70% with the additional load, as in Test 1, where 31 sessions were lost due to insufficient capacity.
Additionally, we found that operations involving the "CONTAINS_ALL" search criterion were significantly slower, with response times 6 to 12 times higher compared to other criteria.
Despite the extra load on the PTF-LOC environment, overall performance remained stable, with variations of less than 5%.
NFR_007 - Performance degradation should be less than 10% for each block of active users, which meets the NFR requirements.
NFR_006 - The 95th percentile of response time is greater than 1 second, which does not match the NFR due to the use of the "CONTAINS_ALL" search criterion. However, the 99th percentile of response time is less than 2 seconds, which meets the NFR, even with "CONTAINS_ALL" included.
NFR_004 - The 95th percentile of response time exceeds 2 seconds for some "CONTAINS_ALL" operations, which does not match the NFR, but all other ASA operations match the NFR.
The total number of searching and viewing operations is 31 per second. Since we lost 31 user sessions, we can infer that the operations are close to 33 per second, which meets the NFR requirements.
Test Runs
Test # | Description | Status |
|---|---|---|
Test №1 | Ran the Master Script with 14 different flows as extra load on the PTF-LOC environment and ASA test. | Completed |
Test №2 | Ran the ASA test without extra load on the PTF-LOC environment. | Completed |
|
|---|
Test Results
This table contains response times and percentage degradation for API Load Testing with a Gradual Increase of Users Per Block tests.
transaction | Number Of | Expected | Expected | Actual | Actual | pct90 | NumOfErrors | Error Rate (%) |
|---|---|---|---|---|---|---|---|---|
TC_LS: Open advanced search | 927 | 2000 | 7/s | 121 | 0,542 | 290 |
|
|
TC_LS: Open-Main-Page | 2062 | 2000 | 261 | 1,17 | 288 |
|
| |
TC_LS:1.1Keyword-ContainsAll_|_Title-ContainsAll_|_DateRange-AllTime | 76 | 2000 | 1104 | 0,214 | 2435 |
|
| |
TC_LS:1.2Keyword-ContainsAll_|_Title-ContainsAll_|_DateRange-5years | 89 | 2000 | 961 | 0,231 | 2012 |
|
| |
TC_LS:1.3Keyword-ContainsAll_|_Title-ContainsAll_|_DateRange-CustomRange | 89 | 2000 | 1076 | 0,225 | 2009 |
|
| |
TC_LS:2.1Keyword-StartWith_|_Title-ContainsAll_|_DateRange-AllTime | 85 | 2000 | 1026 | 0,21 | 2490 |
|
| |
TC_LS:2.2Keyword-StartWith_|_Title-ContainsAll_|_DateRange-5years | 92 | 2000 | 1201 | 0,219 | 2473 |
|
| |
TC_LS:2.3Keyword-StartWith_|_Title-ContainsAll_|_DateRange-CustomRange | 100 | 2000 | 1625 | 0,217 | 3779 |
|
| |
TC_LS:3.1Keyword-ContainsAll_|_Title-ContainsAll_|_Author-ContainsAll_|_DateRange-AllTime | 95 | 2000 | 904 | 0,221 | 1697 |
|
| |
TC_LS:3.2Keyword-ContainsAll_|_Title-ContainsAll_|_Author-ContainsAll_|_Subject-ContainsAll_|_DateRange-AllTime | 82 | 2000 | 1084 | 0,222 | 2535 |
|
| |
TC_LS:3.3Keyword-ContainsAll_|_Title-ContainsAll_|_Author-ContainsAll_|_DateRange-AllTime_|_Format-other | 76 | 2000 | 875 | 0,214 | 2055 |
|
| |
TC_LS:3.4Keyword-ContainsAll_|_Title-ContainsAll_|_Author-ContainsAll_|_DateRange-AllTime_|_Format-other_]_Language-English; | 60 | 2000 | 846 | 0,214 | 1817 |
|
| |
TC_LS:3.5Keyword-ContainsAll_|_Title-ContainsAll_|_Author-ContainsAll_|_DateRange-AllTime_|_Format-other_]_Language-Englis_|_Location-Main Library | 84 | 2000 | 1479 | 0,227 | 2908 |
|
| |
TC_LS:Format: SourceTypes Other | 93 | 2000 | 1030 | 0,219 | 3707 |
|
| |
TC_LS:Language:Undetermined | 79 | 2000 | 3043 | 0,232 | 19772 |
|
| |
TC_LS:Location:Annex | 106 | 2000 | 1057 | 0,23 | 2742 |
|
| |
TC_LS:Open 2nd page | 946 | 2000 | 1126 | 0,565 | 2561 | 2 | 0,211 | |
TC_LS:Open 3rd page | 855 | 2000 | 1065 | 0,533 | 2607 | 4 | 0,468 | |
TC_LS:Open-Instance-page | 6028 | 2000 | 143 | 3,33 | 259 | 7 | 0,116 | |
TC_LS:PublicationYear: Past5Years | 80 | 2000 | 913 | 0,229 | 3751 |
|
| |
TC_LS:Publisher:NotIndetified | 95 | 2000 | 799 | 0,216 | 1916 |
|
| |
TC_LS:Search-by-author | 455 | 2000 | 1087 | 0,285 | 2905 | 1 | 0,22 | |
TC_LS:Search-by-keyword | 341 | 2000 | 1233 | 0,242 | 2698 |
|
| |
TC_LS:Search-by-subject | 227 | 2000 | 1513 | 0,211 | 2528 |
|
| |
TC_LS:Status:Available | 443 | 2000 | 1631 | 0,325 | 2574 |
|
| |
TC_LS:Status:Available&Location:Annex | 83 | 2000 | 1135 | 0,214 | 2807 | 1 | 1,2 | |
TC_LS:Status:Available&Location:Annex&PublicationYear: Past5Years | 97 | 2000 | 1215 | 0,228 | 2745 |
|
| |
TC_LS:Subject: Electronic books | 90 | 2000 | 1991 | 0,222 | 9504 |
|
| |
TC_main_AIE: Click action pay - Paying Invoices | 4 | 10000 | 3 | 1512 | 0,2 | 1556 |
|
|
TC_main_AIE: View Invoices Workflow cs00000int - View Invoices | 9 | 10000 | 6 | 1937 | 0,2 | 3712 |
|
|
TC_main_AIE_AIE_SC: Invoices Approve - Approve Invoices | 4 | 10000 | 3 | 429 | 0,2 | 455 |
|
|
TC_main_AIE_SC: Invoices Edit Save - Edit Invoices (via UI) | 8 | 10000 | 6 | 163 | 0,2 | 186 |
|
|
TC_main_AIE_SC: Invoices and invoice lines Create Save - Create Invoices | 8 | 2000 | 6 | 339 | 0,2 | 399 |
|
|
TC_main_APO: Load invoice| Create |Add POLine 85| Workflow cs00000int | 13 | 120000 |
| 127367 | 0,2 | 157281 |
|
|
TC_main_ASA: Searching | 2566 | 2000 | 1 | 202 |