...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
Performance Testing is a software testing process used for testing the speed, response time, stability, reliability, scalability, and resource usage of a software application under a particular workload. The main purpose of performance testing is to identify and eliminate the performance bottlenecks in the software application.
Features and Functionality supported by a software system are not the only concern. A software application’s performance, like its response time, reliability, resource usage, and scalability, do matter. The goal of Performance Testing is not to find bugs but to eliminate performance bottlenecks.
Performance Testing is done to provide stakeholders with information about their application regarding speed, stability, and scalability. More importantly, Performance Testing uncovers what needs to be improved before the product goes to market. Without Performance Testing, the software is likely to suffer from issues such as running slow while several users use it simultaneously, inconsistencies across different operating systems, and poor usability.
The performance testing team (PTF Team) is in charge of the performance testing lifecycle (PTLC).
PTLC contains the following stages:
- Performance acceptance criteria
- Test planning
- System modelling
- Test scripts development
- Test execution
- System tuning
- Test result reporting
Performance acceptance criteria
This includes goals and constraints for throughput, response times and resource allocation. It is also necessary to identify project success criteria outside of these goals and constraints. Testers should be empowered to set performance criteria and goals because often the project specifications will not include a wide enough variety of performance benchmarks. Sometimes there may be none at all. When possible finding a similar application to compare to is a good way to set performance goals.
TBC
Test planning
Test planning is particularly important for performance testing due to the need for the allocation of test environments, test data, tools and human resources. In addition, this is the activity in which the scope of performance testing is established. During test planning, risk identification and risk analysis activities are completed and relevant information is updated in any test planning documentation (e.g., test plan, level test plan). Just as test planning is revisited and modified as needed, so are risks, risk levels and risk status modified to reflect changes in risk conditions.
TBD
System modelling
Know your physical test environment, production environment and what testing tools are available. Understand the details of the hardware, software and network configurations used during testing before you begin the testing process. It will help testers create more efficient tests. It will also help identify possible challenges that testers may encounter during the performance testing procedures.
PROD Config
Test ENV #1 - ncp3
Test ENV #4 - ncp4
Environment
Use the default UChicago dataset - 27M records
Other datasets and their sizes: Check with P.Os,
...
depending on the workflow to test.
Run two environments - 1 with a profiler and the other one withOUT a profiler.
Test development
Determine how usage is likely to vary amongst end users and identify key scenarios to test for all possible use cases. It is necessary to simulate a variety of end users, plan performance test data and outline what metrics will be gathered.
In the implementation phase, performance test cases are ordered into performance test procedures. These performance test procedures should reflect the steps normally taken by the user and other functional activities that are to be covered during performance testing. A test implementation activity is establishing and/or resetting the test environment before each test execution. Since performance testing is typically data-driven, a process is needed to establish test data that is representative of actual production data in volume and type so that production use can be simulated.
Test execution
Test execution occurs when the performance test is conducted, often by using performance test tools. Test results are evaluated to determine if the system’s performance meets the requirements and other stated objectives. Any defects are reported.
System tuning
Test result reporting
PreTest
- Establishing test scenarios and conditions, and SLA with POs, especially for the scenarios that we come up with.
- Maintaining a test log - write down time of tests execution and conditions (see the attachment for sample logs)
- parameters:
- dataset name or the number of records in the database
- log level of all modules and/or a specific module
- FOLIO version and/or specific modules versions
- With or Without profiler
- Number of users
- Duration
- Other configurations or settings (TBD)
- parameters:
- Feasible to restart the cluster so that all the ECS services have a fresh starting point in terms of CPU and memory?
- Short duration tests, no need to restart environment every time
- Keep an eye on env's metrics such as CPU and memory utilization, may need to take proactive action to restart the module or the whole env if the metrics reach a critical level.
- Long-duration tests, need to restart the environment to have a clean starting point.
- Short duration tests, no need to restart environment every time
- Baseline tests/results:
- Only when absolutely required? E.g., a whole new set of workflow
- Each time adding a new version of a module
- If parameters haven't changed, then don't need to rerun the baseline.
- pgHero is a tool that captures slow queries. Clear out pgHero if it has not been cleared already.
- Run a smoke test to verify that there are no functional errors or that the environment has been set up successfully
- Longevity tests
- Take a heap dump
- Triple-check the Jenkins job's parameters
- If the environment has been restarted, make sure that all ECS services are stable for at least 15 minutes
...
- Capture any observations.
- Capturing heap dumps, esp. for longevity tests - manual process at the moment, maybe automated in the future. At a minimum: the beginning, middle, and end of the test run.
Post Test:
- Data collection - what pieces of data are important to collect
- Average response time (Obtained from Grafana)
- Errors (thresholds for failing an API call (Obtained from Grafana))
- Modules logs to see if any errors entries
- TPS - transactions per secondssecond
- CPU utilization for utilization for a particular module or for any abnormal behavior behaviour observed (from any module)
- Memory usage for a particular module or for any abnormal behavior behaviour observed (from any module)
- Updating test log
- With the observations above, any anomalies
- Update the timestamps or a Grafana URL so that we can go back and look at the graphs later
- Capture PgHero stats
- Save-As from browser
- If running a series of tests, don't wait until the end of the runs to look at the data. Check the data after each run to make sure nothing is questionable and if there were, we can address them right away.
- Write report
| View file | ||||
|---|---|---|---|---|
|