Performance Testing Methodology

Environment

Use the default UChicago dataset - 27M records
Other datasets and their sizes: Check with P.Os, depends on the workflow to test.
Run two environments - 1 with profiler and the other one withOUT profiler.

Establishing test scenarios and conditions, SLA with POs, especially for the scenarios that we come up with.
Maintaining a test log - write down time of tests execution and conditions
- parameters:
  - dataset name or the number of records in the database
  - log level of all modules and/or a specific module
  - FOLIO version and/or specific modules versions
  - With or Without profiler
  - Number of users
  - Duration
  - Other configurations or settings (TBD)
Feasible to restart the cluster so that all the ECS services have a fresh starting point in terms of CPU and memory?
- Short duration tests, no need to restart environment every time
  - Keep an eye on env's metrics such as CPU and memory utilization, may need to take proactive action to restart the module or the whole env if the metrics reach a critical level.
- Long duration tests, need to restart environment to have a clean starting point.
Baseline tests/results:
- Only when absolutely required? E.g., whole new set of workflow
- Each time adding new version of module
- If parameters haven't changed, then don't need to rerun baseline.
pgHero is a tool that captures slow queries. Clear out pgHero if it has not been cleared already.
Run a smoke test to verify that there are no functional errors or that the environment has been set up successfully
Longevity tests
- Take a heap dump
Triple check the Jenkins parameter
If environment has been restarted, make sure that all ECS services are stable for at least 15 minutes