...
- Establishing test scenarios and conditions, SLA with POs, especially for the scenarios that we come up with.
- Maintaining a test log - write down time of tests execution and conditions
- parameters:
- dataset name or the number of records in the database
- log level of all modules and/or a specific module
- FOLIO version and/or specific modules versions
- With or Without profiler
- Number of users
- Duration
- Other configurations or settings (TBD)
- parameters:
- Feasible to restart the cluster so that all the ECS services have a fresh starting point in terms of CPU and memory?
- Short duration tests, no need to restart environment every time
- Keep an eye on env's metrics such as CPU and memory utilization, may need to take proactive action to restart the module or the whole env if the metrics reach a critical level.
- Long duration tests, need to restart environment to have a clean starting point.
- Short duration tests, no need to restart environment every time
- Baseline tests/results:
- Only when absolutely required? E.g., whole new set of workflow
- Each time adding new version of module
- If parameters haven't changed, then don't need to rerun baseline.
- pgHero is a tool that captures slow queries. Clear out pgHero if it has not been cleared already.
- Run a smoke test to verify that there are no functional errors or that the environment has been set up successfully
- Longevity tests
- Take a heap dump
- Triple check the Jenkins job's parameter
- If environment has been restarted, make sure that all ECS services are stable for at least 15 minutes