Karate Test Stabilization
- kseniia_dubniak
1 root cause: Kafka Memory Limit
During running tests, Kafka pod restarts several times before and during tests. Pod restarts with OOMKilled status.
After resizing limit memory to 4096M and Xmx parameters to 2662m several modules show more stable result:
- mod-orders
- mod-circulation
- mod-ebsconet
- edge-inn-reach
Not stable modules
After resizing memory for Kafka several modules have float issues:
- data-import
- mod-kb-ebsco-java
- mod-quick-marc
More popular issues
A lot of issues look like this:
And issues that I don`t understand https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/324/cucumber-html-reports/report-feature_138_1684211029.html
Not all tests start due to issues:
Timeout issue:
Different status codes:
2 root cause: Nodes large load
Before running the test adds new nodes to the cluster. First time on some "old" nodes have large load:
For resolving this issue we add a timeout before starting the test in 1 hour. After this have the next situation:
Next steps:
Like temporary solutions destroying the environment is disabled. So QA team could check the environment after running tests and investigate float issues.
More information about runs:
1 sheet - test running
2 sheet - after Kafka memory resizing