Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device.
Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
During running tests, Kafka pod restarts several times before and during tests. Pod restarts with OOMKilled status.
After resizing limit memory to 4096M and Xmx parameters to 2662m several modules show more stable result:
mod-orders
mod-circulation
mod-ebsconet
edge-inn-reach
Not stable modules
After resizing memory for Kafka several modules have float issues:
data-import
mod-kb-ebsco-java
mod-quick-marc
More popular issues
A lot of issues look like this: https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/322/cucumber-html-reports/report-feature_138_1684211029.html https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/319/cucumber-html-reports/report-feature_141_2494525229.html https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/312/cucumber-html-reports/report-feature_138_1684211029.html And issues that I don`t understand https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/324/cucumber-html-reports/report-feature_138_1684211029.html
Not all tests start due to issues: https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/326/cucumber-html-reports/report-feature_203_1387567880.html https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/322/cucumber-html-reports/report-feature_203_1387567880.html https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/319/cucumber-html-reports/report-feature_203_1387567880.html https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/318/cucumber-html-reports/report-feature_203_1387567880.html
Timeout issue: https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/322/cucumber-html-reports/report-feature_242_3577100403.html Different status codes: https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/334/cucumber-html-reports/report-feature_226_3577100403.html https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/326/cucumber-html-reports/report-feature_242_3577100403.html https://jenkins-aws.indexdata.com/job/Testing/job/Karate%20tests/317/cucumber-html-reports/report-feature_242_3577100403.html
2 root cause: Nodes large load
Before running the test adds new nodes to the cluster. First time on some "old" nodes have large load:
For resolving this issue we add a timeout before starting the test in 1 hour. After this have the next situation:
Next steps:
Like temporary solutions destroying the environment is disabled. So QA team could check the environment after running tests and investigate float issues.
