[FOLIO-3056] Capture heap dump on OOM for mod-password-validator in reference environments Created: 04/Mar/21 Updated: 18/Mar/21 Resolved: 18/Mar/21 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | P2 |
| Reporter: | Wayne Schneider | Assignee: | Wayne Schneider |
| Resolution: | Won't Do | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||
| Sprint: | DevOps Sprint 110 | ||||||||
| Development Team: | FOLIO DevOps | ||||||||
| Description |
|
mod-password-validator has been crashing fairly regularly on the reference environments. docker inspect shows that it is OOM-killed, which means that it exceeded the memory limit configured in the launch descriptor. It's not clear what could cause this. The team has asked us to try to capture heap dumps for the module. We should be able to do this by adding the -XX:+HeapDumpOnOutOfMemoryError switch to the java command line for the container. |
| Comments |
| Comment by Wayne Schneider [ 16/Mar/21 ] |
|
Added -XX:+HeapDumpOnOutOfMemoryError to JAVA_OPTIONS for mod-password-validator for snapshot/testing builds. Full command line:
exec java -XX:MaxRAMPercentage=66.0 -XX:+HeapDumpOnOutOfMemoryError -Dspring.datasource.username=folio_admin -Dspring.datasource.password=folio_admin5757 -Dspring.datasource.url=jdbc:postgresql://10.36.1.142:5432/okapi_modules -Dspring.datasource.username=folio_admin -Dspring.datasource.password=folio_admin5757 -Dspring.datasource.url=jdbc:postgresql://10.36.1.142:5432/okapi_modules -XX:+ExitOnOutOfMemoryError -cp . -jar /usr/verticles/mod-password-validator-fat.jar
This change has been tested and merged, and should take effect in the next reference environment bulds. |
| Comment by Wayne Schneider [ 17/Mar/21 ] |
|
This update is in the reference environments, however, when I checked folio-snapshot, mod-password-validator had been OOM-killed without generating a heap dump. I posted on the Slack #development channel for some help. If there isn't a good way to capture a heap dump when the module is OOM-killed, Sobha Duvvuri, you may need to look at other strategies for doing the analysis required for
|
| Comment by Wayne Schneider [ 18/Mar/21 ] |
|
I've reverted the commit and am closing this issue for now. |