[FOLIO-3056] Capture heap dump on OOM for mod-password-validator in reference environments Created: 04/Mar/21  Updated: 18/Mar/21  Resolved: 18/Mar/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Wayne Schneider Assignee: Wayne Schneider
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Relates
relates to MODPWD-58 Analyze memory consumption of module Closed
Sprint: DevOps Sprint 110
Development Team: FOLIO DevOps

 Description   

mod-password-validator has been crashing fairly regularly on the reference environments. docker inspect shows that it is OOM-killed, which means that it exceeded the memory limit configured in the launch descriptor. It's not clear what could cause this.

The team has asked us to try to capture heap dumps for the module. We should be able to do this by adding the -XX:+HeapDumpOnOutOfMemoryError switch to the java command line for the container.



 Comments   
Comment by Wayne Schneider [ 16/Mar/21 ]

Added -XX:+HeapDumpOnOutOfMemoryError to JAVA_OPTIONS for mod-password-validator for snapshot/testing builds. Full command line:

exec java -XX:MaxRAMPercentage=66.0 -XX:+HeapDumpOnOutOfMemoryError -Dspring.datasource.username=folio_admin -Dspring.datasource.password=folio_admin5757 -Dspring.datasource.url=jdbc:postgresql://10.36.1.142:5432/okapi_modules -Dspring.datasource.username=folio_admin -Dspring.datasource.password=folio_admin5757 -Dspring.datasource.url=jdbc:postgresql://10.36.1.142:5432/okapi_modules -XX:+ExitOnOutOfMemoryError -cp . -jar /usr/verticles/mod-password-validator-fat.jar

This change has been tested and merged, and should take effect in the next reference environment bulds.

Comment by Wayne Schneider [ 17/Mar/21 ]

This update is in the reference environments, however, when I checked folio-snapshot, mod-password-validator had been OOM-killed without generating a heap dump.

I posted on the Slack #development channel for some help. If there isn't a good way to capture a heap dump when the module is OOM-killed, Sobha Duvvuri, you may need to look at other strategies for doing the analysis required for MODPWD-58 Closed .

Comment by Wayne Schneider [ 18/Mar/21 ]

I've reverted the commit and am closing this issue for now.

Generated at Thu Feb 08 23:25:17 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.