[FOLIO-3672] Jenkins build node broken after OS updates Created: 22/Dec/22  Updated: 23/Jan/23  Resolved: 23/Jan/23

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Bug Priority: TBD
Reporter: John Malconian Assignee: John Malconian
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Sprint: DevOps Sprint 155, DevOps Sprint 157
Development Team: FOLIO DevOps
RCA Group: TBD

 Description   

While investigating another potential issue, OS patches were applied to the jenkins build node called folio-jenkins-slave02. After rebooting the system, jenkins is unable to launch jenkins builds in containers on the node. The error is:

There is insufficient memory for the Java Runtime Environment to continue.


 Comments   
Comment by John Malconian [ 22/Dec/22 ]

Was able to confirm that Java within both the java-11 and java-17 based build containers does not work.

 docker run -it --rm folioci/jenkins-slave-all:java-17 /bin/bash

root@52380d8833db:/home/jenkins# java -version
[0.003s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# /home/jenkins/hs_err_pid10.log

Looks like this issue. https://github.com/adoptium/containers/issues/215#issuecomment-1142046045
The OS updates must have triggered a change with causes this. Probably in glibc or another library as downgrading Docker to the previous version does not resolve the issue. The Jenkins build node is Ubuntu Xenial while the build containers are Jammy. Unfortunately, the last release of Docker on Xenial is 20.10.7 but the fix is in version 20.10.10.

I've implemented a workaround by configuring Jenkins to run docker with the option '--security-opt seccomp=unconfined' on folio-jenkins-slave02. This is not idea and should only be temporary. The best solution is to upgrade the build nodes to Jammy.

Comment by Jakub Skoczen [ 23/Jan/23 ]

Please cratea a ticket to remove the workaround John Malconian

Generated at Thu Feb 08 23:29:51 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.