[FOLIO-987] Packer builds failing on folio-jenkins-de Created: 13/Dec/17 Updated: 12/Nov/18 Resolved: 10/Jan/18 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | Continuous Integration |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Bug | Priority: | P2 |
| Reporter: | Wayne Schneider | Assignee: | Wayne Schneider |
| Resolution: | Done | Votes: | 0 |
| Labels: | ci, sprint28, sprint29 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | 1 day, 3 hours | ||
| Original estimate: | Not Specified | ||
| Sprint: |
| Description |
|
The folio-blackbox packer builds are failing regularly on folio-jenkins-de. It appears to be network related, but it is hard to diagnose. I get the same behavior running Vagrant on folio-jenkins-de; I don't see the issue locally. |
| Comments |
| Comment by Wayne Schneider [ 13/Dec/17 ] |
|
Tried refreshing system software and rebooting, didn't seem to help. |
| Comment by Wayne Schneider [ 13/Dec/17 ] |
|
Bumped up some timeouts in folio-ansible, which may have helped. Refactored the job in Jenkins to build boxes sequentially, instead of in parallel – which misses some of the point of Packer, but might help with network I/O and memory consumption on folio-jenkins-de |
| Comment by Wayne Schneider [ 14/Dec/17 ] |
|
Reverted changes to folio-blackbox Jenkins job (they didn't help). Reverted docker daemon configuration change. Current strategy – allow nightly builds to fail, build locally if necessary. Explore other bare metal options like packet.net. |
| Comment by John Malconian [ 15/Dec/17 ] |
|
Reconfigured Jenkins job to use a new slave node hosted by Packet. Preliminary job runs look promising. Let's continue to monitor for a few days. |
| Comment by Wayne Schneider [ 10/Jan/18 ] |
|
Have migrated jobs to folio-jenkins-slave01 on packet.net |