[FOLIO-2731] Update reference build OS to Ubuntu Focal and OpenJDK 11 Created: 17/Aug/20  Updated: 04/Sep/20  Resolved: 04/Sep/20

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: TBD
Reporter: John Malconian Assignee: Wayne Schneider
Resolution: Done Votes: 0
Labels: devops-backlog
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Relates
relates to FOLIO-2730 New Jenkins release pipeline for Okap... Closed
Sprint: DevOps: Sprint 95, DevOps: Sprint 96
Development Team: FOLIO DevOps

 Description   

Existing reference builds install Ubuntu Xenial and OpenJDK 8 and the Okapi Debian package relies on the locally installed OpenJDK. Okapi 4.x requires OpenJDK 11. This is good time to update the reference OS to Focal and OpenJDK 11.



 Comments   
Comment by Wayne Schneider [ 18/Aug/20 ]

We will also need to update the base image for the Vagrant build.

Comment by Wayne Schneider [ 01/Sep/20 ]

This turns out to be a bit of a rabbit hole.

First attempts to run a folio-snapshot build resulted in an error:

TASK [common : create folio group] *********************************************
fatal: [10.36.1.111]: FAILED! => {"changed": false, "module_stderr": "<stdin>:17: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\nTraceback (most recent call last):\n  File \"/tmp/ansible_group_payload_jimnbqyf/ansible_group_payload.zip/ansible/module_utils/basic.py\", line 279, in get_distribution\nAttributeError: module 'platform' has no attribute '_supported_dists'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"<stdin>\", line 113, in <module>\n  File \"<stdin>\", line 105, in _ansiballz_main\n  File \"<stdin>\", line 48, in invoke_module\n  File \"/usr/lib/python3.8/imp.py\", line 234, in load_module\n    return load_source(name, filename, file)\n  File \"/usr/lib/python3.8/imp.py\", line 169, in load_source\n    module = _exec(spec, sys.modules[name])\n  File \"<frozen importlib._bootstrap>\", line 604, in _exec\n  File \"<frozen importlib._bootstrap_external>\", line 783, in exec_module\n  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n  File \"/tmp/ansible_group_payload_jimnbqyf/__main__.py\", line 501, in <module>\n  File \"/tmp/ansible_group_payload_jimnbqyf/__main__.py\", line 449, in main\n  File \"/tmp/ansible_group_payload_jimnbqyf/__main__.py\", line 89, in __new__\n  File \"/tmp/ansible_group_payload_jimnbqyf/ansible_group_payload.zip/ansible/module_utils/basic.py\", line 337, in load_platform_subclass\n  File \"/tmp/ansible_group_payload_jimnbqyf/ansible_group_payload.zip/ansible/module_utils/basic.py\", line 289, in get_distribution\nAttributeError: module 'platform' has no attribute 'dist'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
	to retry, use: --limit @*****-snapshot-test/folio-infrastructure/CI/ansible/folio-snapshot-test-builds.retry

Research indicated that the issue may be that we need to use Python 3 for Ansible on both the control host (jenkins-slave-all) and the target host.

The java-11 tagged version of folioci/jenkins-slave-all is based on Focal and uses Python 3 for Ansible (thanks, Ian Hardy). Unfortunately, the ec2.py dynamic inventory script fails with:

 [WARNING]:  * Failed to parse *****-
snapshot-test/folio-infrastructure/CI/ansible/inventory/ec2.py with script
plugin: Inventory script (*****-snapshot-
test/folio-infrastructure/CI/ansible/inventory/ec2.py) had an execution error:
/usr/bin/env: ‘python’: No such file or directory

This is because Focal does not include Python 2.x by default, and the executable for Python 3.x is not python.

Currently attempting to rebuild jenkins-slave-all image with the python-is-python3 package.

Comment by Ian Hardy [ 02/Sep/20 ]

Hi Wayne, wonder if its worth updating the ec2.py script. Here's the version that comes w/ansibel 2.7 stable https://raw.githubusercontent.com/ansible/ansible/stable-2.7/contrib/inventory/ec2.py. Assuming python-is-python3 is some kind of symlink which is probably more globally helpful, but could also change the shebang to python3.

Comment by Wayne Schneider [ 02/Sep/20 ]

Ian Hardy thanks, updating the ec2.py script was my next thing to try. The latest run threw a new error, however:

TASK [launch_ec2_instance : launch ec2 instance] *******************************
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'image_id'\n\nThe error appears to have been in '*****-snapshot-test/folio-infrastructure/CI/ansible/roles/launch_ec2_instance/tasks/main.yml': line 3, column 4, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: launch ec2 instance\n   ^ here\n"}
	to retry, use: --limit @*****-snapshot-test/folio-infrastructure/CI/ansible/folio-snapshot-test-builds.retry
Comment by Ian Hardy [ 02/Sep/20 ]

Maybe this is empty? https://github.com/folio-org-priv/folio-infrastructure/blob/master/CI/ansible/launch_ec2.yml#L36

Looking at the role a bit you can pass an ami id to either image_id or image.

Comment by Wayne Schneider [ 02/Sep/20 ]

Something doesn't make sense:

TASK [debug] *******************************************************************
ok: [localhost] => {
    "latest_ami": "{'architecture': 'x86_64', 'creation_date': '2020-08-18T17:26:37.000Z', 'image_id': 'ami-05cf2c352da0bfb2e', 'image_location': '099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20200817', 'image_type': 'machine', 'public': True, 'owner_id': '099720109477', 'platform_details': 'Linux/UNIX', 'usage_operation': 'RunInstances', 'state': 'available', 'block_device_mappings': [{'device_name': '/dev/sda1', 'ebs': {'delete_on_termination': True, 'snapshot_id': 'snap-02e73e95ff3e474e4', 'volume_size': 8, 'volume_type': 'gp2', 'encrypted': False}}, {'device_name': '/dev/sdb', 'virtual_name': 'ephemeral0'}, {'device_name': '/dev/sdc', 'virtual_name': 'ephemeral1'}], 'description': 'Canonical, Ubuntu, 20.04 LTS, amd64 focal image build on 2020-08-17', 'ena_support': True, 'hypervisor': 'xen', 'name': 'ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20200817', 'root_device_name': '/dev/sda1', 'root_device_type': 'ebs', 'sriov_net_support': 'simple', 'virtualization_type': 'hvm', 'tags': {}}"
}

TASK [debug] *******************************************************************
ok: [localhost] => {
    "latest_ami.image_id": "VARIABLE IS NOT DEFINED!"
}
Comment by Ian Hardy [ 02/Sep/20 ]

wat.

My only thought is latest_ami is being treated as a string and not a dictionary.

Comment by Wayne Schneider [ 03/Sep/20 ]

Upgrading to Ansible v2.9.13 in the jenkins-slave-all container resulted in a successful test build.

Warnings:

[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details
[DEPRECATION WARNING]: The 'ec2_ami_facts' module has been renamed to 
'ec2_ami_info'. This feature will be removed in version 2.13.
[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 
'docker_compose'.. This feature will be removed in version 2.12.

So I think no emergencies (whew).

Comment by Wayne Schneider [ 03/Sep/20 ]

Changes merged into master of folio-infrastructure and folio-tools, new Docker image available as folioci/jenkins-slave-all:java-11. folio-snapshot and folio-testing builds have been testing. Putting issue "In Review" for builds to run overnight.

Comment by Wayne Schneider [ 04/Sep/20 ]

Builds look good, closing the issue.

Generated at Thu Feb 08 23:22:49 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.