85 Commits

Author SHA1 Message Date
Mark Goddard
0bb3edcabb Add retries for get_url and package tasks
Occasionally we see connection issues in CI.

Change-Id: I93cc6793cf23fd21462cec6a4deb2ea35d36da21
2023-05-18 09:24:57 +01:00
Mark Goddard
2c2c8b9504 Fix CI failures
* new ansible-lint complaints
* clamp ansible-lint to <7
* tox 4 support
* switch to CentOS NFV Open vSwitch repository for OVS - rocky 9 RDO install was failing
* switch from Kayobe CentOS stream 8 to Rocky Linux 9 and Ubuntu Jammy
  jobs - these are the current supported distros in Kayobe master

Change-Id: Ic607ccccc866da65c100053bfa15ee141a98679e
2023-05-18 09:24:55 +01:00
Mark Goddard
d0ded32016 Follow ups to venv
Follow ups to Icd60aca084323cede944ee261d2cf5941b320e4b.

Change-Id: Id25a8b35ad42ee7f095098c210bc14926aec18ed
2022-10-25 08:05:40 +00:00
Radosław Piliszek
aa922ad2f8 [ansible-lint] Unignore jinja[spacing]
Fix in one place, ignore in the other.

Only 4 warnings now.

Change-Id: I3c745914e2ffdcdd7e70d9e06ae1df05dc120f79
2022-09-10 09:11:28 +00:00
Radosław Piliszek
6c59da3daf Use venv instead of virtualenv
virtualenv is not a part of the core python installation, but
venv is. Let's use it to be slimmer and avoid issues with
virtualenv installation.

At the same time, remove some existing hacks around selinux,
systemd, PyYAML and libvirt.

Change-Id: Icd60aca084323cede944ee261d2cf5941b320e4b
2022-09-09 15:56:21 +00:00
Bartosz Bezak
d92c731aa2 Add Ubuntu 22.04 Jammy support.
CI and docs updated - openstack-tox-py310 and Jammy tenks jobs.

libzmq5 added as a dependency of vbmc/zmq

Change-Id: Ibf0ca87bc86152953650e7ef8384e5c839d7a8bc
2022-07-26 16:57:19 +02:00
Bartosz Bezak
d7312db645 [CI] Drop Python 3.7 job. Fix ansible-lint errors.
Two changes to fix CI:
- Drop Python 3.7 job as Victoria is in extended maintenance now [1][2]
- Name tasks to fix 'unnamed-task' ansible-lint errors

[1]: https://review.opendev.org/c/openstack/governance/+/693743
[2]: https://review.opendev.org/c/openstack/governance/+/743847

Change-Id: I7324467e54deda1a05063cfa1b52a276f7622442
2022-07-26 10:45:19 +02:00
Pierre Riteau
3a64cdc911 Remove Python 2 leftovers
Change-Id: I7f6fa2ea66a092f07605b593154cb89d8d31c0c2
2022-07-05 09:35:05 +02:00
Mark Goddard
e47ff6bd82 Use jinja2.pass_context instead of contextfilter
Jinja2 3.0.0 deprecated contextfilter in favour of pass_context. 3.1.0
dropped contextfilter.

Fall back to contextfilter for Jinja2 2.x.

This change also fixes some issues caused by Ansible lint 6.0. Issues
found by the yaml plugin are fixed, while the FQCN for builtin actions
plugin is skipped.

Change-Id: I97b25551eb26da2c9100120bcd646c88fdb33ba6
2022-03-29 12:36:52 +01:00
Mark Goddard
9111328e8a Use ansible_facts to reference facts
By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Tenks
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Change-Id: I4831769909c341c72bb178daf8df3309773a56ad
Co-Authored-By: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2022-02-23 18:17:43 +00:00
Mark Goddard
fc1b42c60d Support UEFI boot mode
Adds support for using UEFI boot mode for nodes. This is done via node
capabilities, as it is in Ironic.

The default boot mode is now configurable via the default_boot_mode
variable. The default boot mode remains legacy BIOS for now, although
this may change.

Updates the existing CI jobs, with the OVS jobs using BIOS boot mode,
and the linuxbridge jobs using UEFI boot mode.

Depends-On: https://github.com/stackhpc/ansible-role-libvirt-vm/pull/83
Depends-On: https://github.com/stackhpc/ansible-role-libvirt-host/pull/50
Depends-On: https://review.opendev.org/c/openstack/kayobe/+/827486

Change-Id: Ifaf95ecfd4f6e925d3c69d4b324fdf2cd6b0ca52
2022-02-08 10:52:28 +00:00
Mark Goddard
7c6e87e598 Explicitly set boot_mode to legacy BIOS
Since we currently only support legacy BIOS boot mode, set it explicitly
in node capabilities. This is especially important since Ironic changed
the default boot mode to UEFI in Yoga. If the capabilities field is
provided, the boot_mode should be set explicitly.

This change should be followed by one that adds support for UEFI boot
mode.

Change-Id: I0c1e0cc50119cc2f9ba3e0b0a3548baeca9dca2e
2022-02-01 15:06:36 +00:00
Pierre Riteau
5746ed2a27 Fix 'The libvirt module is not importable' error
Upper constraints were updated [1] to allow libvirt-python 7.5.0. The
first Kayobe builds using this version worked fine, but later they
started failing with the error:

    The `libvirt` module is not importable. Check the requirements.

The main difference is that successful jobs were using the source
archive libvirt-python-7.5.0.tar.gz, while failing jobs were using the
wheel libvirt_python-7.5.0-cp36-cp36m-linux_x86_64.whl.

Install libvirt-python from source to avoid this issue.

Update the ansible-lint configuration to ignore errors coming from the
stackhpc.libvirt-vm role, fixing the tenks-tox-ansible-lint job.

[1] https://review.opendev.org/c/openstack/requirements/+/799347

Change-Id: I746d01ed355330c965b90657264d89a4ef7995d2
2021-07-09 15:04:27 +02:00
Pierre Riteau
6f304c81e2 Update pip and setuptools before installing requirements
We are seeing the kayobe-overcloud-ubuntu-focal CI job fail while
deploying Tenks with:

    ModuleNotFoundError: No module named 'pkg_resources.extern.six'

Updating pip and setuptools to the latest allowed in upper constraints
appears to fix it.

Change-Id: Ida85adca948664d7d4427edd6c3ab15fd75fb3fe
2021-05-17 18:13:20 +02:00
Pierre Riteau
f34bb772a0 Fix virtualbmc installation after release of cryptography 3.4
Installing virtualbmc system-wide on CentOS 8 fails with:

    ModuleNotFoundError: No module named 'setuptools_rust'

This error appeared following the release of cryptography 3.4, which now
includes Rust code. It can be installed without Rust using a Python
wheel, but only with more recent pip than version 9.0.3 available as RPM
on CentOS 8. The cryptography bug report [1] recommends pip>=19.1.1.

Also ignore PyYAML when installing system-wide to avoid conflicts with
an existing RPM package installation.

[1] https://github.com/pyca/cryptography/issues/5753

Change-Id: Ibd61e090611b3b7a7e0670c854362b512454bf3c
Story: 2008607
Task: 41788
2021-02-08 15:54:17 +01:00
Riccardo Pittau
996921a5b6 Fix ansiblelint 208
File permissions not mentioned

Change-Id: I5efbb06ccee9ac7e83bfcf4171c5ae7d2930128b
2020-09-01 11:33:13 +02:00
Iury Gregory Melo Ferreira
5abd97149c Py2 - Remove six, mock and future
Since we've dropped support for Python 2.7, it's time to look at
the bright future that Python 3.x will bring and stop forcing
compatibility with older versions.

This patch removes the six library , switches the mock library
to unittest.mock and removes future.

Change-Id: I71b11f13691d13df162b203f7ea5979b30c272df
2020-05-04 15:22:23 +02:00
Riccardo Pittau
ece9f6e25f Add pep8 import order check
Set flake8-import-order version to 0.17.1
Full py3 compatible version.
Add all Python3 modules to stdlib list.
Also includes fix to an enum34 dependency bug.

Change-Id: I58dd80fc33942656c69ce67ee523a1d57f693e93
2020-04-21 16:26:52 +02:00
Mark Goddard
1b01c94f15 Fix issue with VirtualBMC client path
If SELinux is enforcing we install VirtualBMC as a system Python
package. The module is executed using become, and root typically doesn't
have /usr/local/bin/ in it's PATH.

Use an absolute path to resolve this issue.

Change-Id: Ia278518d2c55351c55abca1122e7cad7179ad4d8
2020-04-08 12:43:07 +01:00
Mark Goddard
75273302ed Refactor virtualbmc-domain tasks into a module
Virtualbmc domain creation can be unreliable, in particular when domains
already exist. This is in part due to the vbmc stop command not functioning
properly [1]. By moving the domain management into a Python module we can
better control the process of creation, and improve performance.

[1] https://storyboard.openstack.org/#!/story/2003534

Change-Id: I52cb08cd0d300630cb6341f50eb0484c1d16daa4
2020-04-03 09:34:12 +01:00
Mark Goddard
ced8a4a528 Support linux bridge in addition to OVS
While OVS was used initially due to availability of a
networking-generic-switch driver for it, Linux bridge can be useful for
environments where OVS is not available.

This is configured via bridge_type variable. It defaults to
'openvswitch', but may be set to 'linuxbridge'.

Change-Id: I750a73cebc743edcbcd8c23c67e4920d0058bead
2020-03-13 11:12:37 +00:00
Mark Goddard
042fcb4754 CentOS 8 support
* Support Python 3
* Use Open vSwitch from Delorean master in CI
* Add a tenks-deploy-teardown-centos8 CI job

Depends-On: https://github.com/stackhpc/ansible-role-libvirt-host/pull/26
Depends-On: https://github.com/stackhpc/ansible-role-libvirt-vm/pull/53

Change-Id: Ibd504389523bf1197aa5f7c3e9ef6cad8966ddc2
2020-03-11 17:05:21 +00:00
Mark Goddard
512174a828 Bump ansible version constraints to >=2.7,<2.10
This means that ansible 2.9 is typically used, and causes ansible-lint
to fail. Two workarounds have been added to address this:

* Use .get() to avoid passing an undefined object through to_nice_yaml
  filter
* Create a dummy tenks_update_state.py module to avoid ansible-lint
  being confused about our action plugin.

Change-Id: I54388f713e17a47601be2db63a26fc43ca7ac6c8
2020-03-11 16:55:09 +00:00
Mark Goddard
3db82fd551 Pin virtualbmc to <2 when using Python 2
This is necessary until we drop support for CentOS 7.

Also use Train upper constraints with Python 2, since Train is the last
release to support Python 2.

Change-Id: I085d107ae2d338a7609dececb7b62555937b7b1e
2020-03-11 16:38:28 +00:00
Mark Goddard
13de0edd6c Install virtualbmc to system when SELinux is enabled
CentOS 8 nodepool images come with SELinux enabled, and prevent systemd
from accessing files in users' home directories. This prevents vbmcd
from starting.

To resolve this, install virtualbmc to the system when SELinux is
enabled.

Change-Id: I5d30aca2b1e1ae897568f6e11fceda39e0b06e17
2020-02-27 15:47:44 +00:00
zhufl
b783f1c66d Add missing ws between words
This is to add missing white space between words.

Change-Id: Iea4cfde0143acbb4cdd15f5b6b5e81dbe093fea4
2019-11-15 15:24:40 +08:00
Will Szumski
c7ec7909c2 Add debugging statements to wait for resources loop
When there is a timeout waiting for resources to become available,
it can be diffciult to determine why this happened. This should
aid in diagnosing the issue.

Change-Id: Id2cb5202a54c03b5e23316aa1d0874072e4e2efe
2019-09-20 15:57:40 +01:00
Mark Goddard
26fcfd5c74 Ignore traits with a COMPUTE_ prefix in wait for placement
Traits with a COMPUTE_ prefix are for signalling driver capabilities in
nova. These may be expected to change over time, so ignore them all.

Change-Id: I8cb667bc456e88abad0283970e69bb1cdb1cd249
Story: 2005675
Task: 30986
2019-05-14 15:35:39 +01:00
Mark Goddard
bda7320d6e Ignore COMPUTE_NET_ATTACH_INTERFACE trait when waiting for placement
As of Stein, nova adds a 'COMPUTE_NET_ATTACH_INTERFACE' trait to ironic
resource providers in placement. This means that they no longer match our
expected set of traits.

This patch removes this trait from the list of expected traits.

Change-Id: I84bf367d31b10144617c0d66cc21dbcb5b0908a8
Story: 2005309
Task: 30210
2019-03-27 16:54:24 +00:00
Will Szumski
8e2f8a3ecb Fix ansible lint and ubuntu deployment
This patch combines two different fixes, since both are required to get
CI tests passing.

1. Fix ansible lint 301 with ip link set dev <device> up

This resolves:

[301] Commands should not change things if nothing needs doing
/home/will/code/tenks/ansible/roles/veth-pair/tasks/present.yml:13
Task/Handler: Bring each end of veth up

ip link set dev <device> up is idempotent:

707f612c00/ip/iplink.c (L264)
63bdf4284c/net/core/rtnetlink.c (L2485)
63bdf4284c/net/core/dev.c (L7632)
63bdf4284c/net/core/dev.c (L7563)

2. Add pkg-config as an explicit dependency

This is to fix the following error:

Exception: pkg-config binary is required to compile libvirt-python

TrivialFix

Change-Id: I9a2db7b5f2cbdcb64227bd4f1de245bcb7f71f5b
2019-03-12 09:18:10 +00:00
Zuul
529824e01d Merge "Support specifying deploy images via URL" 2019-02-01 13:18:37 +00:00
Mark Goddard
a04f607c37 Support specifying deploy images via URL
In standalone environments we might not have Glance, and will therefore access
images via a URL instead.

Change-Id: I0f764545788775876d0a5751621a51602e1ad5ac
2019-02-01 11:00:36 +00:00
Mark Goddard
c58a22a2b1 Default Ironic node properties are bogus
Unless you add a 'properties' field under the 'ironic_config' for a node spec,
ironic nodes are registered with silly default properties (originating in the
os_ironic module) which cause validation to fail:

cpus:1, memory_mb: 1, local_gb: 1

We have enough info to add sensible defaults based on the VM configuration, so
let's do that.

Change-Id: I43fbe726d1d787d96be811f3ecd4234ea3b6bac3
Story: 2004908
Task: 29258
2019-01-31 15:00:51 +00:00
Mark Goddard
7ad354c2af Use node state rather than 'cmd'
There are two cases when we may want to delete things:

1. if a desired node configuration or spec changes, we delete existing node
   resources.
2. if we use the teardown.yml playbook, cmd == teardown, delete
   everything, including global state.

In both cases, Tenks state (state.yml), all nodes to be deleted are given a
'state' of absent'. We should therefore use this rather than the 'cmd' variable
which only works in the latter case.

Change-Id: Icc033340c7fd882e61d90e3d086f7ba4a5b673bf
Story: 2004907
Task: 29257
2019-01-31 14:59:47 +00:00
Mark Goddard
ff3a73f37d Check for specific error messages in ironic version check
When checking for the 1.34 ironic API version for physnet support, explicitly
check for known error messages to make it more robust.

Change-Id: I8c42e85c8b3abe8712879c5737d85ee72e231f9e
2018-12-20 15:47:04 +00:00
Mark Goddard
d28635d2cf Changes to support standalone ironic
For standalone ironic deployments, such as those deployed using bifrost, there
is typically no keystone or placement service running. Use of the ironic API is
without authentication. We use clouds.yaml in this case.

Change-Id: I61ed791d019e43f6dfc632fa185f079b385c290c
Story: 2004425
Task: 28077
2018-12-20 09:48:21 +00:00
Mark Goddard
1a0d953737 Support ocata release of ironic
In the Ocata release, the physical_network field of ports did not exist.
The safest thing to do is to simply ignore it in this case.

It is also possible for the local_link_connection fields to not have switch_id,
switch_info or port_id attributes, so we now use .get(attr) to avoid an
attribute lookup error.

Change-Id: Ib4829dbd89fcedc121f493d6af76702421b399f7
TrivialFix
2018-12-19 18:55:36 +00:00
Mark Goddard
a30ac7de48 Fix ansible-lint nits
Ansible lint has become more fussy about its nit picking, blocking CI.

This change makes it happy again.

Change-Id: Ifb86c5e071697acda014bb543cab7ebf2b648dc9
2018-12-19 18:55:36 +00:00
Mark Goddard
347ee1cc7f Fix skipping glance check if deploy images not defined
Somewhere along the line, the default None value for deploy_kernel and
deploy_ramdisk gets converted to a string. Allow for both possibilities.

Change-Id: Ia06bff344262f82956da1d6548bae9a30fcdffd9
TrivialFix
2018-11-26 10:43:57 +00:00
Mark Goddard
1e2bd2cc2f Fail nicely if ironic port lookup fails
Change-Id: I7e1a937b2870af08a6748adf9dcfa71d49be9957
TrivialFix
2018-11-26 10:38:29 +00:00
Mark Goddard
6a2fb44a78 Fix deploy and teardown without node enrolment
If the ironic_config field is not present in a node definition, then
that node should not be registered with ironic. In this case we also
don't need to query glance for images or wait for resources to appear
in placement.

If nova_flavors is empty, we do not need to register flavors, so no need
to install the client or check for OS_USERNAME.

Also fixes an issue in the action plugin where it would fail if
ironic_config field is not present in a node definition.

Finally, in CI deploy/teardown tests, rather than skipping tasks, we now
remove the ironic_config field from the node definition, and set
nova_flavors to empty, such that we exercise these code paths.

Change-Id: I29b42665f1609c99e80f12ab6f3815d0e0918dae
Story: 2004412
Task: 28056
2018-11-22 19:20:59 +00:00
Zuul
6545bf3fa2 Merge "Add support for setting the libvirt connection uri" 2018-11-20 17:17:38 +00:00
Will Szumski
ffcdee61dd Add support for setting the libvirt connection uri
This, for example, allows you to run libvirt with non-default
socket path.

Change-Id: Ia01f39f425cccc6acebd31f77b46b1948e24d215
Story: 2004344
Task: 27935
2018-11-19 17:41:01 +00:00
Will Szumski
d4e7e58619 Wait for resources to become available
This will wait for any resources that are created to become available
before exiting the script. This allows you to avoid a race condition
where a server could be created before the resource tracker had been
updated with the new resources; server creation would fail.

Change-Id: I57f8c93cb1ebbc284b96ef1ced2c4edd59b27795
Story: 2004274
Task: 27823
Depends-On: https://review.openstack.org/617642
2018-11-19 16:40:43 +00:00
Will Szumski
602fdfa26b Remove dependency on bridge-utils
bridge-utils is considered legacy and many distributions do
not ship it as default.

Change-Id: Id2f9d779d12a5135d7d7e7a4783c6f3a0c8647ca
Story: 2004302
Task: 27861
2018-11-08 15:15:30 +00:00
Will Szumski
a5e1b65756 Only set Ironic port attributes when changed
This fixes a lint issue:

[301] Commands should not change things if nothing needs doing
/home/will/code/tenks/ansible/roles/ironic-enrolment/tasks/port.yml:20
Task/Handler: Set Ironic port attributes

TrivialFix

Change-Id: I7940fe25d3518daf268484e292e337abc925080d
2018-11-02 17:27:10 +00:00
jovial
7a04432226
Merge pull request #25 from jovial/usr-bin
Use /usr/bin/virsh instead of /bin/virsh
2018-10-03 17:40:30 +01:00
Will Szumski
bc66151567 Use /usr/bin/virsh instead of /bin/virsh
On RedHat based distros /bin/ is a symlink to /usr/bin:

[will@dev-director ~]$ ls -lia /bin
410146 lrwxrwxrwx. 1 root root 7 Jun 11 12:07 /bin -> usr/bin

Whereas on the Ubuntu it is seperate directory. Using
/usr/bin/virsh should work on both.

ubuntu:

stack@ubuntu:~/ansible-role-libvirt-vm$ whereis virsh
virsh: /usr/bin/virsh /usr/share/man/man1/virsh.1

centos:

[will@dev-director ~]$ whereis virsh
virsh: /usr/bin/virsh /usr/share/man/man1/virsh.1.gz

Also removed a misleading comment. QEMU uses differnt
"instances" for privaleged and user access i.e the
"system" and "session" instances. The comment seemed
to suggest using /bin/virsh would connect to the
system instance (at least that is how I read it).
2018-10-01 13:47:18 +01:00
Will Szumski
7ef64ff974 Support Debian based GNU/Linux distributions
- use package module instead of yum
- libvirt-devel is called libvirt-dev on debian based
  distributions
2018-10-01 09:49:34 +01:00
Mark Goddard
14e508588b
Merge pull request #18 from w-miller/veth-up
Bring each end of veth pair up after creation
2018-09-21 17:58:33 +01:00