201 Commits

Author SHA1 Message Date
Dmitry Tantsur
27d87522f1 install-deps: install setuptools early for Debian
Seems like bindep cannot work without it but does not pull it for
some reason.

Change-Id: I6c989e2aa8e71ca2fa802ec403d34612262eef2f
2020-06-22 16:32:15 +02:00
Iury Gregory Melo Ferreira
fa1b8ea9a0 Add openstack collection
- add ansible-collection-requirements.yml so we
can specify the collections we need.
- Changed env-setup to install collections unsing
ansible-galaxy.

- Moved taks "Dynamic enrollment" to use the openstack.cloud
collection

Change-Id: I1cf21a505b923f333b6853944965593170cbe8cd
2020-06-19 10:36:53 +02:00
Dmitry Tantsur
12c6f7120f Quick start Bifrost CLI
This change adds a local CLI ./bifrost-cli that allows to easily
run bifrost playbooks. It is targeting early adopters and thus
is opinionated and does not expose all possible options.

Only the very minimum is provided in this patch. More options
will be added later as we decide they are important.

Documentation will also be provided later as it's going to be
quite large. This feature should be considered technical preview
at this point until we give it more testing.

Change-Id: I2205e759431024124518716eccd07f79bda14f3a
2020-06-10 15:03:47 +02:00
Zuul
bd4a22f682 Merge "Moving to ansible 2.9" 2020-06-05 12:51:39 +00:00
Iury Gregory Melo Ferreira
9c9c014908 Moving to ansible 2.9
Some code changes were necessary since the Ansible-provided jinja tests
as filters was removed in ansible 2.9, see [1]

[1] https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.5.html#jinja-tests-used-as-filters

Co-authored-by: Kaio Oliveira <kaio.kassiano.oliveira@gmail.com>
Change-Id: I2f1c2ece0543eb60ac5ac5f9ad1465ffd2d66592
2020-06-05 12:34:22 +02:00
Dmitry Tantsur
8aa8086ab2 Switch to using virtual envs by default
They're already used in the CI, and chances are high that non-venv installation
is broken on at least some supported systems.

Change-Id: Ie3c93a0a1537658eaccf69da3f64d0fc43f292d1
2020-06-05 11:46:04 +02:00
Dmitry Tantsur
7e363d4318 Drop hard requirement on debootstrap and EPEL
EPEL is only needed for debootstrap, debootstrap is only needed when
building Debian/Ubuntu images. Install it on demand and update the docs.

Change-Id: I6b90c91f20b5acd198cabbb89f21a72caaf90c63
2020-06-03 16:41:04 +02:00
Dmitry Tantsur
85e1b82078 CI: reduce memory and disk for testing nodes
There is not reason to use 10 or 12G of disk for Cirros.

Also change the RAM to 512M, although we'll have to revert it back
on master and ussuri once we switch to DIB images by default.

Change-Id: Ibada78686ffb4db1d8dae05acf980081d5265fc6
2020-06-03 11:49:12 +02:00
Zuul
69818cfb0c Merge "Stop installing openstackclient if keystone is not enabled" 2020-06-02 09:02:26 +00:00
Zuul
f95bd5e601 Merge "Fix pip3 path" 2020-06-02 00:08:38 +00:00
Zuul
ad3e3df582 Merge "Fix check packages command" 2020-06-02 00:08:37 +00:00
Dmitry Tantsur
d8ff9ffad4 CI: move the libvirt storage pool to /opt
The default location is too small on some testing nodes.

Removed the MySQL configuration code that has never been executed
because of the non-existing ZUUL_REF variable.

Change-Id: I5fbe2de5423486fdc81d67d4c4bd6d6f3ed79f13
2020-05-28 16:26:09 +02:00
Riccardo Pittau
5182388b7c Fix pip3 path
The pip3 binary can be in a different location from python3.

Change-Id: I19f32c0a8316dc0fca452b7e8d36c777effcbeb5
Story: 2007694
Task: 39806
2020-05-28 14:39:22 +02:00
Riccardo Pittau
d75a07ccb2 Fix check packages command
The dpkg -l command always exits with 0, we should use the -s
option instead that exits with 1 if the package is not installed.

Change-Id: I656b111b77b637a10c9a94fd3880aee377c166f1
2020-05-28 14:39:22 +02:00
Dmitry Tantsur
a381b0f9c0 install-deps: use system pip and bindep for installing dependencies
Updating pip was required long ago to work around Ubuntu and CentOS 7
problems. Now even CentOS 8 has pip 9.0 which should be enough for us.

Also moving binary dependencies to bindep and install them before
we try to install Python dependencies (to be able to compile them).

Workarounds is needed for segfault on Ubuntu because of cryptography:
remove python3-cryptography package and avoid --ignore-installed.

This change makes install-deps work on openSUSE again.

Change-Id: Ib9b81075f35068c046880de46b20f98ef71a8d8e
2020-05-26 12:17:53 +02:00
Dmitry Tantsur
0b7fd20e9e Stop installing openstackclient if keystone is not enabled
We have a standalone command now, let's use it.

Change-Id: I446d7b490d4a14ede01384fd72926f85660ea6e7
2020-05-25 12:38:33 +02:00
Riccardo Pittau
cc415e590e Install python3-pip system package
Without the python3-pip package, installation on ubuntu bionic
without virtualenv will fail.

Change-Id: I666879c06b119920b1e925c1e2f8848caa6cead4
Story: 2007691
Task: 39803
2020-05-19 14:59:56 +02:00
Riccardo Pittau
80022d9cd4 No need to import print from future anymore
The future is now!

Change-Id: Ia219a23da0b468bb85866cfdbbe07b57de1ef102
2020-04-23 10:24:48 +02:00
Riccardo Pittau
a47c553e05 Fix dhcp job
Change-Id: Ibb1d32ae8b69e9ad47db55d455fb3d7f0cfc4219
2020-03-20 10:06:56 +01:00
Mark Goddard
21cb4ea6fc Remove --ignore-installed from CentOS 8 pip 19 install
In the kolla bifrost container image this results in a broken pip
installation. The pip 19 package is downloaded, but this message is
printed at the end:

Successfully installed pip-20.0.2

The subsequent 'pip freeze' gets the following error:

ImportError: cannot import name 'PackageFinder'

This change removes the --ignore-installed argument, which results in
the virtualenv getting pip 19 as expected.

Change-Id: Idf2b03a4772111f63f79203f63ea95f805e7a43d
2020-03-17 18:13:42 +00:00
Riccardo Pittau
19b9f685b5 Fix for PEP517 issue with Python 3.6.8
When creating a virtualenv with Python 3.6.8 using the
--system-site-packages option and pip>=19.0.0 a bug affects
the virtualenv breaking pep build isolation.
This patch downgrade pip to version 19.0.0 as a
workaround.

Also converts pip3 command to pip.

See [1] for more info.

[1] https://github.com/pypa/pip/issues/6264

Change-Id: I738d7c2e50914a773920e5381305e8c5131ccd5f
2020-03-13 14:24:56 +01:00
Riccardo Pittau
1d2ab2b841 Fix and add logs and info
This patch fixes collection of logs adding sudo to commands that
need that.
Also adds collection of some more useful info for libvirt
and qemu, for example increasing the log verbosity of libvirt.

Reordered commands to divide them by section.

Change-Id: I86ae0dac8de63449405012a99ddf94f915b304b0
2020-03-11 10:37:07 +01:00
Riccardo Pittau
5f9ba09c5e Make centos8 job vote again
Giving centos8 tinyipa vote rights again.

Fixing up some of the test scripts for CentOS 8.

- kvm_stats util has been moved to kernel-tools, the package
  qemu-kvm-tools doesn't exist anymore
- qemu-system-x86 doesn't exist anymore, therefore we run
  /usr/libexec/qemu-kvm instead of /bin/qemu-system-x86_64
- installing package trousers to add tss user and group
  needed by libvirt
- Stop setting CPU mode - allow libvirt to select. host-model was failing on
  libvirt 4.5.0 on CentOS 8 due to
  https://bugzilla.redhat.com/show_bug.cgi?id=1804224

Change-Id: I9488dcb8af70ea06de37ffec94620f8681c21944
2020-03-10 15:32:45 +00:00
Zuul
c0b3e45658 Merge "Remove unused check_get_module function" 2020-03-09 16:16:42 +00:00
Zuul
786fadd647 Merge "Explicitely set ansible version to use" 2020-03-09 16:02:07 +00:00
Zuul
f259b6ce3f Merge "Prepare to switch CentOS 8 job voting again" 2020-03-09 15:04:17 +00:00
Mark Goddard
6513e5a6dc Run vbmcd as a Systemd service
Virtualbmc 2.0 dropped the daemon autostart feature. This means we need
to explicitly start it. Running it as a Systemd unit, code adapted from
Tenks.

In order to allow Systemd to execute the daemon on systems with SELinux
enabled, it must be installed to the system rather than a virtualenv.

Change-Id: I66e8e5f5a9c16ac90e132074c826d508458ae4d1
2020-03-09 13:42:44 +00:00
Riccardo Pittau
0c3baf04dd Prepare to switch CentOS 8 job voting again
Fixing things in scripts and playbooks in preparation of
making CentOS 8 tinyipa job vote again.

- python packages are not defined anymore
- change yum to dnf when needed
- change python2 packages to equal python3 when possible
- ipxe binary has changed to ipxe-x86_64.efi

Change-Id: I89989b6d2e35f0291ba1c1aed8b3317cfe2d691d
2020-03-06 13:58:25 +00:00
Riccardo Pittau
eca5ce61b0 Remove unused check_get_module function
The check_get_module is not needed anymore.

Change-Id: I191fc04c8d56b34948118828315574a47375eb34
2020-03-06 14:50:33 +01:00
Riccardo Pittau
adf964481e Explicitely set ansible version to use
Remove the outdated ansible-pip-str.py and rely directly on
ANSIBLE_PIP_VERSION to set the installed version of ansible.

Change-Id: I105b85595cfea101bdb747113721536f4a3ef93a
2020-03-06 14:49:24 +01:00
Zuul
44dc37a6c2 Merge "Fix "unbound variable" error in scripts" 2020-03-06 10:27:54 +00:00
Riccardo Pittau
fdd5fbe2a1 Fix bash syntax in test-bifrost script
Square brackets are redundant and can deal to wrong output.
Also the test can be simplified using -a and removing -n for the
same outcome.

Change-Id: I71a07193e44e7cee402a5c28b955d5da315200b6
2020-02-06 17:11:07 +01:00
Georgy Karataev
c8c4b8c90f Fix "unbound variable" error in scripts
If user runs scripts without VENV defined, then an error
"unbound variable" will be received. This patch fixes this issue

Change-Id: I22b817ad7b945509bb2e89d0467df53975434313
2020-02-05 21:04:52 +05:00
Dmitry Tantsur
6da10694af Check out global requirements when creating test VMs
As part of the test VM preparation, we need to use upper-constraints.

Change-Id: I5aaab91f0590c49972e5eb03d1c70559698b2f39
2020-01-28 17:17:19 +01:00
Zuul
fdeefa810f Merge "Collect output of pip freeze in the CI" 2020-01-28 00:29:30 +00:00
Zuul
01b33d4ae7 Merge "Fix variable checks in install-deps" 2020-01-28 00:25:56 +00:00
Riccardo Pittau
c0d5c4ccb0 Fix variable checks in install-deps
A couple of fixes:
- We check the content of the binary dir as debug step, but this
should be necessary only when in virtualenv.
Also using the correct virtualenv path instead of hardcoding it,
and removing - character after variable name.
- EXTRA_PKG_DEPS is an array and we should treat it like that
when we check if it's empty or not.

Change-Id: I294ef9dce53d7404f4d9285506f1504afabe833f
2020-01-24 11:30:16 +01:00
Dmitry Tantsur
8bf1d8dff1 Collect output of pip freeze in the CI
Grealy simplifies debugging package conflicts.

Change-Id: I6937800c73a914f4a2225cdea2c41d1f9ad8cec8
2020-01-23 13:03:19 +01:00
Riccardo Pittau
68ed2960e3 Set Ansible default version to 2.8
Ansible 2.6 is EOL, we should move to a more recent release that
has better support for Python 3.x and, above all, is maintained
with security and critical bug fixes.

Change-Id: I6aa80b4bc4a0c34ac919951cd940b2fc6b736bdc
2020-01-23 10:05:27 +01:00
Riccardo Pittau
906d7169ac Fix test for python3-venv pkg install
The test to check if the python3-venv package is needed is wrong
and it will always fail, asking to install the package in systems
where that doesn't exist, like CentOS or Fedora.
The python3-venv package is always needed for ubuntu bionic, so
installing it all the time if we're using that distro.

Change-Id: I6fe765609cc0561f359986f488268e79cca469e2
2020-01-22 12:36:21 +01:00
Julia Kreger
6677bb39f6 Hopefully final venv fixes
Also uncapped the proliantutils maximum as upper
constraints is in place.

Set the ansible python interpreter as well for the
main process launch, which magically resolved
venv dependency issues.

Change-Id: Iae06a87e7dbcf76e4be7453e57ec5fd75389caa7
2020-01-20 13:16:53 -08:00
Riccardo Pittau
1a41d469ed Define venv package for debian based distro
On L115 we ask to install the [venv] package in case we're unable
to create the virtual environment.
For Ubuntu Bionic the package needed is python3-venv.

Change-Id: I065b515e66ced54daaa7e607b89137bd10a1e2f7
2020-01-20 09:51:27 +01:00
Julia Kreger
3b859c83f5 remove legacy ansible environment setup hint
Our ansible installation changed quite some time ago,
and it is time to go ahead and remove the outdated reference.

Change-Id: I43c65b742616835662cf5bc3b87a22aa42b79185
2020-01-16 13:52:33 +00:00
Julia Kreger
44196ea91b Switch to use a venv by default for testing
Also updates the method to use the python3 native
venv module instead of the virtualenv module.

Change-Id: Iae4248acec0b3bdf48be62134f64356d40375dde
2020-01-15 17:18:54 +00:00
Julia Kreger
f95050b613 Only perform OpenStack CI prep when in OpenStack ci
While ZUUL_BRANCH was not always being set, the way the roles and
playbooks were being executed the credential set action when locally
testing using test-bifrost.sh.

Change-Id: Iebe3937339572a1a21527fd7129a09dd05e0a8b7
2020-01-08 19:30:54 +00:00
Julia Kreger
1ef02fa3d3 Make a copy of requirements for local testers
The requirements repository is provided to us in most CI
testing scenarios, however we also want to force a constraint
upon installing python libraries for test VM creation dependencies.

As such lets at least get a copy of requirements up front when
testing with test-bifrost.sh or when using vagrant.

Change-Id: Ic6a5b8f7cdbf6f4c6233848bab1bfd71acc9d973
2020-01-08 19:30:37 +00:00
Julia Kreger
8a6cdf7d3a Change install-scripts for py3 with Ubuntu
Reworked the pip install logic so that we run the pypi
pip installer instead of trying to navigate through running
a pip install and upgrade with setuptools additionally.

These steps were orginally only done in order to force
us to a pip version >= 9.0.1, which is now the default
for Ubuntu Bionic.

Change-Id: Ic3a95973338bbd03061d5b3e4080c0d72d6b891a
2020-01-08 11:30:13 -08:00
Julia Kreger
40509c16bc Set apt use to python3.
We had python hard coded instead of allowing ansible to autodiscover
the version of python to use. :(

Change-Id: I3fbd873ee72cb3783bcb77ef14478fea302ffa3a
2020-01-06 13:55:42 -08:00
Radosław Piliszek
c5745daf00 Fix install-deps.sh failing on bare CentOS 7 due to old pip
This change upgrades pip instead of just ensuring it meets
the lowest version constraint.

This is in line with the setuptools being upgraded just below.

This did not affect CI because it provided its own pip.

Change-Id: I36a0e31ecba1dc9d5eec1d2338dc752467be5ac0
Story: 2006833
Task: 37406
2019-11-11 18:48:59 +01:00
Dmitry Tantsur
4bb71fc481 Migrate to Zuul v3 native jobs
This corrects the path handling in the CI and removes a lot of
redundancy in the CI definitions. Also adds fixed log collection
from https://review.opendev.org/#/c/649526/ to make the CI pass.

Removes handling of ZUUL_CHANGES. It is not present in new jobs,
and its use case in Bifrost is completely unclear.

Eventually, we should rewrite the whole test-bifrost.sh in ansible,
but that's a much bigger task.

Change-Id: Id893e816a7806e7d75282628817288d8a34ab8af
2019-10-30 14:26:26 +01:00