283 Commits

Author SHA1 Message Date
Marcin Juszkiewicz
c42b882aff CI: use wheels from infra mirror
If they are available then we do not need to build them. Especially
AArch64 jobs will speedup (once wheels are built).

Change-Id: I79af6c37950e156018a9204fbcc7417cd7d41012
2020-06-18 20:53:58 +00:00
Marcin Juszkiewicz
039a6d431a CI: fix line merge in CentOS builds
Looks like I missed one situation with "base/centos: take care of
whitespaces" change.

Change-Id: Id7b5e8ddf23f9ab13806fb02042e76183bcddf05
2020-06-18 16:34:01 +02:00
Marcin Juszkiewicz
09604a7492 base/centos: take care of whitespaces
Let's get rid of this warning:

Change-Id: I121e2ecb30ec6e8d1b6d88d9b921894438c3e094
INFO:kolla.common.utils.base:ESC[91m[WARNING]: Empty continuation line found in:
INFO:kolla.common.utils.base:    RUN dnf -y install centos-release-opstools epel-release dnf-plugins-core     && dnf config-manager --enable PowerTools     && dnf config-manager --disable centos-ceph-nautilus     && dnf config-manager --disable centos-nfs-ganesha28     && dnf config-manager --disable influxdb && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-OpsTools && dnf -y distro-sync --security --sec-severity=Important --sec-severity=Critical
INFO:kolla.common.utils.base:[WARNING]: Empty continuation lines will become errors in a future release.
INFO:kolla.common.utils.base:ESC[0m
INFO:kolla.common.utils.base:Step 1/39 : FROM centos:8
2020-05-17 17:38:10 +02:00
Marcin Juszkiewicz
8abfa0cc75 CI: make sure that virtualenv is available
Failed to find required executable virtualenv in paths:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

Change-Id: Ie6094e1c5de5c89e0180c42edf5ef94c1c9964b7
2020-04-30 22:19:06 +02:00
Zuul
2f66efdf5e Merge "Remove support for CentOS 7" 2020-04-15 14:02:57 +00:00
Marcin Juszkiewicz
53443c5c71 Remove support for CentOS 7
With the move to RHEL/CentOS 8 we no longer have Python 2 in our images
so there is no need for checking which Python version (2.x or 3.x) is
used inside of containers.

We also no longer have to support yum as a value for
distro_package_manager.

Partially-Implements: blueprint centos-rhel-8

Change-Id: Ie45cf3465fedddbde7856961527421883ba3d5c9
2020-04-15 09:32:06 +00:00
Marcin Juszkiewicz
584da06751 CI: add Centos 8 and Ubuntu jobs for aarch64
also simplify Debian job definition

Depends-on: https://review.opendev.org/718413
Depends-on: https://review.opendev.org/718363
Depends-on: https://review.opendev.org/718236
Depends-on: https://review.opendev.org/718238

Change-Id: I2c75dbc2ea063dd3507eaaf93a484c50acae5fb5
2020-04-08 21:11:44 +00:00
Zuul
7528397120 Merge "CI: Ansiblify setup of builders" 2020-04-04 16:31:28 +00:00
Zuul
baec0b913a Merge "CI: Avoid tox" 2020-04-04 16:31:27 +00:00
Zuul
ee542d05eb Merge "Update hacking for Python3" 2020-04-03 20:03:12 +00:00
Radosław Piliszek
2cdcf1821d CI: Ansiblify setup of builders
CI is set up using Ansible playbooks now.
This also drops redundant shell scripts.

Co-authored-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>

Change-Id: I5264746fd04276448f4668c3bb3a218a40e8660b
2020-04-03 15:59:20 +02:00
Radosław Piliszek
c089eab392 CI: Avoid tox
We don't need tox nor tell our users to use it when doing kolla builds.

Tox is going away from base infra images (it is already gone from
aarch64 ones).

Depends-on: https://review.opendev.org/713134
Co-authored-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>

Change-Id: Ib1d97a783951ac42740ebf91bcc6ecaf2bf70853
2020-04-03 15:46:11 +02:00
Zuul
3da0aaa845 Merge "CI: Add https match for mirror.centos.org to template-overrides" 2020-04-03 12:53:31 +00:00
Michal Nasiadka
1a28aefec8 CI: Add https match for mirror.centos.org to template-overrides
Some CentOS repos (currently one - Ceph NFS Ganesha) use
https://mirror.centos.org instead of http:// and we break
their definition by removing mirrorlist= entry and not inserting baseurl
entry due to a missing match.

Change-Id: I7b29f8c6be80820042023d2b85162d5cc9a65f08
2020-04-03 11:23:40 +02:00
Radosław Piliszek
924e7a40f1 Use unittest.mock instead of PyPI mock
Now that py2 is gone, oslotest dropped dependency on mock and will
soon affect Ussuri CI [1], let's use unittest.mock built in py3.

This also fixes py38 jobs and proactively prevents py36 and py37
failing due to [1]. This is because we never included mock in
test-requirements (but in lower-constraints where it does not
really belong at all) and instead relied on oslotest to bring
it in.

[1] https://review.opendev.org/716322

Change-Id: Iceedcc7ad0d087414f6f94dfc0235fda4754be63
2020-04-02 18:22:38 +02:00
Andreas Jaeger
1456d8b6f9 Update hacking for Python3
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found by updated hacking version.

Update local hacking checks for new flake8.

Remove hacking and friends from lower-constraints, they are not
needed in installations.

Change-Id: I67e5ca24c550163c218fb06692de0b7c528d8318
2020-04-01 21:42:41 +02:00
Marcin Juszkiewicz
c2a3380707 CI: publish AArch64 Debian/source images
We want to test upgrades on aarch64. For this we need to publish images.

AArch64 images will get "-aarch64" added to tag to not collide with
x86-64 ones.

Change-Id: Ibae2578b4bc18c0ab7fcc1840c39bb86659ab920
2020-03-19 15:28:00 +00:00
Zuul
0e25302fae Merge "CI: Install tox" 2020-03-17 21:14:19 +00:00
Mark Goddard
a81413fc52 CI: Install tox
tox will be removed from the base image. Install it before that happens.

This change is made in a simple way that can be easily backported.

Change-Id: I2a24c87aa44c1e03ba76a2fb0bd71caf54eb4211
2020-03-17 17:45:04 +00:00
Mark Goddard
8294bac01b CentOS 8: Switch default base image, tag as master
Changes the default base tag and distro for CentOS images from 7 to 8.

Also changes the tag used by the publisher jobs for CentOS 8 images from
'master-centos8' to 'master'.

Change-Id: I5498141ade23219d63f34debc2b8e1330dec2e68
Partially-Implements: blueprint centos-rhel-8
2020-03-17 16:38:43 +00:00
Radosław Piliszek
5912f2c16e CI: Use network mode = host
This avoids issues with IPv6 not enabled in docker.

Change-Id: Iff80392357ef5e2dd3f2e24dbf4df299660805b9
2020-03-13 18:29:44 +00:00
Radosław Piliszek
c328097725 Revert "Upgrade virtualenv in pre"
Reverts temporary.

This reverts commit 063a46a7de4471504535499ca2a972a79e4ce5e5.

Change-Id: Ibf2848db954862f917a23eb0aa1594ccc4e7ef67
2020-02-13 11:32:26 +00:00
Michal Nasiadka
063a46a7de Upgrade virtualenv in pre
Since virtualenv 20.0 (amongst other changes) six version >1.12.0 is required.
This change adds upgrade of virtualenv and six in pre - to be reverted once
infra CentOS images are sorted out.

Depends-On: https://review.opendev.org/707377
Change-Id: I75249055989309d0afdfc29c894beb6b1fcd1994
2020-02-12 13:48:46 +01:00
Mark Goddard
1c31fe9cd6 CentOS 8: Fix CentOS 7 publisher tag
Currently we use CentOS 8 nodepool images for all CI build jobs. This
means we can't use the ansible_distribution_major_version fact to detect
whether we are building CentOS 7 or 8 images. Due to this, all CentOS
images (including CentOS 7) are currently published with a
'master-centos8' tag.

This change fixes the issue by using the base_tag variable which is
defined for CentOS 8 build jobs.

Change-Id: I29b7b8ee742120107a11bd419ac58ab5c09b0b75
Partially-Implements: blueprint centos-rhel-8
2020-02-05 14:20:45 +00:00
Zuul
e97239f03e Merge "CentOS 8: Update packages in images" 2020-01-30 20:13:03 +00:00
Mark Goddard
cde5405b90 CI: Fix tag suffix logic for CentOS 8
On CentOS, ansible_distribution is 'CentOS' rather than 'centos'.
Our logic when setting the image tag for publisher jobs is incorrect,
meaning that the centos8 publisher publishes images as 'master'
rather than 'master-centos8'. Since this job only has two images it
is likely to complete before the CentOS 7 publisher, so hopefully
images will not be broken for long.

This change fixes the issue by using a case-insensitive comparison.

Change-Id: I66c93e07fe5c6c143e9cf65c6451e573bbfc51d8
Partially-Implements: blueprint centos-rhel-8
2020-01-30 12:19:13 +00:00
Zuul
7bec0316a8 Merge "CentOS 8: base and openstack-base images" 2020-01-29 11:55:19 +00:00
Mark Goddard
4b4e26262b CentOS 8: Update packages in images
* Some further changes for python2 vs python3 packages
* Allow rabbitmq 3.7.*, since a newer erlang is available
* Switch from qemu-img-ev to qemu-img on CentOS 8
* bridge-utils no longer available on CentOS 8
* libvirt-daemon-driver-lxc no longer available on CentOS 8
* Mark some more images buildable for CentOS 8

Change-Id: Iaf5b68ff6d944ae730ca0b1d5832172c106a6c08
Partially-Implements: blueprint centos-rhel-8
Partially-Implements: blueprint centos-rhel-python-3
2020-01-29 11:41:38 +00:00
Radosław Piliszek
4a0182f15a CI: Fix symlinks to failed logs
Stop renaming files. Swift should be able to serve text/plain files
based on content, not extension.

Failed logs symlinks were broken because we stopped gzipping log
files and improperly rewrote the logic there.

Change-Id: Idddf51730821644948f51577807f5beaa18c6262
2020-01-20 09:30:27 +00:00
Radosław Piliszek
1cb4760b05 Revert "Debian/source: do not force tag in build jobs"
This reverts commit ca4da16836e4b3ef1d84059a63378c3f5bb2517e.

This is adapted to revert only the missed part of
aarch64 publisher which now blocks publishing for debian.

Change-Id: I02932857454bdb09b72f539038d394760457109d
2020-01-20 08:17:54 +00:00
Zuul
c10f9d7acf Merge "Revert "Add AArch64 publisher for Debian/source"" 2020-01-16 12:21:42 +00:00
Marcin Juszkiewicz
b66f144c42 Revert "Add AArch64 publisher for Debian/source"
Turned out that linaro-london network pipe is too thin 
to support this job. Pushing images took 3 hours so job
timed out.

Will check it once new nodes arrive.

This reverts commit f2801fabdd727fc72a28b056d79787af484424a7.

Change-Id: I8897ede43f11090902fe7c0c096f17f57c1c9c67
2020-01-15 16:33:04 +00:00
Marcin Juszkiewicz
ca4da16836 Debian/source: do not force tag in build jobs
Multiarch publisher uses 'branch-arch' so set it in kolla-build.conf
file.

Change-Id: I1af17edc9671237d270c5c6fe70e4420199bdb02
2020-01-14 15:27:53 +01:00
Zuul
f91ddee1c1 Merge "CentOS 8: Don't force tag in build jobs" 2020-01-14 14:16:56 +00:00
Zuul
acfda84bc1 Merge "Ubuntu: use local APT mirrors if possible" 2020-01-14 12:00:45 +00:00
Zuul
0fdf656427 Merge "Add AArch64 publisher for Debian/source" 2020-01-14 10:15:44 +00:00
Marcin Juszkiewicz
02c7efd42d Ubuntu: use local APT mirrors if possible
APT has a way to fetch mirror list and autoselect best mirror.

Change-Id: I8209770ca1353fd89aacef0859ca3a373645c6b2
2020-01-14 08:15:04 +01:00
Mark Goddard
8788a452a9 CentOS 8: Don't force tag in build jobs
We should allow the tag set in kolla-build.conf to be used for the image
tags, since our deploy jobs need to set it to change_XXXXXXX when images
are built for the job.

Change-Id: I705514ee0249ff1d309e280001d653271e3686af
Partially-Implements: blueprint centos-rhel-8
2020-01-13 12:01:13 +00:00
Zuul
0811df6ac0 Merge "CI: use Debian/Ubuntu mirrors in base image" 2020-01-13 10:15:38 +00:00
Zuul
fc572d1cda Merge "Use Debian mirror in CI jobs" 2020-01-13 10:15:36 +00:00
Zuul
86d616bbe7 Merge "CI: Fix symlinks for failed build logs" 2020-01-13 00:07:33 +00:00
Marcin Juszkiewicz
ad03d7eaa2 CI: use Debian/Ubuntu mirrors in base image
So far we built 'base' image using upstream mirrors and then replaced
them with OpenStack infra ones. Let do that right after we copy own
APT sources.list file.

Change-Id: Icb745f628c1c50d9f20df022b08ddc46fccc270c
2020-01-10 15:57:15 +00:00
Marcin Juszkiewicz
2cf3f2309a Use Debian mirror in CI jobs
There is Debian mirror on each CI worker so we should use it.

Change-Id: Id3b55ac470a977be7ee7decdef3b49e8d89a130f
2020-01-10 16:42:32 +01:00
Zuul
178df355c2 Merge "CentOS 8: Add zuul build and publish jobs" 2020-01-10 12:44:45 +00:00
Marcin Juszkiewicz
f2801fabdd Add AArch64 publisher for Debian/source
This time proper approach involves job dependencies.

First we build and publish x86-64 and aarch64 images with architecture
name in image tag. Then we create multiarch manifests to create 'master'
images.

Change-Id: I6b9505dc326636fc6df8798c62c3a5eb1f728a51
2020-01-10 12:50:34 +01:00
Mark Goddard
63f773f569 CI: Fix symlinks for failed build logs
Follow up to I5b5f323471676317a2898875cbbf297082328fcc. Logs are no
longer gzipped, so we do not need to modify the 000_FAILED_* symlinks.

Change-Id: I5df42ac0ff91b1b97537a4fbc33fc7970f29d60e
2020-01-10 09:37:31 +00:00
Mark Goddard
3c618a1f6f CentOS 8: base and openstack-base images
Adds support to the base and openstack-base images for CentOS 8.
These images have been removed from the list of unbuildable images, and
replaced with all dependent images.

Change-Id: I18fba8a5c2f99b700990cbf41c76077a1ec2408a
Partially-Implements: blueprint centos-rhel-8
2020-01-09 17:51:42 +00:00
Mark Goddard
b060623b90 CentOS 8: Add zuul build and publish jobs
Adds build and publishing jobs for CentOS 8 images:

- kolla-build-centos8-binary
- kolla-build-centos8-source
- kolla-publish-centos8-binary
- kolla-publish-centos8-source

The 'base' image has been added to the list of unbuildable images for
CentOS 8 (now separate from the list for CentOS 7). This means that no
images will be built - the jobs should essentially be a 'noop'.

Change-Id: Ie9a0109bd4eef337358d714e4b5088b8e3de2f23
Partially-Implements: blueprint centos-rhel-8
2020-01-09 16:23:39 +00:00
Michal Nasiadka
f44b0b778b Stop gzipping logs in post
As per [1] we should stop compressing the logs sent to swift in order
to get them back readable via a browser.

[1]: http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011875.html

Change-Id: I5b5f323471676317a2898875cbbf297082328fcc
2020-01-09 16:34:37 +01:00
Marcin Juszkiewicz
d6935edb43 Add Debian/source aarch64 CI job
AArch64 is getting popular and not all changes working on x86-64 work
properly on aarch64 architecture. Let check them.

Job is running in separate pipeline due to low number of aarch64 nodes.

Authored-By: Xinliang Liu <xinliang.liu@linaro.org>
Co-Authored-By: Jeffrey Zhang <zhang.lei.fly@gmail.com>
Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>

Implements: blueprint aarch64-3rd-party-ci-kolla

Depends-On: https://review.opendev.org/#/c/698606

Change-Id: I7e7802dfa4f4d9c61d01dba4b77c01cfb24301c0
2019-12-16 08:17:38 +00:00