225 Commits

Author SHA1 Message Date
Radosław Piliszek
1d0c1e51af CI: Use docker mirror proxy
TODO: Debian build still requires access to original repo.

Change-Id: Ie4bac5334369a63793d36568a9d3a58a0cdc7667
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-09-19 10:31:41 +02:00
Mark Goddard
fa8e174be6 Remove support for OracleLinux
We have agreed to remove support for Oracle Linux.

http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006896.html

Change-Id: I961dfba1aeef2c509ecdaf376dfe6928926ffb39
Depends-On: https://review.opendev.org/682858
2019-09-18 13:42:47 +01:00
Radosław Piliszek
6d853dc04e CI: CentOS - override mirrors ASAP + crmsh mirror override
This makes mirrored repos used as soon as they are installed
in the image, instead of hitting upstreams.

This patch also adds crmsh override.
Usable only after infra change merges.

Depends-on: https://review.opendev.org/682576
Change-Id: Ic7494eff4c740b6e528da4508dc88a64e697bc04
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-09-17 17:42:56 +00:00
Marcin Juszkiewicz
016c0f577c Drop Percona XtraBackup in favour of MariaDB Backup
XtraBackup started failing to install on Ubuntu and does not support
MariaDB 10.3 version.

Related-Bug: #1843043

Change-Id: I9cf62c8e03a02574d85e5349df1d23a51d4c99be
2019-09-12 14:14:08 +02:00
Marcin Juszkiewicz
3a4b223d6a Change how build process treats unbuildable images
We have a list of images which are unbuildable for distro/buildtype/arch
combos. This patch renames it from SKIPPED_IMAGES to UNBUILDABLE_IMAGES
because 'skipped' images is something else in our code.

Now all unbuildable images are marked as such and so are their children.

At the end of build there is information provided about skipped images
(due to --skip-parents or --skip-existing options) and then information
about not buildable images. Both list should only contain entries
related to requested build.

Change-Id: If9b521339f564e483cba03d52e7c4eba271821a5
2019-07-21 17:39:06 +00:00
Marcin Juszkiewicz
0b796fbf98 cleanup setup of Debian/Ubuntu CI nodes
Change-Id: I3a771793dfbbfc26f6d5716ba7a5b60c2e6154fa
2019-07-15 13:41:30 +02:00
Radosław Piliszek
e7e0adad6d CI: Clean up Zuul repo mirrors for CentOS
- unify RDO mirror definition with others
- drop definitions of unused mirrors: grafana, elastic
- clean up the process of mirror substitution:
  - use the more common style of sed d command
  - apply more specific substitutions

Change-Id: Idad2c6956cca647d4c865d6f6149ea37164db568
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-07-03 16:58:49 +02:00
Radosław Piliszek
8270ae20f9 Extract CI vars for use in kolla-ansible jobs
Change-Id: If58f15e18e420aca790496132352f1c1dcdb2640
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-06-28 20:18:14 +02:00
Zuul
4d88a4f20a Merge "Clean up test process" 2019-06-27 15:54:22 +00:00
Radosław Piliszek
ccabadc342 Remove Oracle Linux stuff in CI code
Oracle Linux is no longer tested via CI after [1].
This patch removes OL support from Zuul files.

[1] https://review.opendev.org/664217

Change-Id: I527aba3635ef82e2d2a88ee7f5250bf03e420800
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-06-25 19:29:46 +02:00
Radosław Piliszek
c7ac0f0de0 Clean up test process
If stestr failed, then no node info was logged.
Logging node info is now handled in pre and post stages of Zuul.
Additionally Zuul v3 migration recommendations were applied
and some unused code removed improving general readability.

Change-Id: Iae9d37cb76997ba8389e21c7e2a3bf5905abe029
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-06-20 10:25:41 +02:00
Radosław Piliszek
de58339a0e Fix issues with Percona in Zuul and clean up a little bit
Percona site has connectivity issues from OVH hosts (of Zuul).
This patch makes it so that:
  - the internal repo is actually used
  - RPM GPG keys are downloaded from the actually used repo
  - Ubuntu mimics the RHEL behavior (i.e. the Percona repo is
    enabled only in the xtrabackup image)
    extra benefit: no need for APT preferences for Percona repo
    to spare MariaDB deps

Change-Id: Id27831f6b404762f9e44fd6bd1c4702bf06b05f8
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-06-08 18:54:19 +02:00
Marcin Juszkiewicz
5859180c2c Debian/Ubuntu: allow to use unsigned repositories
When we run Debian 'buster' or Ubuntu 18.04 in CI job they refuse to
work as infra mirror is not signed.

This patch instructs APT to use unsigned repos anyway.

Change-Id: I530fde5b57894afda2de98d7c217e735497e3feb
2019-06-07 15:55:58 +00:00
Mark Goddard
cdcd5610e9 Fix lower-constraints and kolla CI jobs
Multiple fixes to fix multiple problems.

1. Bump setuptools requirement to 21.0.0

The 16.0.0 requirement breaks the lower-constraints job on Ubuntu
bionic. We see the following error:

AttributeError: module 'importlib._bootstrap' has no attribute 'SourceFileLoader'

2. ubuntu-binary horizon image needs to use python3 for django
manage.py

3. ubuntu-binary nova-placement-api image needs to use python3 for
apache mod wsgi.

4. nova-libvirt needs exact qemu dependencies due to packaging change

5. OracleLinux repos definition was changed in official image.

6. Fetch InfluxDB gpg key for Ubuntu base image from influx server.

Change-Id: Ic797d41f53a5f146d112619f1683cbfbf3a85dc2
2019-03-15 12:01:25 +01:00
Michal Nasiadka
a490eef0d5 Fix py36 gate job
python 3.6 pycache also includes links - so those also
need to be removed

Depends-On: https://review.openstack.org/636750
Change-Id: I0f995df45bf203acc79cc7bd52b9ac51577468da
2019-02-14 09:13:32 +01:00
confi-surya
8acba6ff56 Update hacking version
Use latest release 1.1.0 and compatible changes w.r.t pep8

Change-Id: Ia3d1166a2746a46591e4e5a04ea91e03ddc28d93
2018-12-27 12:02:17 +00:00
Zuul
e215e51fc1 Merge "Use infra mirrors for ODL and percona repos" 2018-12-19 13:48:02 +00:00
Zuul
1bcd0617b6 Merge "No need to use dict_object.keys()" 2018-12-18 10:41:33 +00:00
Eduardo Gonzalez
0c4238da7c Use infra mirrors for ODL and percona repos
Change-Id: I212a53b235f2785d93d474660c87a8521f3cc8b2
2018-12-18 09:13:52 +00:00
Eduardo Gonzalez
65296e27e6 Fix Oraclelinux build, disable collectd in gate
Oraclelinux repositories has been upgraded and
previous file removed, this change update repos
to new URLs.


Disable collectd in gate as is currently broken and
blocks mergingin voting jobs.

Change-Id: I402cff1ac86a2ba108f2f5378cc98db5f6ecc673
2018-12-05 08:46:29 +00:00
confi-surya
8504a20609 No need to use dict_object.keys()
for in dict_object.keys() can be replaced with
for in dict_object, so just a short cleaup

Change-Id: Id934eae6db05298b85b800531b9dfdcb28782cc5
2018-12-03 10:21:27 +05:30
confi-surya
24c993a059 Fix the parameter order of assertEqual
On assertEqual, the order of parameters should be (expected, observed).
But, some part of kolla tests was written with invalid order.
This patch fix the problem.

Change-Id: I9fcfb3923716243ddb1886223e932ec12829fb2e
Related-Bug: #1277104
2018-11-29 16:20:25 +05:30
Eduardo Gonzalez
9c9a7f4303 Suppress log copy output in gates
Copying collectd logs is half of the job ouput.
This change is to remove the verbosity of the syncronize
tasks by applying --quiet to only show errors

Change-Id: I39b47b844aab87d043a2cd745103d6589f3632e3
2018-11-20 10:09:16 +01:00
Zuul
53000a956e Merge "Fix epel mirror replacement in gates" 2018-09-24 12:46:19 +00:00
Mark Goddard
71d8fb478a Disable debug logs for image builds
Logs emitted when building via the tox environments can exceed the
maximum length supported by subunit [1], causing the following message:

Length too long: <length>

This change disables debug logging in these tox environments to reduce
the length of the output.

[1]
8cdd703980/python/subunit/v2.py (L208)

Change-Id: Icb5793a8ea58edafe2f3a249eb1da7d4ae2e395f
2018-09-21 14:55:58 +01:00
Eduardo Gonzalez
3275274616 Fix epel mirror replacement in gates
Change-Id: I7b74497931de5e2a89fc1a2b827a1cf6d64ab7e9
2018-09-19 11:15:35 +00:00
Martin André
3a644f7764 Add CI script to retrieve info from build logs
This should serve as a base for retrieving useful information from the
kolla build logs.

For now, the script warns about obsolete package names and helps track
packages that are installed too many times in different images.

It only supports rpm-based distros for now.

Change-Id: I160b6ede1336bd0c202621a48ca0218f72fc1270
2018-07-10 10:14:36 +02:00
Martin André
7e770472b1 Show disk space used by docker images in CI
The `docker system df` command can show the total size of the docker
cache. If you pass it the `-v` option, it will break it down per image,
replacing the use of `docker ps -a` and `docker images`.

This should be very useful in CI to see the variation in the images
size.

Change-Id: I407434d88f59df65693974265551c624420d4aaf
2018-07-10 10:12:48 +02:00
Martin André
c4eb866ad5 Stop creating an empty system_logs dir in CI
The system logs from the building node are copied to the system_log
directory. There is no need to create an empty system_logs directory.
It was certainly a typo.

Change-Id: I1f0eea485550bd7b41c9954a6b7077251518b31d
2018-07-09 21:23:58 +02:00
Zuul
5caa5e30de Merge "Bump Ubuntu base image to 18.04" 2018-06-04 10:39:19 +00:00
Jeffrey Zhang
14374e715d Bump Ubuntu base image to 18.04
Ubuntu 18.04 has been released. We need upgrade to it this cycle.

Changes made in this patch

- Bump ubuntu repo from xenial to bionic. Some repos do not have bionic
  packages, so still using xenial repos.
- Drop mariadb, percona and nfs-ganesha repo, because these packages are
  already provided by Ubuntu official repo.
- Since qdrouterd does not have bionic repo and xenial packages does not
  work on Ubuntu bionic, just add it into ignore image list.
- Use python-rtslib-fb and targetcli-fb rather than python-rtslib
  and targetcli
- use xen-utils package name rather than with package version one.
- Seems unsigned repo won't work on Ubuntu bionic, stop using infra
  mirrored unsigned repo in gate building job.

Needed-By: https://review.openstack.org/568728
Implement blueprint: ubuntu-bionic
Change-Id: I4e3b0ca669cfbf6316bf591f2d8428fa1a0d6182
2018-05-26 00:24:30 +08:00
Martin André
58742c212e Create symlinks for failed builds logs
This allows to know which image failed to build without having to parse
the huge job-output.txt file.

This commit creates a convenient 000_FAILED_image_name.log symlink to
the relevant log files when kolla is configured to output to a log
directory. This creates symlinks only for the parent image with error,
i.e. it skips images that failed to build because of parent error.

Change-Id: I05954e2335dc3fac469276809c91f9c87e3ecab2
2018-05-08 09:08:05 +02:00
Zuul
69a4b7e049 Merge "No use of mox/mox3" 2018-04-04 02:40:05 +00:00
Zuul
6a3c5c58fe Merge "Revert "Seem there are some wrong in infra ubuntu repo"" 2018-04-03 10:13:07 +00:00
Surya Prakash Singh
edb4368fee No use of mox/mox3
Now onwards no support of mox/mox3, please check storyboard link
for reference

https://storyboard.openstack.org/#!/story/2001546

Change-Id: Ic1fa65951f1e43e1279d8c72fcfc5e09609faa70
2018-03-30 03:06:07 +00:00
Jeffrey Zhang
abc61c012c Support push kolla tag image to hub.docker.com site
Change-Id: I8a21f60a37ba038e46c29d0bcff9ffbd2457a257
2018-03-21 16:53:15 +08:00
Eduardo Gonzalez
01a5388169 Verify YAML syntax in gates
Test yaml syntax is correct on new changes.
Fixes current warnings and errors

Change-Id: I5888f8e4a9d27a08506036df2c564b9f2081ccee
2018-02-27 21:45:57 +00:00
Jeffrey Zhang
ad8666f5a0 Revert "Seem there are some wrong in infra ubuntu repo"
This reverts commit 7b562990b5b717bb83d56673b368de9b355bb315.

Change-Id: Id5f0e9532a7714a18125832d4f38bbe7bb849c84
2018-02-26 07:37:55 +00:00
Jeffrey Zhang
7b562990b5 Seem there are some wrong in infra ubuntu repo
Revert this after infra ubuntu repo is OK.

Change-Id: I8bbf8257d5a932b7e9f2e9a64d62cd5892e3f1bb
2018-02-12 21:04:54 +08:00
Zuul
4a423d387b Merge "Add collectd in ci to collect system information" 2018-01-16 08:44:59 +00:00
Jeffrey Zhang
77612f7eed Add collectd in ci to collect system information
Change-Id: Ic5cbdfd4ebca1d907ada5c39f27dc4a7889daa21
2017-12-18 13:08:10 +08:00
Jeffrey Zhang
e6036f52ed Use separate publish jobs in periodic pipeline
The secrets can not be inherited.

Change-Id: Ie0f99f470a419c57865eb8b0a14e465ae228c799
2017-12-06 06:03:54 +00:00
Jeffrey Zhang
5feb113033 Change more repo url to infra mirror
Change-Id: I7e9fd2760ae3202f3a84833bb65edfdcffa67fe4
2017-12-04 17:23:31 +08:00
Jeffrey Zhang
9846a27a6b Improve gate post jobs
- Reuse build jobs to publish images
- Add binary jobs in periodic pipeline
- Push all image to dockerhub
- Use branch name for image tag

Change-Id: Ifefb96bcbcc9c6840cf865f7786b43a43fae4c17
2017-11-28 04:45:30 +00:00
Michal (inc0) Jastrzebski
9674e2da1c Fixup small bug with dockerhub publisher
Jinja2 vs gotpl...

Change-Id: I27a54eb9202ac5f3e9b3ef9c307e81475e8833f5
2017-11-08 15:00:13 -08:00
Michal (inc0) Jastrzebski
c506e626e7 Dockerhub publisher job
This patchset includes pushing mechanism for daily periodic jobs.

Change-Id: Ic4179ffd3a6e640abfb181dbd000bc75cc64b750
2017-11-06 09:18:22 -08:00
Jeffrey Zhang
b64198cce0 Moving jobs to kolla repo
This is zuulv3 migration patch. This patch only implement build jobs.
Deploy jobs will be pushed in another one.

Partial-Bug: #1720601
Change-Id: Ib9157ed132820d752ef1dbd7a53353ebd68b1934
2017-10-19 10:34:03 +08:00
Jeffrey Zhang
b1b7492e65 Skip known failed kolla images
Change-Id: Ide8478c248e77b9c85201981e8086f3dbf06d131
2017-09-23 16:06:51 +08:00
Eduardo Gonzalez
7ba7fdd855 Enable mistral in test_build centos source
For some reason, mistral was disabled in test_build since
long time ago. I've been using mistral with no problem.
Also current gates build mistral images properly.

Change-Id: I89e79c22f0e15b8165b23fa8cd98df309efdaf45
2017-08-29 11:27:55 +02:00
Eduardo Gonzalez
7b2dea5506 Implement tacker-conductor image, rework all tacker
Tacker is broken in pike without conductor images.
Conductor is needed to communicate with mistral for
VIM monitoring
This change rework tacker image to allow tacker-conductor
service image.
Split tacker global image into tacker-server and tacker-conductor.

Closes-Bug: #1710874
Change-Id: I52778e86e4f2c297ead8d4b09983e5e38ca88c70
2017-08-20 09:58:27 +00:00