31 Commits

Author SHA1 Message Date
Marcin Juszkiewicz
541daaa3ff build and publish wheels for aarch64 architecture
- CentOS 7, 8
- Debian 'buster'
- Ubuntu 'focal', 'bionic', 'xenial'

Change-Id: Iebb1062269011664411691ef8fd55219afaccd1e
2020-05-27 14:30:44 +10:00
Marcin Juszkiewicz
3445673eea build and publish wheels for more distributions (x86-64)
- CentOS 8
- Debian 'buster'
- Ubuntu 'focal'

Change-Id: I767359d48e04473fad0992b35c804ed511cfaa62
2020-05-22 15:44:17 +02:00
Andreas Jaeger
398adb791f Bye, Bye, Trusty
This removes trusty from the repo and thus from OpenDev.

Afterwards the AFS volume mirror.wheel.trustyx64 can be deleted.

Depends-On: https://review.opendev.org/702771
Depends-On: https://review.opendev.org/702818
Change-Id: I3fa4c26b0c8aeacf1af76f9046ea98edb2fcdbd0
2020-01-19 16:00:55 +01:00
Ian Wienand
f8849904e8 Remove legacy-install-afs-with-puppet role
This role uses the system-config:install_puppet.sh script to first
install puppet on the wheel-builder host, and then uses
puppet-kerberos/puppet-openafs to bring up the AFS environment.

This is replaced with ansible roles to do the same AFS bring-up.  A
new "openafs-client" playbook is added to configure this for the infra
environment.  The wheel-building base job is reworked to use this.

Roles have been tested under zuul with
https://review.openstack.org/596018

kerberos-client role:
Depends-On: https://review.openstack.org/593486
openafs-client role:
Depends-On: https://review.openstack.org/593487
add system-config to linters:
Depends-On: https://review.openstack.org/596645
Change-Id: I9a3af36048e4a4439fe86b36582667ff0af68035
2018-08-27 17:26:27 +10:00
Paul Belanger
ef48688dc6
Bring publish-wheel-mirror-ubuntu-bionic job online
The releases project has been updated to support ubuntu-bionic wheels,
start publishing them to AFS.

Change-Id: I30c1a7c702a7f2cf7f1544ddd62a7a0e719d9572
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-03-07 10:02:01 -05:00
Andreas Jaeger
fca8d78211 Use host_vars for wheel builds
No need to use magic names since we now have host_vars, instead set
wheel_python explicitly.

Remove TODO comment.

Change-Id: I5051a32d843b106b013b94d57fd24b63896b741a
2018-02-17 16:40:04 +01:00
Andreas Jaeger
3a806df6c6 Move wheel-build scripts out of jenkins/scripts
We only need the three wheel-X scripts from jenkins/scripts and need
them only for the playbook and its roles. Move the scripts in the
roles directory and copy them only if needed - using the scripts module.

Remove copying of complete jenkins/script for this playbook using the
role legacy-copy-project-config-scripts, it's not needed anymore.

Change-Id: I2e481fcf7ca148aac8a36ae99d8598ba26078a25
2018-02-13 21:12:45 +01:00
Monty Taylor
09da196701 Remove explicit bindep parameters
All of these invocations of the bindep role are passing what amount to
the defaults. In zuul-jobs, doing bindep_dir: "{{ zuul_work_dir }}" is
important because it allows in-repo uses of those jobs to override how
they work for things like triggering from one repo but running on
another.

The publication jobs in project-config do not need that level of
flexibility, so just remove params and let defaults take over.

As a follow up, we could also add defaults for zuul_work_dir - and
honestly for bindep_dir - to the base job. That way we can always count
on them being set to a sane default value, and they're still
overrideable in a job.

Change-Id: I5bcda84aefa4726d0a904e8bc9698c21ca07f9a5
2018-01-05 22:17:17 +01:00
Dirk Mueller
0360aea196 Avoid adding centos repos on non-centos distros
Change-Id: I08b94d388c8d62fb3190cff02468323f3bb0026d
2017-11-24 23:41:48 +01:00
Paul Belanger
4b5a512313
Add centos-release-openstack-pike support for wheel-build jobs
The openstack/requirements projects needs to pull in a few more
dependencies to build wheels. Add back centos-release-openstack-pike
which was recently deleted from base images.

Change-Id: I2093905f8b48d9dfedd07bcf8f3721c658c4acf8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-24 09:57:07 -05:00
Paul Belanger
b181395568
Upload build-wheel-mirror jobs into python specific folder
Because we run multi-node jobs of our wheel-mirror jobs, we need to
hash by the version of python the node is running. Otherwise, both
nodes will upload logs to the same place.

Change-Id: I5557747db0773873a6542c5ebbac25ca4d4c779e
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-21 10:22:24 -05:00
Paul Belanger
842e4330a4
Split build-wheel-mirror into publish-wheel-mirror
This allows us to add the build-wheel-mirror jobs to
openstack/requirements check pipelines, but not release wheels to afs.
Making it easier to properly debug these jobs, otherwise an infra-root
must manually enqueue-ref for periodic pipeline.

Change-Id: Iabedbfeb62f1773d10462e8b07bbe82f1fb76389
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-17 11:21:52 -05:00
Paul Belanger
251bce14c0
Fix afs_slug for centos-7
Because, why can't things be the same across distros?  We need
ansible_distribution_major_version for centos (7) and
ansible_distribution_version for ubuntu (16.04).

Change-Id: I06f3b587a78165674d0d09432164d79caa5ecf02
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-16 19:33:41 -05:00
Paul Belanger
a35cacd718
Collect logs from build-wheel-mirror jobs
We generate build logs when we build wheels, collect them so we can
review if there is an issue.

Change-Id: Ibc69a6c1ffa87a698c8452fcd03951f2a7aff5a7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-16 13:21:01 -05:00
Zuul
771ad39aec Merge "Move copy-wheels roles to post-run" 2017-11-16 15:29:51 +00:00
Paul Belanger
3a01751efe
Use become: yes for yum_repository task
We need to be root to add repo files.

Change-Id: Ie1e56e49fb37792c74cc6b045a8f345b95a6d755
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-15 23:45:59 -05:00
Paul Belanger
ec55e70ddb
Fix "Parameter 'description' is required" error
Guess we should also send a patch to ansible documentation:

  http://docs.ansible.com/ansible/latest/yum_repository_module.html

Change-Id: I7117aa91894bc3edb1b55643b649c42c879a82c4
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-15 23:19:43 -05:00
Paul Belanger
14616db9c5
Move copy-wheels roles to post-run
Currently, our secrets are not included in the
build-wheel-mirror-base, so this means we cannot access the afs secret
to copy files into /afs/.openstack.org. Move this into pre distro jobs
where the secret lives.

Again, once we have protected final jobs, we can clean this up a bit.

Change-Id: I8e8b92af771bc84a3930a8f98c96c8216388d1cf
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-15 23:14:03 -05:00
Paul Belanger
9373ce5bea
Add package-afs-centos7 for puppet-openafs module
Because centos-7 doesn't have an openafs-client, we have to install
the version we built ourself.

Change-Id: If8fbb9bf7c585eebd2eb7abbb220375bdf38f682
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-15 22:55:21 -05:00
Paul Belanger
7c5c7c83a9
Add more EPEL packages for build-wheel-mirror centos-7 job
So we don't have to enable the EPEL repo.

Change-Id: I9243a1731700e75e60be439c1dd1cfb3e9efe29d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-15 22:09:37 -05:00
Paul Belanger
0f2c4cd9b7
Fix syntax error with when statement for build-wheel-mirror
Doesn't look like --syntax-check or linters caught this.  Update to be
more correct.

Change-Id: I95f34611ef5733e930eead1cbe5ee4573ea7aadf
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-15 16:15:43 -05:00
Paul Belanger
8cc7bab0f7
Move yum install for parallel into pre_tasks
We need this be done before we call the bindep role. Oops.

Change-Id: Id25594a3ba3bfc4964d558fddfa0c5660ec11c7f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-15 15:47:01 -05:00
Paul Belanger
f366c3bc13
Install parallel from EPEL for build-wheel-mirror
Right now, it seems we only need parallel from EPEL. Rather then
enabling the whole repo, we can just opt to install it first, then
bindep will be happy.

Change-Id: Ifd1eb186f78c395f549d1430cfe4414ac1121f63
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-15 15:04:27 -05:00
Paul Belanger
b4f940481b
Set strategy: free for build-wheel-mirror jobs
While we are using multi-node jobs, neither host needs to talk to the
other. So, have the race fast as possible to the end of their
playbooks.

Change-Id: I6fa39a5444d1c3e6188e1067ffaa45d0fe97e90d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-15 02:26:39 -05:00
Paul Belanger
73e3ca2698
Fix mirror.wheel AFS volumes for releasing
We had a typo in our volume names.

Change-Id: I75021b65cf53556c63b1ff076adcc1c85d695c42
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-13 15:36:49 -05:00
Andreas Jaeger
35b8dfe3b8 Fix wheel/release
The role is release-afs-volume in zuul-jobs.

Change-Id: Idb8be9f98b77b2dbc5d01d367523ccb2cf29b53d
Co-Authored-By: Jesse Keating <omgjlk@us.ibm.com>
2017-10-21 17:52:51 +02:00
Paul Belanger
5501f12f23
Add bindep support for build-wheel-mirror job
Our wheels will end up requiring dependencies, so add support for
bindep.

Change-Id: Ifa73ba475284ac8435ba29ffaf228a54dce59b09
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-09-23 14:54:23 -04:00
Monty Taylor
02c84560b8
Make wheel mirror builders normal zuul nodes
Including puppet madness!

We don't need the long-lived wheel builders - we build from scratch
every time. So all we need is AFS running on a node.

AFS, however, is tricky on CentOS. Rewriting all that right now is meh.
We can do that later.

This gets us to not needing special nodes for wheel builders. Yay!

Change-Id: I6d92c47f4f7a26ca290cd400452cb0a73872b24d
NOTE: It may be inefficient to run this on our low-bandwidth nodes.
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-09-13 14:25:37 -04:00
Monty Taylor
622f04f214
Treat wheel builders more like build nodes
Pass the wheel keytab across and aklog with it.

Also, split the build role into a build and a copy so that we only have
the token when we need it, and not while we're running the wheel builds
themselves. And split release from build so that we can release the
mirror once for a consistent state across python2 and python3.

Change-Id: Id30a99f5b515dd520d013fabbd356e32c82030a0
2017-09-06 14:44:06 -04:00
Monty Taylor
8a7456ed90
Ansiblify the wheel build a little more
Change-Id: I669e7b5ca9eac4b00bf0284c1cd4f9ad8c3f9d09
2017-09-06 14:44:04 -04:00
Monty Taylor
7e09fc71a3
Add wheel mirror jobs
We need to run periodic jobs to build the wheel mirrors.

This removes them from the v2 config because we really should not run
these jobs from both systems at the same time and once this lands it'll
be live.

It copies the scripts from project-config to the homedir, so we can
ditch the puppetted copies of them.

Change-Id: I87a78509a97d53eb627a84c646a6a3f0a227d9a1
2017-09-06 14:44:02 -04:00