Commit Graph

554 Commits

Author SHA1 Message Date
Thierry Carrez
0f12fa2d5a [maintain-github-mirror] Skip abandoned gov files
Following the deletion of the foundation-board-repos.yaml file,
the script failed as it was expecting it.

Remove that file from the scanned list, and make the script more
resilient in case other extra files go missing in the future (like
user-committee.yaml which should soon be cleaned up).

Change-Id: I83cff14f19a829b5e56771442ffb3a0341e82d69
2020-08-10 11:13:15 +02:00
Zuul
36b665536c Merge "maintain-github-mirror: add requests dependency" 2020-07-14 15:26:29 +00:00
Zuul
131b76147c Merge "update-constraints: Install pip for all versions" 2020-07-14 14:43:17 +00:00
Thierry Carrez
baa350ef23 maintain-github-mirror: add requests dependency
maintain-github-mirror makes direct use of the requests module. That
module is currently installed as a dependency of PyGithub, but that's
brittle, as PyGithub may well opt for a different library in the
future.

Let's proactively fix that before it bites us.

Change-Id: I8fc288e6c3978e20ddf48f905968931f89834e53
2020-07-13 17:07:07 +02:00
Andreas Jaeger
7a8ed31b71 Fix maintain-github-mirror
The script needs yaml, install it.

Change-Id: Ied64412a8814978d0c15acb7bc98f2d66a7ad18f
2020-07-12 08:56:01 +02:00
Sean McGinnis
848d080c1a
update-constraints: Install pip for all versions
Depends-on: https://review.opendev.org/739272

Change-Id: Ic6444e6c186c82a21e7fe2868c27fda3378c187a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-07-10 08:55:50 -05:00
Thierry Carrez
2f534a2d46 maintain-github-mirror: fix repository names
Job definition referenced opendev/governance and opendev/project_config
instead of openstack/governance and openstack/project-config, causing
the job to fail.

Also replace variable names so that they make more sense.

Change-Id: I7cd026a1d0bc1a8ad2d0c6b7bf3a5a0d37ecd423
2020-07-10 11:22:18 +02:00
Thierry Carrez
299017c496 maintain-github-mirror: fix dependency name
The script needs the 'github' module, but this is provided by the
'PyGithub' library on Pypi, not the 'github' library.

Change-Id: I7c6d4b30a11d38e060fb4e6b2cdf29301f6f03e3
2020-07-09 11:15:11 +02:00
Thierry Carrez
0197d9f92a Define maintain-github-openstack-mirror job
Opendev no longer automatically creates repositories on the
GitHub mirror, nor does it update descriptions or closes open PRs.

Add a playbook and a job for periodically maintaining the GitHub
mirror for the 'openstack' organization:

- updating descriptions based on Gerrit project descriptions
- creating on GitHub newly-added openstack repositories
- archiving from GitHub recently-retired openstack repositories
- closing any open PR with a healpful message

This job makes use of a GitHub API token (from the openstack-mirroring
user) and is defined to run periodically on project-config.

Change-Id: Ic02f436eb655dcbe84824b304ea2933e16312e67
2020-07-08 15:42:06 +02:00
Sean McGinnis
118a7ba3d4
Make tox global for update proposal jobs
If00afb50c37b45aefbb45070da4efef3e43e62b2 updated the requirements
update job to use tox as part of the update checking script. We have
ensure-tox as one of the roles in the pre playbook for this job, but it
is not set to be made available globally, which results in the script
failing with being unable to find tox.

This updates the role inclusion to set ensure_global_symlinks: true so
tox will be available in the script.

Change-Id: I5d3ed4ce62cd8df604802fedf522d644d5664698
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-06-24 10:34:18 -05:00
Zuul
0ff9b37818 Merge "Fix venv use in requirements propose_update.sh" 2020-06-23 16:04:49 +00:00
Sean McGinnis
bfc06ae553
Fix venv use in requirements propose_update.sh
This script creates a virtualenv to run a requirements script to update
constraints. virtualenv is not globally available anymore. Rather than
trying to call it directly, this now uses tox to set up the virtual
environment.

Change-Id: If00afb50c37b45aefbb45070da4efef3e43e62b2
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-06-23 07:51:44 -05:00
Jens Harbott
540bdac7ff Fix wheel release playbook
Somehow an empty role invocation without setting the needed variable
ended up there.

Change-Id: I0a2ab8e4c119f1efcaf7e4231efb1804301637d8
2020-06-23 14:50:49 +02:00
Jeremy Stanley
3f72ab620c Add ensure-tox to pti-python-tarball
The publish-openstack-python-branch-tarball job run by many projects
uses tox (in python-branch-tarball/post.yaml), so needs it installed.

Change-Id: Iba219ad14523ce900d9562ef56c78cc6c15aa01d
2020-06-18 21:43:32 +00:00
Andreas Jaeger
6376aa1341 Fix translation jobs
The translation jobs use scripts that expect tox pre-installed, install
tox globally for them.

Change-Id: I633c868a96721f38f063f8e057b0b3c950d5d3e9
2020-06-18 20:05:02 +02:00
Sean McGinnis
9c91006304 Ensure pip is installed for propose-update-constraints
Pip is used by this task, make sure ensure-pip is used so it is
present.

While here, install git-review to a venv and symlink it globally to
keep things separate.

Change-Id: If6b1abd2545c68f727bddc5317797236f451b67c
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Signed-off-by: Ian Wienand <iwienand@redhat.com>
2020-06-17 16:02:46 +10:00
Zuul
6f2b723a9c Merge "Install git-review under python3 for proposed updates" 2020-06-11 23:45:48 +00:00
Ian Wienand
92b378cc9e Don't release centos7 aarch64 volume
We ended up not creating centos7 arm64 wheels and don't have this
volume, don't try and release it.

Change-Id: Ibe4d459ce6c9c6e26330944f30272925b930ba3d
2020-06-11 11:12:05 +10:00
Clark Boylan
8ee0a3a869 Install git-review under python3 for proposed updates
We were installing git-review under python2 in order to propose updates
to Gerrit. git-review happily runs under python3 so update the
installation to avoid python2.

Change-Id: I1671345d71cd60f6508a2c067c4e308680234afe
2020-06-10 15:12:34 -07:00
Andreas Jaeger
b59a59da35 Use python 3 venv for generate-constraints
With change I15d451f6a832b2e698b28e7351a36ef7aea92abe
generate-constraints needs python 3, setup the venv using python 3.

Change-Id: I3dba35267dc6f344ad27910c7dc18c1a33abf4ba
2020-06-07 08:39:19 +02:00
Ian Wienand
6c001b528c Move wheel build role/job outside project-config
We can not speculatively test changes to the wheel build jobs while it
is project config.  These jobs are in the gate for the requirements
project, which run them when bindep.txt changes.

https://review.opendev.org/731394 moved the build-wheel-mirror-base
job/playbook/role into openstack-zuul-jobs, renamed as
build-wheel-cache-* to a) distinguish the name and b) make it a little
clearer that we're building the wheels that goes on the mirror, rather
than passing through to mirroring something existing.

This updates the job names in project-config to reference these new
jobs.

Note the publishing steps are kept here along with the AFS secret.

Depends-On: https://review.opendev.org/732085
Depends-On: https://review.opendev.org/732087
Depends-On: https://review.opendev.org/732083
Depends-On: https://review.opendev.org/732084
Depends-On: https://review.opendev.org/732086

Change-Id: Iac3a906803177e8365f4cfb611800b5ccaed4a6e
2020-06-01 20:10:42 +10:00
Andreas Jaeger
8b09e0cf98 Add root_marker_dir for releasenotes publish
To use the write-root-marker role from opendev/base-jobs, set
the root_marker_dir to the same value that is currently set
by the openstack-zuul-jobs role.

Needed-By: https://review.opendev.org/731460
Change-Id: I169f069aa458bc7d02f9c0295702c1316fff825f
2020-05-28 10:32:58 +02:00
Zuul
463eb880ac Merge "build and publish wheels for aarch64 architecture" 2020-05-27 14:46:11 +00:00
Zuul
8d307eb8ca Merge "build and publish wheels for more distributions (x86-64)" 2020-05-27 14:46:09 +00:00
Marcin Juszkiewicz
541daaa3ff build and publish wheels for aarch64 architecture
- CentOS 7, 8
- Debian 'buster'
- Ubuntu 'focal', 'bionic', 'xenial'

Change-Id: Iebb1062269011664411691ef8fd55219afaccd1e
2020-05-27 14:30:44 +10:00
Ian Wienand
8892414842 Match Debian in AFS slug generation
So we can build/publish Debian wheels

Change-Id: Idac3455109f7d1eddbd50ffa0535e873e6eee2db
2020-05-27 10:20:00 +10:00
Andreas Jaeger
8e607896c4 Fix sphinx playbook: install-if-python was renamed
The new name is ensure-if-python, use it to unbreak the sphinx job.

Change-Id: If03c299eea48aa73e0ef320d0f4903b2effb3ba1
2020-05-26 15:42:45 +02:00
Marcin Juszkiewicz
3445673eea build and publish wheels for more distributions (x86-64)
- CentOS 8
- Debian 'buster'
- Ubuntu 'focal'

Change-Id: I767359d48e04473fad0992b35c804ed511cfaa62
2020-05-22 15:44:17 +02:00
Zuul
2e9c2c6fd0 Merge "install python3.8 now that victoria is in development" 2020-05-17 18:59:57 +00:00
Matthew Thode
dc799bd092
install python3.8 now that victoria is in development
Change-Id: I7758b3879215f8977be82bba273e771d84cd3155
2020-05-17 13:32:40 -05:00
Matthew Thode
5b69bb5edd
drop python2.7 from generate-constraints
Change-Id: I5e51b46afbc26332368525f61a84ff9bd0536a1e
2020-05-15 18:36:42 -05:00
Andreas Jaeger
c6d2d35efc Finish retiring x/pbrx
Depends-On: https://review.opendev.org/726462
Change-Id: I0a5b2a352cdbf443976b18db4e4d210e4c7af581
2020-05-08 21:32:16 +02:00
Andreas Jaeger
cdc4871cae Fix typo in playbooks/yaml2ical/post.yaml
Fix:
  rsync: mkdir "/src/yaml2ical/calendars" failed: No such file or directory (2)

It's /srv, not /src as a few lines above.

Change-Id: I86326fa7de68a0db7e6893e8ab291ba33fe6018d
2020-05-02 16:36:47 +02:00
Andreas Jaeger
2808da5bad Fix irc-meetings publishing
The periodic job fails, example:
https://zuul.opendev.org/t/openstack/build/85f380fb6dde462e85a09bc9b5105ba9/

error is:
rsync: change_dir "/home/zuul//src/opendev.org/openstack/project-config/output" failed: No such file or directory (2)

Problem is that the output directory is in the irc-meetings directory
instead.

Use correct path.

Change-Id: I8e15cbfc9a0c01066de68730709c8c30b234c5f5
2020-05-02 09:24:50 +02:00
Zuul
d2be1e943f Merge "Use zuul deployment keys for yaml2ical" 2020-04-25 15:16:58 +00:00
Sorin Sbarnea
5ea7f35c0a Enable promote to unarchive gz archives in addition to bz2
This should enable us to change archive types.

Change-Id: Id79a4ece4eb9e658afc75cb653bd5e02c3f47d0d
Needed-By: https://review.opendev.org/#/c/715028/
2020-04-22 09:08:07 +01:00
Monty Taylor
295224c41b Use zuul deployment keys for yaml2ical
We're currently adding the private key of a jenkins user. But we
don't use jenkins, so this is lame. Use the zuul deploy key for
project-config instead.

This needs to run as project-config so that per-project deploy
keys work. To do that, shift it from being triggered by irc-meetings
to being triggered hourly by project-config.

As a followup, we should probably convert this to just publish to
AFS and serve the content from there.

Depends-On: https://review.opendev.org/721098
Change-Id: I7874ef46a616e8fd68cf8d95afc3928d3440ba51
2020-04-21 12:40:25 -05:00
Matthew Thode
45ed2e8fa8
add py38 to constraint bot so victoria work can begin
after requirements branches we will remove py27 and py37

Change-Id: I5fde605634108dd628df546a2318731e1832b6b3
Signed-off-by: Matthew Thode <mthode@mthode.org>
2020-04-12 11:24:02 -05:00
Andreas Jaeger
0aedb7733c Use ensure-* roles
The install- roles have been renamed to ensure-*, follow the rename.

Depends-On: https://review.opendev.org/716698/
Change-Id: I2522028e9091efdbe292529474de7cce2cf29039
2020-04-06 19:49:19 +02:00
Ian Wienand
0326363e48 Update ansible-lint and add fake zuul_return
- Use zuul_return fake module to make ansible lint happy, this allows
  to remove Zuul import.
- While ansible-lint 4.2.0 is now able to detect playbooks/roles, this
  is broken, so don't use it
- move its config out of tox.ini so it can be used by any tools, or
  without tox

Change-Id: Ie8935f47db855647e19ae091044e5ac1871f1551
Co-Authored-By: Sorin Sbarnea <ssbarnea@redhat.com>
Co-Authored-By: Andreas Jaeger <aj@suse.com>
2020-03-18 22:28:10 +01:00
Andreas Jaeger
f1acd97b9d Fix translation-source location
The previous change published to
https://tarballs.opendev.org/translation-source/

but we should publish to
https://tarballs.opendev.org/openstack/translation-source/ .

Fix it. We should remove the files from
https://tarballs.opendev.org/translation-source/ afterwards.

Change-Id: I46a857800326db2ad06be276da65279852cc4a90
2020-03-06 07:11:16 +01:00
Clark Boylan
2e12d5df0b Fix afs publishing of translation sources
There are at least two problems here. The first is we can't create an
afs token because we don't have access to the afs secret. The old ssh
based secret was removed by:

  https://review.opendev.org/#/c/706734/2/zuul.d/jobs.yaml

Fix that by adding the new opendev tarballs secret to the job.

Next we need to prefix translation sources with translation-source/ in
the filesystem path. This requires us to write out to a different afs
path than normal. To address that we add a new playbook that is aware of
this different path.

Change-Id: I8b49e4d91600377c98e003b806a13fc89d2614ec
2020-03-05 13:18:29 -08:00
Ian Wienand
cb982781d0 Remove base-ozj job
I do not believe this is used any more.

Story: #2006598
Task: #37735
Change-Id: If8ee6fbbb67c6d1904ba78ce7a9c84fd1e2d7b92
2020-03-04 16:09:25 +11:00
Ian Wienand
391dc9388a Remove base-minimal job
I believe this is now replaced by the base-minimal job from
opendev/base-jobs [1].  This hasn't been updated to upload logs to
swift, so it hasn't been used in quite some time.

[1] 9b235be0c0/zuul.d/jobs.yaml (L96)

Story: #2006598
Task: #37735
Change-Id: I097142dba378b7913e796d06c5a393d86e9e0bbf
2020-03-04 16:09:25 +11:00
Zuul
041bb832bb Merge "Move stackviz publish in-tree" 2020-03-04 04:13:37 +00:00
Andreas Jaeger
fd782318e4 Remove site_specs and playbooks/publish/service-types.yaml
playbooks/publish/service-types.yaml is not used anymore, with AFS
publishing it got obsoleted and can now be removed.

site_specs and site_specs_promote are not used anymore, remove them.

Change-Id: I0c3ca2455b4f8b721d48d44780dbc397ddc9cb20
2020-03-03 21:30:26 +01:00
Andreas Jaeger
15e763dc6a Move stackviz publish in-tree
The new job publish-openstack-stackviz-element is working now,
move the trigger in-tree and change the publish location, we do not need
the "-new" in the name anymore.

Content is published at:
https://tarballs.opendev.org/openstack/stackviz/dist/
The tarballs are equal, only difference is filenames of sources.

Mark the job to run only on master, since it publishes always to the
same location and each invocation overrides the file. This is a
safety measure.

Depends-On: https://review.opendev.org/710754
Change-Id: If9488d7c60b58f51a7a0a4bdf0740d16f61c241e
2020-03-03 16:03:24 +01:00
Andreas Jaeger
8cadacce53 Don't use zuul_work_dir in post-stackviz
The stackviz publish job fails with
"The task includes an option with an undefined variable. The error was:
'zuul_work_dir' is undefined"

Use zuul.project.src.dir instead of zuul_work_dir.

Change-Id: I443d0b4bfb30a06d68321d402802ae1b433bab6b
2020-03-02 10:43:46 +01:00
Andreas Jaeger
14ccbb2e84 Rework post-stackviz to fix uploading
The fetch-javascript-output does not work with the artifact uploading,
it does not create an artifact directory on the server.

Remove use of the role and instead copy the needed bits of
fetch-javascript-content-tarball into the playbook.

Change-Id: I3ed46c1acf023b6d2b323f56ce6bdfe07f70775f
2020-02-29 09:57:45 +01:00
Andreas Jaeger
41e5721d38 Rework stackviz elements publishing
We cannot change the javascript-content job as desired, use
specific post job to rename and fetch the tarball.

Change-Id: I63872db707bd674ebfaa9bd49a1cdf3ae6906a60
2020-02-28 17:56:04 +01:00