213 Commits

Author SHA1 Message Date
Doug Hellmann
a1df5de264 prune dead branches when syncing with origin
The release scripts rely on having an actually up to date version of
the repo being tagged or branched. Since the repos in the CI images
may have branches that have been deleted in the upstream repository,
we need to add the --prune option when we fetch any updates to cause
those branches to be removed locally.

Change-Id: I806283102be4de5ae865ca67978216a76de96a71
Related-commit: I25848275f90ec440e85765f87a23cd1413aab6cb
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-08-16 15:44:31 -04:00
Andreas Jaeger
989103cfde Remove eol tagging from eol_branch.sh
Tagging at eol is now done in the releases repository, thus remove
tagging from the script.

Change-Id: I9e91fb030b42cf8b238313c27d11da593a77b851
2018-08-06 19:17:53 +02:00
Andreas Jaeger
18aea9c6aa Import eol_branch.sh
Import this script from now retired openstack-infra/release-tools
repository. It is run manually at eol time to retire a branch.

Change-Id: Idcffdc5c5c540bd73ee7ee3e5be9274d693d5f1f
2018-08-06 19:12:51 +02:00
Zuul
c41eb578d8 Merge "[translation] doc generatepot jobs for 3 projects" 2018-06-26 10:30:03 +00:00
Zuul
5f91596c90 Merge "be more resilient in make_branch.sh for retired repositories" 2018-06-26 06:00:59 +00:00
Doug Hellmann
2cefc3550c use python3 to run launchpad commenting script
Use python3 when running the script to comment on launchpad tickets as
part of the release process. The depends-on patch adds the python3
version of launchpadlib via bindep.

Change-Id: I189d83462381723bb8309c248d0ace21ae6f30c1
Story: #2001691
Task: #22329
Depends-On: https://review.openstack.org/577290
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-06-21 16:30:26 -04:00
Zuul
a74aef67c2 Merge "Just use chmod instead of file for log permissions" 2018-06-18 13:51:56 +00:00
Monty Taylor
d45f691599
Just use chmod instead of file for log permissions
The file module with recurse goes to the bad place when the content has
a symlink to '.'. Rather than fighting it - just use chmod -R, which
does what we want.

Change-Id: Ied4cc2a7e252db555ffdc3f1fd47edaf93b666da
2018-06-18 08:15:45 -05:00
Doug Hellmann
124973271c fix the check-python-release role
The instructions for setting up the virtualenv were wrong, leading to
it being unusable. This patch fixes the setup and adds a variable to
control the location of the virtualenv, to mirror the way
ensure-sphinx and some of our other roles work.

Change-Id: I2d8d443ec1fb51d58e70ca0b7837e8ccf586513c
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-06-15 14:36:40 -04:00
Zuul
e297b4357d Merge "add a job to check the metadata for python packages" 2018-06-14 17:52:05 +00:00
Monty Taylor
8f8c6774e1
Don't follow symlinks when setting log permissions
Ansible 2.5 changed the default value of follow to true.
We don't want to follow symlinks here, so set follow to false directly.

Needed-By: https://review.openstack.org/551989/
Change-Id: I858f328118a301d648bef485cc606a7fd6364ada
2018-06-14 09:06:45 -05:00
Zuul
0a8818359c Merge "Build wheels against constraints" 2018-06-13 20:46:04 +00:00
Tony Breeds
63fda045cd Skip released that are close but not fully EOLd when building wheels
The requirements project is very slow to actually tag branches as EOLd as
that can cause some testing environments and devstack to fail.  For
example if tox.ini refers to a constraints file in git[1].

However building wheels for those releases isn't actually helpful as
they're not chnaging *and* can cause failed runs as pypi etc move on.

This change adds an regex for the closed branches to exclude them from
building.  The regex is a little more complex than the single case
(newton) requires but allows for easier future additions.

[1] http://git.openstack.org/cgit/openstack/nova/tree/tox.ini?h=newton-eol#n12

Change-Id: Ia0c589989cb6e7e81e7b08e39d81dac78c55bf18
2018-06-13 14:45:14 +10:00
Ian Wienand
fe43d6d81b Build wheels against constraints
I was looking at wheel build failures, and in some cases builds fail
because they're not constrained to versions already in the
constraints.

For example; pylxd failed to build a wheel [1] due to [2], but
upper-constraints.txt would have kept urllib to 1.22.  Similarly
several django plugins fail on python2 as they try to bring in Django2
which only works with python3.

Pass the upper-constraints.txt to the wheel build to ensure consistent
building.

[1] http://logs.openstack.org/periodic/git.openstack.org/openstack/requirements/master/publish-wheel-mirror-centos-7/723e75b/python2/failed.txt

[2] https://github.com/requests/requests/issues/4673

Change-Id: I788994b69afd2769489454a3b16f84bea4c56e59
2018-06-13 14:30:07 +10:00
Doug Hellmann
60d1cd0808 add a job to check the metadata for python packages
Change-Id: I4900089d36b114c5a70d8598a9be90a2ab6a73af
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-06-12 16:57:18 -04:00
Doug Hellmann
65d3be0cdb be more resilient in make_branch.sh for retired repositories
Retired repositories may not have a .gitreview file in master, so we
want to check out the tag we are given to branch from instead. There's
no point in setting up git-review if we aren't going to create the
branch, so move the setup logic after the test for the existing
branch.

Change-Id: Ie8c69a9729e5e0d470c8a9292da999e275c9611a
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-06-12 10:37:56 -04:00
Ian Y. Choi
de0fd555aa [translation] doc generatepot jobs for 3 projects
This commit adds horizon, openstack-ansible, and openstack-helm to test
project doc translation generation jobs by adding openstack-ansible
and openstack-helm to project translation job and dealing with
pot files.

Since project doc translation structure is generally divided into
multiple directories such as admin, user, install, and contributor,
it is a good idea to split into different pot files and manage as
different documents in Zanata.

Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
Co-Authored-By: Frank Kloeker <f.kloeker@telekom.de>

Change-Id: I84a65f375dbd73f52d7b5d88218f92c53b99a2e9
Implements: blueprint project-doc-translation-support
2018-06-09 00:40:33 +09:00
Zuul
e9de791b5f Merge "ignore the series_status.yaml when processing release requests" 2018-04-27 09:41:46 +00:00
Doug Hellmann
394cf0c4e3 ignore the series_status.yaml when processing release requests
The series_status.yaml file does not contain release data so
process_release_requests.py should ignore it.

Change-Id: I661c5a10b0497fd9112daea6d4737a5d15dc8b79
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-04-26 09:05:53 -04:00
Doug Hellmann
c83de06ad1 update the release tools to mark newton as closed
This prevents the release tools from processing changes in the newton
branch, including re-creating newton branches that have been deleted.

Story: #2001852
Task: #14346
Change-Id: I5de04f1bbea61b00fda4d312aea77b1ea4efaa7e
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-04-19 16:37:49 -04:00
Zuul
86c88c12f2 Merge "Catch errors in release processing script" 2018-03-28 17:39:13 +00:00
Doug Hellmann
9e269c347e improve error handling of clone_repo
Do not explicitly return 0 at the end of the function in case set -e
doesn't work inside the function the way we expect.

Use git -C instead of a subshell.

Remove an unused variable.

Change-Id: I166563cb6e84dd5af38e11b2f69189bbf6befd0e
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-03-28 09:26:39 -04:00
Doug Hellmann
bf4d1c1677 check out the series branch when tagging a release
When a project is retired, the .gitreview file is removed. Without
that file, "git review -s" (run during cloning) fails, and the tag job
ends up failing.

This patch passes an assumed branch name to clone_repo.sh so that if
the stable branch for the series exists we get it. If the stable
branch does not exist, clone_repo.sh falls back to checking out
master.

Change-Id: Ib1221b5672c518eb8e3037232440807af6c90412
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-03-28 09:26:39 -04:00
Thierry Carrez
9210444245 Catch errors in release processing script
The releases playbook calls process_release_requests.sh, which
calls process_release_requests.py, which calls release.sh and
make_branch.sh as necessary.

If an error happens during release.sh or make_branch.sh, the error
is propagated back to process_release_requests.py, but that script
failed to propagate the error back to process_release_requests.sh.

This could result in the job appearing to complete successfully,
while it actually failed to do anything (whatever the reason is).

Change-Id: I10e5362e4d14cb9d787cf09f62d0b67f7dce8a03
2018-03-28 15:04:13 +02:00
Zuul
a003b828ea Merge "Test change to upload-logs" 2018-03-27 17:06:35 +00:00
James E. Blair
28a3f190c8 Test change to upload-logs
This tests the change in https://review.openstack.org/556877

Change-Id: I2dfdb858b4ea1ecceabcb6064b3515ad90dbba72
2018-03-27 09:16:43 -07:00
Andreas Jaeger
c0b40b37e0 Silence bashate
The remaining bashate warnings was E042:
 The return value of "local" is always 0; errors in subshells
 used for declaration are thus hidden and will not trigger "set
 -e".

Fix the few cases this triggered so that we don't get confused anymore
about the message.

Remove -v from bashate invocation, we don't need to print out all the
filenames anymore.

Change-Id: I47991a7040c8b9183bc72cce8e5d95b2cec7e6c5
2018-03-26 09:13:55 +02:00
Zuul
7b3158698d Merge "Make the task that remove the remote idempotent" 2018-03-19 04:30:45 +00:00
Doug Hellmann
407fccbc47 do not treat newton as closed to releases
Some teams are still wrapping up their newton EOL work, so we need to
let them tag releases.

Change-Id: I2b49a9cef44896aa9f147e3fbc487b15efed9590
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-03-08 07:48:34 -05:00
Andreas Jaeger
88b52ff134 Fix rsync invocation
Add extra "/" so that files are copied into the destination, not as a
directory in it.

Change-Id: If0fe37f7b405fcdff73331baabe9241a50b3a5b7
2018-03-06 09:24:28 +01:00
Andrea Frittoli
cf1ae5235b Make the task that remove the remote idempotent
The current implementation of the task is not idempotent.
That's fine in CI but when running locally it forces a cleanup
of the environment every time.

Change-Id: Ifdb573bc09ddb2d39674218e9f631e8c15f936cb
2018-03-05 23:29:38 +00:00
Andreas Jaeger
72eed13ec2 Ensure that /home/zuul/scripts/ exist
Ensure that the directory exist for copying, this fixes
a failure with releasing changes.

Change-Id: Ib3e2285a147fd31bc4425ec733affa56cada8a49
2018-03-05 18:35:21 +01:00
Zuul
409d4d38ca Merge "Remove role legacy-copy-project-config-scripts" 2018-03-05 12:39:48 +00:00
Zuul
4fffe3de5b Merge "Move release-tools out of jenkins/scripts" 2018-03-05 12:39:46 +00:00
Andreas Jaeger
0bf3827550 Fix prep-zanata
We download already on the remote system, so tell unarchive that the
content is there already and does not need to copied over.

The role currently fails since ansible tries to copy the file over from
the local host.

Change-Id: I8b879b976b19ab5672a95db35346cdea8467e726
2018-03-02 12:46:01 +01:00
Andreas Jaeger
bcb30b0f7a Fix wrong indent for role prep-zanata
Fix "ERROR: Unsupported parameters for (get_url) module:
delay,register,retries,until."

Change-Id: Ic5d16f32752e26ebf4e1482eb40b18ee0b270adb
2018-03-02 11:53:04 +01:00
Andreas Jaeger
939cb6d41d Update zanata-cli to 4.3.3
The server is now running version 4.3.3, update the client as well.

Change-Id: I68727dc5fe9efef392bbb8250260b2c4524c1153
2018-03-01 09:03:48 +01:00
Andreas Jaeger
2707445b3e Remove role legacy-copy-project-config-scripts
This role is now unused, remove it.

Change-Id: I9696c93a1d545a2d1a79db4c27b4028bf9221fb8
2018-02-27 19:30:17 +01:00
Andreas Jaeger
fbc1f58c30 Move release-tools out of jenkins/scripts
To not install these on the images, create a new scripts directory to
hold them.

Create new copy-release-tools-scripts role to copy the files over,
we do not need using legacy-copy-project-config-scripts anymore since
everything is copied elsewhere.

Add the role to all playbooks that use release-tools scripts.

Change-Id: I36c458e297fec58023799f625739b341c4d403cf
2018-02-27 19:27:35 +01:00
Andreas Jaeger
fca8d78211 Use host_vars for wheel builds
No need to use magic names since we now have host_vars, instead set
wheel_python explicitly.

Remove TODO comment.

Change-Id: I5051a32d843b106b013b94d57fd24b63896b741a
2018-02-17 16:40:04 +01:00
Andreas Jaeger
affdbab218 Consolidate common.sh script copying
Create new role for copying common scripts, this includes for now only
common.sh. Use this for proposal and translation jobs.

In the past we duplicated the file, this allows to have one definition
only.

Change-Id: I72c14341f511fdaa2c67a3758387afc9445869cc
2018-02-15 08:03:40 +01:00
Andreas Jaeger
3a806df6c6 Move wheel-build scripts out of jenkins/scripts
We only need the three wheel-X scripts from jenkins/scripts and need
them only for the playbook and its roles. Move the scripts in the
roles directory and copy them only if needed - using the scripts module.

Remove copying of complete jenkins/script for this playbook using the
role legacy-copy-project-config-scripts, it's not needed anymore.

Change-Id: I2e481fcf7ca148aac8a36ae99d8598ba26078a25
2018-02-13 21:12:45 +01:00
Andreas Jaeger
a03837a2c0 Fix translation scripts
The scripts are run from {{ ansible_user_dir }}/scripts/, so copy to
that place. In this case we do not need sudo access.

See playbooks/translation/upstream-translation.yaml for
invocation.

Change-Id: I792d991ead1217a223e3f81fe9096d6a8ba737d9
2018-02-08 22:24:11 +01:00
Andreas Jaeger
e20c9b0d52 Move translation scripts out of jenkins/scripts
No need to install these files on all nodes. Instead copy them over only
when needed and place them together in the translation directory.

Change-Id: Iebffa4f7b5998f2e1a3ee660f1242a11abf1e3f8
2018-02-07 21:05:17 +01:00
Zuul
54ad1853a2 Merge "Add modified upload-logs as test-upload-logs to base-test" 2018-01-25 21:50:23 +00:00
Monty Taylor
068a754bad
Add modified upload-logs as test-upload-logs to base-test
Change-Id: I645727fb30de6abe325c694d04886711c205e620
2018-01-25 15:04:13 -06:00
Monty Taylor
63404d3d0d Rename fetch-translations-stestr-output
The role name is misleading. It's not fetching stestr output, it's
fetching subunit output.

Add a note to the README explaining what it does and that it's not using
fetch-subunit-output on purpose.

Use  "{{ zuul.project.src_dir }}" instead of
"src/{{ zuul.project.canonical_name }}".

Change-Id: I1b5ab65f9697ec184969ca7899fb40294cdb853c
2018-01-23 13:40:51 +01:00
Clark Boylan
448b7d7768 Index zuul executor
It would be useful to know what zuul executor ran a job so that we can
track potential executor specific issues back to that executor via
logstash. Add the zuul executors hostname to the log processing payload
which will result in us storing it in elasticsearch.

Change-Id: Ifbd7c7f91a006bd63e3df6053297ccc36bb3240b
2018-01-09 15:37:21 -08:00
Andreas Jaeger
707cf6cd82 Improve zanata-cli download
Improve the code as suggested during code review.

Co-Authored-By: David Moreau Simard <dmsimard@redhat.com>
Change-Id: I51a948d0ec590f9e1db2f9f60287a360bb8ca362
2017-12-15 08:08:00 +01:00
Zuul
eb4237bcd8 Merge "Convert back to zuul.projects" 2017-12-13 15:55:45 +00:00