openstack/project-config
Code Issues Proposed changes
2da8eb7f10
project-config/zuul/openstack_functions.py

55 lines
2.2 KiB
Python
Raw Normal View History

Make all devstack-vm jobs single-use Use the new gearman-plugin single-use feature with all 'devstack-vm' jobs. This is compatible with the current devstack-gate scripts, though once this is in place, they can be updated so that they no longer perform relabel operations. This is also compatible with future devstack-gate nodepool operations. Change-Id: I6fcadb47ba5eeead3d11926019b3c041305c95f7
2013-08-12 13:27:50 -07:00
# Copyright 2013 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
Explicitly set node label for every zuul job. In order to facilitate the transition from precise to trusty for Juno testing we need to make zuul the decider of node labels for every job. Do this by refactoring how we apply parameter functions to zuul jobs. Since we can have only one parameter function per job it is easier to do decision logic in a single global parameter function. Add this single function and have it maintain backward compatbility with old parameter function selections. But also make it select the ZUUL_NODE for every job. Change-Id: I42fa26fc42c7dc4e94d3a7382cc8de0a5e6d1f36
2014-06-19 10:41:35 -07:00
import re
Fix pep8 errors in modules dir There weren't that many of them - just fix them. Change-Id: I6aa6d056746c4a6804146b89c12e6313f9d22d9a
2013-09-20 14:58:20 -07:00
Switch Zuul/JJB to use LOG_PATH This calculates the full LOG_PATH in the Zuul config and pases it to Jenkins. The new path is similar to the old but with the substitution of a short ZUUL_UUID at the end instead of the Jenkins build number in order to avoid collisions from multiple Jenkins masters. Periodic jobs add a node name to their log path to avoid collisions from multiple masters. Unfortunately, that value is not accessible to the logstash worker. This can be solved by having Zuul trigger periodic jobs. Add the ZUUL_REF to logstash as "build_ref". Requires https://review.openstack.org/#/c/39130/ Change-Id: I40bad59e3ad8ed6b4706762ed8b833fd15c13b0d
2013-07-29 10:46:53 -07:00
def set_log_url(item, job, params):
Set the Zuul/Jenkins base log URL in a function And use that when constructing log paths and URLs. Use a substr of the change id or commit sha when constructing URLs so the log directories are deeper. Make copying the test results a macro (it's used in several places). Update the gearman log client to take advantage of the new parameter. Requires https://review.openstack.org/#/c/36304/ Change-Id: I64faa35eddc4105271efa3de4f83b608b77655c2
2013-07-09 10:53:27 -07:00
if hasattr(item.change, 'refspec'):
zuul's log path should always be a folder zuul sets the path for where logs are stored. In all our cases this is a folder containing the console.log and other assets. We should denote this is a folder by leaving a trailing slash. This also means that os-loganalyze can determine if it should load an index.html faster by looking for a trailing slash. Change-Id: Ia1de627c01ee43c461d25b184fb61cbfd91fb19d
2014-12-12 14:54:41 +11:00
path = "%s/%s/%s/%s/" % (
Set the Zuul/Jenkins base log URL in a function And use that when constructing log paths and URLs. Use a substr of the change id or commit sha when constructing URLs so the log directories are deeper. Make copying the test results a macro (it's used in several places). Update the gearman log client to take advantage of the new parameter. Requires https://review.openstack.org/#/c/36304/ Change-Id: I64faa35eddc4105271efa3de4f83b608b77655c2
2013-07-09 10:53:27 -07:00
params['ZUUL_CHANGE'][-2:], params['ZUUL_CHANGE'],
params['ZUUL_PATCHSET'], params['ZUUL_PIPELINE'])
Support periodic jobs in log custom function Change-Id: Ice4db15fd16e0eea886ddc5f793d49693a0a9986
2013-08-01 15:54:12 -07:00
elif hasattr(item.change, 'ref'):
zuul's log path should always be a folder zuul sets the path for where logs are stored. In all our cases this is a folder containing the console.log and other assets. We should denote this is a folder by leaving a trailing slash. This also means that os-loganalyze can determine if it should load an index.html faster by looking for a trailing slash. Change-Id: Ia1de627c01ee43c461d25b184fb61cbfd91fb19d
2014-12-12 14:54:41 +11:00
path = "%s/%s/%s/" % (
Set the Zuul/Jenkins base log URL in a function And use that when constructing log paths and URLs. Use a substr of the change id or commit sha when constructing URLs so the log directories are deeper. Make copying the test results a macro (it's used in several places). Update the gearman log client to take advantage of the new parameter. Requires https://review.openstack.org/#/c/36304/ Change-Id: I64faa35eddc4105271efa3de4f83b608b77655c2
2013-07-09 10:53:27 -07:00
params['ZUUL_NEWREV'][:2], params['ZUUL_NEWREV'],
params['ZUUL_PIPELINE'])
Support periodic jobs in log custom function Change-Id: Ice4db15fd16e0eea886ddc5f793d49693a0a9986
2013-08-01 15:54:12 -07:00
else:
Fix zuul log path again In I97362b1cc6a8c19b54903e53638aba8cb4928e19 an extra '/' was removed from most instances of BASE_LOG_PATH, except for ones generated for changes in periodic pipelines. Add the missing '/' in that case. Change-Id: Ibd99b88651d5cc3a7627fd4843a9a24f9be68464
2015-03-10 11:55:24 -07:00
path = params['ZUUL_PIPELINE'] + '/'
Set the Zuul/Jenkins base log URL in a function And use that when constructing log paths and URLs. Use a substr of the change id or commit sha when constructing URLs so the log directories are deeper. Make copying the test results a macro (it's used in several places). Update the gearman log client to take advantage of the new parameter. Requires https://review.openstack.org/#/c/36304/ Change-Id: I64faa35eddc4105271efa3de4f83b608b77655c2
2013-07-09 10:53:27 -07:00
params['BASE_LOG_PATH'] = path
Fix the log path returned from zuul Logs are currently coming back from zuul with two slashes in them. The paths already have a trailing slash so the final LOG_PATH doesn't need to add it. Change-Id: I97362b1cc6a8c19b54903e53638aba8cb4928e19
2014-12-17 12:09:59 +11:00
params['LOG_PATH'] = path + '%s/%s/' % (job.name,
params['ZUUL_UUID'][:7])
Make all devstack-vm jobs single-use Use the new gearman-plugin single-use feature with all 'devstack-vm' jobs. This is compatible with the current devstack-gate scripts, though once this is in place, they can be updated so that they no longer perform relabel operations. This is also compatible with future devstack-gate nodepool operations. Change-Id: I6fcadb47ba5eeead3d11926019b3c041305c95f7
2013-08-12 13:27:50 -07:00
Explicitly set node label for every zuul job. In order to facilitate the transition from precise to trusty for Juno testing we need to make zuul the decider of node labels for every job. Do this by refactoring how we apply parameter functions to zuul jobs. Since we can have only one parameter function per job it is easier to do decision logic in a single global parameter function. Add this single function and have it maintain backward compatbility with old parameter function selections. But also make it select the ZUUL_NODE for every job. Change-Id: I42fa26fc42c7dc4e94d3a7382cc8de0a5e6d1f36
2014-06-19 10:41:35 -07:00
def reusable_node(item, job, params):
if 'OFFLINE_NODE_WHEN_COMPLETE' in params:
del params['OFFLINE_NODE_WHEN_COMPLETE']
Correct set_node_options argument list With no longer setting a node default, Zuul won't pass it as a parameter to the set_node_options() parameter function. Drop it from the required arguments for this function. Change-Id: I2aa8f7179ecb0bdf6bd8785653f1726321bbcb7f
2015-12-22 06:03:57 +00:00
def set_node_options(item, job, params):
Have Zuul force tox to obey Zuul TOX_TESTENV_PASSENV is a variable that lists env variables that tox should allow through to test environments. Make sure that the ZUUL_ variables are always on the list so that zuul-cloner may always be used inside of a tox environment. Change-Id: Iacba1611e5c50fbccd6ae700b3456028b142fc5c
2016-01-07 17:29:18 -08:00
# Force tox to pass through ZUUL_ variables
zuul_params = [x for x in params.keys() if x.startswith('ZUUL_')]
Correct tox env override TOX_TESTENV_PASSENV is space separated, not comma separated. Change-Id: I4bd3d2b61d27772bb88aa7911220baba0a08620c
2016-01-08 12:08:33 -08:00
params['TOX_TESTENV_PASSENV'] = ' '.join(zuul_params)
Fix typo in word parameter Change-Id: I08d998716881fb43651e43c182ce93e5f61cc953
2015-10-30 17:36:39 +02:00
# Set up log url parameter for all jobs
Make all devstack-vm jobs single-use Use the new gearman-plugin single-use feature with all 'devstack-vm' jobs. This is compatible with the current devstack-gate scripts, though once this is in place, they can be updated so that they no longer perform relabel operations. This is also compatible with future devstack-gate nodepool operations. Change-Id: I6fcadb47ba5eeead3d11926019b3c041305c95f7
2013-08-12 13:27:50 -07:00
set_log_url(item, job, params)
Explicitly set node label for every zuul job. In order to facilitate the transition from precise to trusty for Juno testing we need to make zuul the decider of node labels for every job. Do this by refactoring how we apply parameter functions to zuul jobs. Since we can have only one parameter function per job it is easier to do decision logic in a single global parameter function. Add this single function and have it maintain backward compatbility with old parameter function selections. But also make it select the ZUUL_NODE for every job. Change-Id: I42fa26fc42c7dc4e94d3a7382cc8de0a5e6d1f36
2014-06-19 10:41:35 -07:00
# Default to single use node. Potentially overriden below.
# Select node to run job on.
Make all devstack-vm jobs single-use Use the new gearman-plugin single-use feature with all 'devstack-vm' jobs. This is compatible with the current devstack-gate scripts, though once this is in place, they can be updated so that they no longer perform relabel operations. This is also compatible with future devstack-gate nodepool operations. Change-Id: I6fcadb47ba5eeead3d11926019b3c041305c95f7
2013-08-12 13:27:50 -07:00
params['OFFLINE_NODE_WHEN_COMPLETE'] = '1'
Run translation liberty jobs Enhance regex to handle the translation liberty jobs like nova-propose-translation-update-liberty. Change-Id: I592443baa981b44437e09209b933f69a45a85b7c
2015-09-25 08:10:18 +02:00
proposal_re = r'^.*(merge-release-tags|(propose|upstream)-(.*?)-(constraints-.*|updates?|update-liberty))$' # noqa
Run npm upload job on relase slave We need to be explicit that we want the npm upload job to run on the release slave otherwise gearman will attempt to run it on the default slave label which will fail because that job won't be registered. To do this rename the job so that it matches existing convention which makes the regex easier to read/write. Then update the reuseable slave regex to include the job which will force it to run on release slave without offlining the slave. Change-Id: I26355680f2d8fc3fa5f514370d1586deb701c21c
2015-10-07 15:56:50 -07:00
release_re = r'^.*-(forge|jenkinsci|mavencentral|pypi-(both|wheel)|npm)-upload$'
Move hook-{name}-rtfd to proposal node Our RTFD hook just does a curl request to upstream. As such, it is very expensive to nodepool to build a node for the purpose of just using curl. Lets move the job to use the proposal node which is long living. Change-Id: I8f407fdaeb890a4ae6a9b99caefe836b8f2f7819 Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-07-28 12:28:50 -04:00
hook_re = r'^hook-(.*?)-(rtfd)$'
Remove precise/trusty label mapping in Zuul config Now that there are no jobs declaring both .*-precise and .*-trusty nodes at the same time, it is safe to remove the associated mapping in Zuul and let it trust that any worker registering the job name is capable of running it sanely. This should not be approved until manual jenkins-jobs updates have been completed and confirmed successful on all Jenkins masters. Change-Id: I3b40021ffbf89842ed3a53412dfffa119f311797
2015-12-21 20:58:43 +00:00
# jobs run on the persistent proposal and release workers
Move hook-{name}-rtfd to proposal node Our RTFD hook just does a curl request to upstream. As such, it is very expensive to nodepool to build a node for the purpose of just using curl. Lets move the job to use the proposal node which is long living. Change-Id: I8f407fdaeb890a4ae6a9b99caefe836b8f2f7819 Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-07-28 12:28:50 -04:00
if (re.match(proposal_re, job.name) or re.match(release_re, job.name) or
re.match(hook_re, job.name)):
Explicitly set node label for every zuul job. In order to facilitate the transition from precise to trusty for Juno testing we need to make zuul the decider of node labels for every job. Do this by refactoring how we apply parameter functions to zuul jobs. Since we can have only one parameter function per job it is easier to do decision logic in a single global parameter function. Add this single function and have it maintain backward compatbility with old parameter function selections. But also make it select the ZUUL_NODE for every job. Change-Id: I42fa26fc42c7dc4e94d3a7382cc8de0a5e6d1f36
2014-06-19 10:41:35 -07:00
reusable_node(item, job, params)
Copy Permalink