zuul
/
zuul
Code Issues Proposed changes

Merge "Remove internal Swift support" into feature/zuulv3

This commit is contained in:
Jenkins 2017-03-07 23:28:46 +00:00 committed by Gerrit Code Review
parent e7800a008b 92e953a807
commit a981227809
13 changed files with 3 additions and 525 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
doc/source/launchers.rst
View File

@ -11,8 +11,6 @@
.. _`Turbo-Hipster Documentation`:
http://turbo-hipster.rtfd.org/
.. _FormPost: http://docs.openstack.org/developer/swift/misc.html#module-swift.common.middleware.formpost
.. _launchers:
Launchers
@ -138,34 +136,6 @@ And finally a reference being altered::
Your jobs can check whether the parameters are ``000000`` to act
differently on each kind of event.
Swift parameters
~~~~~~~~~~~~~~~~
If swift information has been configured for the job zuul will also
provide signed credentials for the builder to upload results and
assets into containers using the `FormPost`_ middleware.
Each zuul container/instruction set will contain each of the following
parameters where $NAME is the ``name`` defined in the layout.
*SWIFT_$NAME_URL*
The swift destination URL. This will be the entire URL including
the AUTH, container and path prefix (folder).
*SWIFT_$NAME_HMAC_BODY*
The information signed in the HMAC body. The body is as follows::
PATH TO OBJECT PREFIX (excluding domain)
BLANK LINE (zuul implements no form redirect)
MAX FILE SIZE
MAX FILE COUNT
SIGNATURE EXPIRY
*SWIFT_$NAME_SIGNATURE*
The HMAC body signed with the configured key.
*SWIFT_$NAME_LOGSERVER_PREFIX*
The URL to prepend to the object path when returning the results
from a build.
Gearman
-------

112
doc/source/zuul.rst
View File

@ -172,76 +172,6 @@ can be found on the :doc:`merger` page.
Path to PID lock file for the merger process.
``pidfile=/var/run/zuul-merger/merger.pid``
.. _swift:
swift
"""""
To send (optional) swift upload instructions this section must be
present. Multiple destinations can be defined in the :ref:`jobs` section
of the layout.
If you are sending the temp-url-key or fetching the x-storage-url, you
will need the python-swiftclient module installed.
**X-Account-Meta-Temp-Url-Key** (optional)
This is the key used to sign the HMAC message. If you do not set a
key Zuul will generate one automatically.
**Send-Temp-Url-Key** (optional)
Zuul can send the X-Account-Meta-Temp-Url-Key to swift for you if
you have set up the appropriate credentials in ``authurl`` below.
This isn't necessary if you know and have set your
X-Account-Meta-Temp-Url-Key.
If set, Zuul requires the python-swiftclient module.
``default: true``
**X-Storage-Url** (optional)
The storage URL is the destination to upload files into. If you do
not set this the ``authurl`` credentials are used to fetch the url
from swift and Zuul will requires the python-swiftclient module.
**authurl** (optional)
The (keystone) Auth URL for swift.
``For example, https://identity.api.rackspacecloud.com/v2.0/``
This is required if you have Send-Temp-Url-Key set to ``True`` or
if you have not supplied the X-Storage-Url.
Any of the `swiftclient connection parameters`_ can also be defined
here by the same name. Including the os_options by their key name (
``for example tenant_id``)
.. _swiftclient connection parameters: http://docs.openstack.org/developer/python-swiftclient/swiftclient.html#module-swiftclient.client
**region_name** (optional)
The region name holding the swift container
``For example, SYD``
Each destination defined by the :ref:`jobs` will have the following
default values that it may overwrite.
**default_container** (optional)
Container name to place the log into
``For example, logs``
**default_expiry** (optional)
How long the signed destination should be available for
``default: 7200 (2hrs)``
**default_max_file_size** (optional)
The maximum size of an individual file
``default: 104857600 (100MB)``
**default_max_file_count** (optional)
The maximum number of separate files to allow
``default: 10``
**default_logserver_prefix**
Provide a URL to the CDN or logserver app so that a worker knows
what URL to return. The worker should return the logserver_prefix
url and the object path.
``For example: http://logs.example.org/server.app?obj=``
.. _connection:
connection ArbitraryName
@ -786,48 +716,6 @@ each job as it builds a list from the project specification.
**tags (optional)**
A list of arbitrary strings which will be associated with the job.
**swift**
If :ref:`swift` is configured then each job can define a destination
container for the builder to place logs and/or assets into. Multiple
containers can be listed for each job by providing a unique ``name``.
*name*
Set an identifying name for the container. This is used in the
parameter key sent to the builder. For example if it ``logs`` then
one of the parameters sent will be ``SWIFT_logs_CONTAINER``
(case-sensitive).
Each of the defaults defined in :ref:`swift` can be overwritten as:
*container* (optional)
Container name to place the log into
``For example, logs``
*expiry* (optional)
How long the signed destination should be available for
*max-file-size** (optional)
The maximum size of an individual file
*max_file_size** (optional, deprecated)
A deprecated alternate spelling of *max-file-size*.
*max-file-count* (optional)
The maximum number of separate files to allow
*max_file_count* (optional, deprecated)
A deprecated alternate spelling of *max-file-count*.
*logserver-prefix*
Provide a URL to the CDN or logserver app so that a worker knows
what URL to return.
``For example: http://logs.example.org/server.app?obj=``
The worker should return the logserver-prefix url and the object
path as the URL in the results data packet.
*logserver_prefix* (deprecated)
A deprecated alternate spelling of *logserver-prefix*.
Here is an example of setting the failure message for jobs that check
whether a change merges cleanly::

1
test-requirements.txt
View File

@ -6,7 +6,6 @@ sphinxcontrib-blockdiag>=1.1.0
fixtures>=0.3.14
python-keystoneclient>=0.4.2
python-subunit
python-swiftclient>=1.6
testrepository>=0.0.17
testtools>=0.9.32
sphinxcontrib-programoutput

21
tests/base.py
View File

@ -32,7 +32,6 @@ from six import StringIO
import socket
import string
import subprocess
import swiftclient
import sys
import tempfile
import threading
@ -60,7 +59,6 @@ import zuul.webapp
import zuul.rpclistener
import zuul.launcher.server
import zuul.launcher.client
import zuul.lib.swift
import zuul.lib.connections
import zuul.merger.client
import zuul.merger.merger
@ -904,18 +902,6 @@ class FakeSMTP(object):
return True
class FakeSwiftClientConnection(swiftclient.client.Connection):
def post_account(self, headers):
# Do nothing
pass
def get_auth(self):
# Returns endpoint and (unused) auth token
endpoint = os.path.join('https://storage.example.org', 'V1',
'AUTH_account')
return endpoint, ''
class FakeNodepool(object):
REQUEST_ROOT = '/nodepool/requests'
NODE_ROOT = '/nodepool/nodes'
@ -1332,11 +1318,6 @@ class ZuulTestCase(BaseTestCase):
self.sched = zuul.scheduler.Scheduler(self.config)
self.useFixture(fixtures.MonkeyPatch('swiftclient.client.Connection',
FakeSwiftClientConnection))
self.swift = zuul.lib.swift.Swift(self.config)
self.event_queues = [
self.sched.result_event_queue,
self.sched.trigger_event_queue,
@ -1367,7 +1348,7 @@ class ZuulTestCase(BaseTestCase):
self.builds = self.launch_server.running_builds
self.launch_client = zuul.launcher.client.LaunchClient(
self.config, self.sched, self.swift)
self.config, self.sched)
self.merge_client = zuul.merger.client.MergeClient(
self.config, self.sched)
self.nodepool = zuul.nodepool.Nodepool(self.sched)

59
tests/fixtures/layout-swift.yaml vendored
View File

@ -1,59 +0,0 @@
pipelines:
- name: check
manager: IndependentPipelineManager
trigger:
gerrit:
- event: patchset-created
success:
gerrit:
verified: 1
failure:
gerrit:
verified: -1
- name: post
manager: IndependentPipelineManager
trigger:
gerrit:
- event: ref-updated
ref: ^(?!refs/).*$
- name: gate
manager: DependentPipelineManager
failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
trigger:
gerrit:
- event: comment-added
approval:
- approved: 1
success:
gerrit:
verified: 2
submit: true
failure:
gerrit:
verified: -2
start:
gerrit:
verified: 0
precedence: high
jobs:
- name: ^.*$
swift:
- name: logs
- name: ^.*-merge$
swift:
- name: logs
container: merge_logs
failure-message: Unable to merge change
- name: test-test
swift:
- name: MOSTLY
container: stash
projects:
- name: org/project
gate:
- test-merge
- test-test

10
tests/fixtures/zuul-connections-bad-sql.conf vendored
View File

@ -12,16 +12,6 @@ git_user_email=zuul@example.com
git_user_name=zuul
zuul_url=http://zuul.example.com/p
[swift]
authurl=https://identity.api.example.org/v2.0/
user=username
key=password
tenant_name=" "
default_container=logs
region_name=EXP
logserver_prefix=http://logs.example.org/server.app/
[connection review_gerrit]
driver=gerrit
server=review.example.com

10
tests/fixtures/zuul-connections-multiple-gerrits.conf vendored
View File

@ -15,16 +15,6 @@ zuul_url=http://zuul.example.com/p
[launcher]
git_dir=/tmp/zuul-test/launcher-git
[swift]
authurl=https://identity.api.example.org/v2.0/
user=username
key=password
tenant_name=" "
default_container=logs
region_name=EXP
logserver_prefix=http://logs.example.org/server.app/
[connection review_gerrit]
driver=gerrit
server=review.example.com

10
tests/fixtures/zuul-connections-same-gerrit.conf vendored
View File

@ -15,16 +15,6 @@ zuul_url=http://zuul.example.com/p
[launcher]
git_dir=/tmp/zuul-test/launcher-git
[swift]
authurl=https://identity.api.example.org/v2.0/
user=username
key=password
tenant_name=" "
default_container=logs
region_name=EXP
logserver_prefix=http://logs.example.org/server.app/
[connection review_gerrit]
driver=gerrit
server=review.example.com

46
tests/unit/test_scheduler.py
View File

@ -3583,52 +3583,6 @@ For CI problems and help debugging, contact ci@example.org"""
self.assertNotIn('logs.example.com', B.messages[1])
self.assertNotIn('SKIPPED', B.messages[1])
@skip("Disabled for early v3 development")
def test_swift_instructions(self):
"Test that the correct swift instructions are sent to the workers"
self.updateConfigLayout(
'tests/fixtures/layout-swift.yaml')
self.sched.reconfigure(self.config)
self.registerJobs()
self.launch_server.hold_jobs_in_build = True
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
A.addApproval('code-review', 2)
self.fake_gerrit.addEvent(A.addApproval('approved', 1))
self.waitUntilSettled()
self.assertEqual(
"https://storage.example.org/V1/AUTH_account/merge_logs/1/1/1/"
"gate/test-merge/",
self.builds[0].parameters['SWIFT_logs_URL'][:-7])
self.assertEqual(5,
len(self.builds[0].parameters['SWIFT_logs_HMAC_BODY'].
split('\n')))
self.assertIn('SWIFT_logs_SIGNATURE', self.builds[0].parameters)
self.assertEqual(
"https://storage.example.org/V1/AUTH_account/logs/1/1/1/"
"gate/test-test/",
self.builds[1].parameters['SWIFT_logs_URL'][:-7])
self.assertEqual(5,
len(self.builds[1].parameters['SWIFT_logs_HMAC_BODY'].
split('\n')))
self.assertIn('SWIFT_logs_SIGNATURE', self.builds[1].parameters)
self.assertEqual(
"https://storage.example.org/V1/AUTH_account/stash/1/1/1/"
"gate/test-test/",
self.builds[1].parameters['SWIFT_MOSTLY_URL'][:-7])
self.assertEqual(5,
len(self.builds[1].
parameters['SWIFT_MOSTLY_HMAC_BODY'].split('\n')))
self.assertIn('SWIFT_MOSTLY_SIGNATURE', self.builds[1].parameters)
self.launch_server.hold_jobs_in_build = False
self.launch_server.release()
self.waitUntilSettled()
def test_client_get_running_jobs(self):
"Test that the RPC client can get a list of running jobs"
self.launch_server.hold_jobs_in_build = True

6
zuul/cmd/scheduler.py
View File

@ -127,7 +127,6 @@ class Scheduler(zuul.cmd.ZuulApp):
import zuul.launcher.client
import zuul.merger.client
import zuul.nodepool
import zuul.lib.swift
import zuul.webapp
import zuul.rpclistener
import zuul.zk
@ -141,11 +140,8 @@ class Scheduler(zuul.cmd.ZuulApp):
self.log = logging.getLogger("zuul.Scheduler")
self.sched = zuul.scheduler.Scheduler(self.config)
# TODO(jhesketh): Move swift into a connection?
self.swift = zuul.lib.swift.Swift(self.config)
gearman = zuul.launcher.client.LaunchClient(self.config, self.sched,
self.swift)
gearman = zuul.launcher.client.LaunchClient(self.config, self.sched)
merger = zuul.merger.client.MergeClient(self.config, self.sched)
nodepool = zuul.nodepool.Nodepool(self.sched)

12
zuul/configloader.py
View File

@ -122,20 +122,8 @@ class NodeSetParser(object):
class JobParser(object):
@staticmethod
def getSchema():
swift_tmpurl = {vs.Required('name'): str,
'container': str,
'expiry': int,
'max_file_size': int,
'max-file-size': int,
'max_file_count': int,
'max-file-count': int,
'logserver_prefix': str,
'logserver-prefix': str,
}
auth = {'secrets': to_list(str),
'inherit': bool,
'swift-tmpurl': to_list(swift_tmpurl),
}
node = {vs.Required('name'): str,

43
zuul/launcher/client.py
View File

@ -17,7 +17,6 @@ import gear
import json
import logging
import os
import six
import time
import threading
from uuid import uuid4
@ -149,10 +148,9 @@ class LaunchClient(object):
log = logging.getLogger("zuul.LaunchClient")
negative_function_cache_ttl = 5
def __init__(self, config, sched, swift):
def __init__(self, config, sched):
self.config = config
self.sched = sched
self.swift = swift
self.builds = {}
self.meta_jobs = {} # A list of meta-jobs like stop or describe
@ -211,42 +209,6 @@ class LaunchClient(object):
self.log.debug("Function %s is not registered" % name)
return False
def updateBuildParams(self, job, item, params):
"""Allow the job to modify and add build parameters"""
# NOTE(jhesketh): The params need to stay in a key=value data pair
# as workers cannot necessarily handle lists.
if 'swift' in job.auth and self.swift.connection:
for name, s in job.swift.items():
swift_instructions = {}
s_config = {}
s_config.update((k, v.format(item=item, job=job,
change=item.change))
if isinstance(v, six.string_types)
else (k, v)
for k, v in s.items())
(swift_instructions['URL'],
swift_instructions['HMAC_BODY'],
swift_instructions['SIGNATURE']) = \
self.swift.generate_form_post_middleware_params(
params['LOG_PATH'], **s_config)
if 'logserver_prefix' in s_config:
swift_instructions['LOGSERVER_PREFIX'] = \
s_config['logserver_prefix']
elif self.config.has_option('swift',
'default_logserver_prefix'):
swift_instructions['LOGSERVER_PREFIX'] = \
self.config.get('swift', 'default_logserver_prefix')
# Create a set of zuul instructions for each instruction-set
# given in the form of NAME_PARAMETER=VALUE
for key, value in swift_instructions.items():
params['_'.join(['SWIFT', name, key])] = value
def launch(self, job, item, pipeline, dependent_items=[]):
uuid = str(uuid4().hex)
self.log.info(
@ -309,9 +271,6 @@ class LaunchClient(object):
params['BASE_LOG_PATH'] = item.change.getBasePath()
params['LOG_PATH'] = destination_path
# Allow the job to update the params
self.updateBuildParams(job, item, params)
# This is what we should be heading toward for parameters:
# required:

168
zuul/lib/swift.py
View File

@ -1,168 +0,0 @@
# Copyright 2014 Rackspace Australia
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import hmac
from hashlib import sha1
import logging
from time import time
import os
import random
import six
from six.moves import urllib
import string
class Swift(object):
log = logging.getLogger("zuul.Swift")
def __init__(self, config):
self.config = config
self.connection = False
if self.config.has_option('swift', 'X-Account-Meta-Temp-Url-Key'):
self.secure_key = self.config.get('swift',
'X-Account-Meta-Temp-Url-Key')
else:
self.secure_key = ''.join(
random.choice(string.ascii_uppercase + string.digits)
for x in range(20)
)
self.storage_url = ''
if self.config.has_option('swift', 'X-Storage-Url'):
self.storage_url = self.config.get('swift', 'X-Storage-Url')
try:
if self.config.has_section('swift'):
if (not self.config.has_option('swift', 'Send-Temp-Url-Key')
or self.config.getboolean('swift',
'Send-Temp-Url-Key')):
self.connect()
# Tell swift of our key
headers = {}
headers['X-Account-Meta-Temp-Url-Key'] = self.secure_key
self.connection.post_account(headers)
if not self.config.has_option('swift', 'X-Storage-Url'):
self.connect()
self.storage_url = self.connection.get_auth()[0]
except Exception as e:
self.log.warning("Unable to set up swift. Signed storage URL is "
"likely to be wrong. %s" % e)
def connect(self):
if not self.connection:
authurl = self.config.get('swift', 'authurl')
user = (self.config.get('swift', 'user')
if self.config.has_option('swift', 'user') else None)
key = (self.config.get('swift', 'key')
if self.config.has_option('swift', 'key') else None)
retries = (self.config.get('swift', 'retries')
if self.config.has_option('swift', 'retries') else 5)
preauthurl = (self.config.get('swift', 'preauthurl')
if self.config.has_option('swift', 'preauthurl')
else None)
preauthtoken = (self.config.get('swift', 'preauthtoken')
if self.config.has_option('swift', 'preauthtoken')
else None)
snet = (self.config.get('swift', 'snet')
if self.config.has_option('swift', 'snet') else False)
starting_backoff = (self.config.get('swift', 'starting_backoff')
if self.config.has_option('swift',
'starting_backoff')
else 1)
max_backoff = (self.config.get('swift', 'max_backoff')
if self.config.has_option('swift', 'max_backoff')
else 64)
tenant_name = (self.config.get('swift', 'tenant_name')
if self.config.has_option('swift', 'tenant_name')
else None)
auth_version = (self.config.get('swift', 'auth_version')
if self.config.has_option('swift', 'auth_version')
else 2.0)
cacert = (self.config.get('swift', 'cacert')
if self.config.has_option('swift', 'cacert') else None)
insecure = (self.config.get('swift', 'insecure')
if self.config.has_option('swift', 'insecure')
else False)
ssl_compression = (self.config.get('swift', 'ssl_compression')
if self.config.has_option('swift',
'ssl_compression')
else True)
available_os_options = ['tenant_id', 'auth_token', 'service_type',
'endpoint_type', 'tenant_name',
'object_storage_url', 'region_name']
os_options = {}
for os_option in available_os_options:
if self.config.has_option('swift', os_option):
os_options[os_option] = self.config.get('swift', os_option)
import swiftclient
self.connection = swiftclient.client.Connection(
authurl=authurl, user=user, key=key, retries=retries,
preauthurl=preauthurl, preauthtoken=preauthtoken, snet=snet,
starting_backoff=starting_backoff, max_backoff=max_backoff,
tenant_name=tenant_name, os_options=os_options,
auth_version=auth_version, cacert=cacert, insecure=insecure,
ssl_compression=ssl_compression)
def generate_form_post_middleware_params(self, destination_prefix='',
**kwargs):
"""Generate the FormPost middleware params for the given settings"""
# Define the available settings and their defaults
settings = {
'container': '',
'expiry': 7200,
'max_file_size': 104857600,
'max_file_count': 10,
'file_path_prefix': ''
}
for key, default in six.iteritems(settings):
# TODO(jeblair): Remove the following two lines after a
# deprecation period for the underscore variants of the
# settings in YAML.
if key in kwargs:
settings[key] = kwargs[key]
# Since we prefer '-' rather than '_' in YAML, look up
# keys there using hyphens. Continue to use underscores
# everywhere else.
altkey = key.replace('_', '-')
if altkey in kwargs:
settings[key] = kwargs[altkey]
elif self.config.has_option('swift', 'default_' + key):
settings[key] = self.config.get('swift', 'default_' + key)
# TODO: these are always strings; some should be converted
# to ints.
expires = int(time() + int(settings['expiry']))
redirect = ''
url = os.path.join(self.storage_url, settings['container'],
settings['file_path_prefix'],
destination_prefix)
u = urllib.parse.urlparse(url)
hmac_body = '%s\n%s\n%s\n%s\n%s' % (u.path, redirect,
settings['max_file_size'],
settings['max_file_count'],
expires)
signature = hmac.new(self.secure_key, hmac_body, sha1).hexdigest()
return url, hmac_body, signature