Bind secrets to their playbooks

Secrets are proving less useful than originally hoped because they
can not be effectively used in any jobs with untrusted children.

This change binds the secrets to the playbooks which use them, so
that child jobs are unable to access the secrets.  This allows us
to create jobs with pre/post playbooks which use secrets which
are suitable for other jobs to inherit from.

Change-Id: I67dd12563f3abd242d6356675afed1de0cb144cf
changes/07/492307/2
James E. Blair 5 years ago
parent ad966ccc46
commit 892cca6afa
  1. 83
      doc/source/user/config.rst
  2. 3
      doc/source/user/jobs.rst
  3. 4
      tests/base.py
  4. 26
      tests/fixtures/layouts/untrusted-secrets.yaml
  5. 147
      tests/unit/test_model.py
  6. 12
      tests/unit/test_scheduler.py
  7. 52
      zuul/configloader.py
  8. 1
      zuul/driver/bubblewrap/__init__.py
  9. 5
      zuul/executor/client.py
  10. 74
      zuul/executor/server.py
  11. 49
      zuul/model.py

@ -187,14 +187,14 @@ success, the pipeline reports back to Gerrit with ``Verified`` vote of
.. attr:: allow-secrets
:default: false
This is a boolean which can be used to prevent jobs which
require secrets from running in this pipeline. Some pipelines
run on proposed changes and therefore execute code which has not
yet been reviewed. In such a case, allowing a job to use a
secret could result in that secret being exposed. The default
is ``false``, meaning that in order to run jobs with secrets,
this must be explicitly enabled on each Pipeline where that is
safe.
This is a boolean which can be used to prevent jobs which use
secrets in the untrusted security context from running in this
pipeline. Some pipelines run on proposed changes and therefore
execute code which has not yet been reviewed. In such a case,
allowing a job to use a secret could result in that secret being
exposed. The default is ``false``, meaning that in order to run
jobs which use secrets in the untrusted security context, this
must be explicitly enabled on each Pipeline where that is safe.
For more information, see :ref:`secret`.
@ -672,19 +672,6 @@ Here is an example of two job definitions:
Authentication information to be made available to the job.
This is a dictionary with two potential keys:
.. attr:: inherit
:default: false
A boolean indicating that the authentication information
referenced by this job should be able to be inherited by
child jobs. Normally when a job inherits from another job,
the auth section is not included. This permits jobs to
inherit the same basic structure and playbook, but ensures
that secret information is unable to be exposed by a child
job which may alter the job's behavior. If it is safe for
the contents of the authentication section to be used by
child jobs, set this to ``true``.
.. attr:: secrets
A list of secrets which may be used by the job. A
@ -907,6 +894,16 @@ Here is an example of two job definitions:
it should be able to run this job, then it must be explicitly
listed. By default, all projects may use the job.
.. attr:: untrusted-secrets
A boolean value which indicates that this job should not be used
in a pipeline where allow-secrets is ``false``. This is
automatically set to ``true`` if this job is defined in a
:term:`untrusted-project`. It may be explicitly set to obtain
the same behavior for jobs defined in :term:`config projects
<config-project>`. Once this is set to ``true`` anywhere in the
inheritance hierarchy for a job, it will remain set for all
child jobs and variants (it can not be set to ``false``).
.. _project:
@ -1062,22 +1059,34 @@ unencrypted as well for convenience.
A Secret may only be used by jobs defined within the same project. To
use a secret, a :ref:`job` must specify the secret within its `auth`
section. To protect against jobs in other repositories declaring a
job with a secret as a parent and then exposing that secret, jobs
which inherit from a job with secrets will not inherit the secrets
themselves. To alter that behavior, see the `inherit` job attribute.
Further, jobs which do not permit children to inherit secrets (the
default) are also automatically marked `final`, meaning that their
execution related attributes may not be changed in a project-pipeline
stanza. This is to protect against a job with secrets defined in one
project being used by another project in a way which might expose the
secrets. If a job with secrets is unsafe to be used by other
projects, the `allowed-projects` job attribute can be used to restrict
the projects which can invoke that job. Finally, pipelines which are
used to execute proposed but unreviewed changes can set the
`allow-secrets` attribute to indicate that they should not supply
secrets at all in order to protect against someone proposing a change
which exposes a secret.
section. Secrets are bound to the playbooks associated with the
specific job definition where they were declared. Additional pre or
post playbooks which appear in child jobs will not have access to the
secrets, nor will playbooks which override the main playbook (if any)
of the job which declared the secret. This protects against jobs in
other repositories declaring a job with a secret as a parent and then
exposing that secret.
It is possible to use secrets for jobs defined in :term:`config
projects <config-project>` as well as :term:`untrusted projects
<untrusted-project>`, however their use differs slightly. Because
playbooks in a config project which use secrets run in the
:term:`trusted execution context` where proposed changes are not used
in executing jobs, it is safe for those secrets to be used in all
types of pipelines. However, because playbooks defined in an
untrusted project are run in the :term:`untrusted execution context`
where proposed changes are used in job execution, it is dangerous to
allow those secrets to be used in pipelines which are used to execute
proposed but unreviewed changes. By default, pipelines will refuse to
run jobs which have playbooks that use secrets in the untrusted
execution context to protect against someone proposing a change which
exposes a secret. To permit this (for instance, in a pipeline which
only runs after code review), the :attr:`pipeline.allow-secrets`
attribute may be set.
If a job with secrets is unsafe to be used by other projects, the
`allowed-projects` job attribute can be used to restrict the projects
which can invoke that job.
.. attr:: secret

@ -121,6 +121,9 @@ Might be used in a template as::
{{ credentials.username }} {{ credentials.password }}
Secrets are only available to playbooks associated with the job
definition which uses the secret; they are not available to playbooks
associated with child jobs or job variants.
Zuul Variables
~~~~~~~~~~~~~~

@ -1451,12 +1451,12 @@ class RecordingAnsibleJob(zuul.executor.server.AnsibleJob):
self.recordResult(result)
return result
def runAnsible(self, cmd, timeout, config_file, trusted):
def runAnsible(self, cmd, timeout, playbook):
build = self.executor_server.job_builds[self.job.unique]
if self.executor_server._run_ansible:
result = super(RecordingAnsibleJob, self).runAnsible(
cmd, timeout, config_file, trusted)
cmd, timeout, playbook)
else:
result = build.run()
return result

@ -0,0 +1,26 @@
- pipeline:
name: check
manager: independent
trigger:
gerrit:
- event: patchset-created
success:
gerrit:
Verified: 1
failure:
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project1-test
untrusted-secrets: true
- project:
name: org/project1
check:
jobs:
- project1-test

@ -54,6 +54,8 @@ class TestJob(BaseTestCase):
encryption.deserialize_rsa_keypair(f.read())
self.context = model.SourceContext(self.project, 'master',
'test', True)
self.untrusted_context = model.SourceContext(self.project, 'master',
'test', False)
m = yaml.Mark('name', 0, 0, 0, '', 0)
self.start_mark = configloader.ZuulMark(m, m, '')
@ -97,16 +99,15 @@ class TestJob(BaseTestCase):
def test_job_inheritance(self):
# This is standard job inheritance.
base_pre = model.PlaybookContext(self.context, 'base-pre', [])
base_run = model.PlaybookContext(self.context, 'base-run', [])
base_post = model.PlaybookContext(self.context, 'base-post', [])
base_pre = model.PlaybookContext(self.context, 'base-pre', [], [])
base_run = model.PlaybookContext(self.context, 'base-run', [], [])
base_post = model.PlaybookContext(self.context, 'base-post', [], [])
base = model.Job('base')
base.timeout = 30
base.pre_run = [base_pre]
base.run = [base_run]
base.post_run = [base_post]
base.auth = model.AuthContext()
py27 = model.Job('py27')
self.assertIsNone(py27.timeout)
@ -118,23 +119,21 @@ class TestJob(BaseTestCase):
[x.path for x in py27.run])
self.assertEqual(['base-post'],
[x.path for x in py27.post_run])
self.assertIsNone(py27.auth)
def test_job_variants(self):
# This simulates freezing a job.
py27_pre = model.PlaybookContext(self.context, 'py27-pre', [])
py27_run = model.PlaybookContext(self.context, 'py27-run', [])
py27_post = model.PlaybookContext(self.context, 'py27-post', [])
secrets = ['foo']
py27_pre = model.PlaybookContext(self.context, 'py27-pre', [], secrets)
py27_run = model.PlaybookContext(self.context, 'py27-run', [], secrets)
py27_post = model.PlaybookContext(self.context, 'py27-post', [],
secrets)
py27 = model.Job('py27')
py27.timeout = 30
py27.pre_run = [py27_pre]
py27.run = [py27_run]
py27.post_run = [py27_post]
auth = model.AuthContext()
auth.secrets.append('foo')
py27.auth = auth
job = py27.copy()
self.assertEqual(30, job.timeout)
@ -151,7 +150,9 @@ class TestJob(BaseTestCase):
[x.path for x in job.run])
self.assertEqual(['py27-post'],
[x.path for x in job.post_run])
self.assertEqual(auth, job.auth)
self.assertEqual(secrets, job.pre_run[0].secrets)
self.assertEqual(secrets, job.run[0].secrets)
self.assertEqual(secrets, job.post_run[0].secrets)
# Set the job to final for the following checks
job.final = True
@ -341,7 +342,7 @@ class TestJob(BaseTestCase):
conf = yaml.safe_load('''
- secret:
name: pypi-credentials
name: trusted-secret
data:
username: test-username
longpassword: !encrypted/pkcs1-oaep
@ -384,8 +385,14 @@ class TestJob(BaseTestCase):
conf['_source_context'] = self.context
conf['_start_mark'] = self.start_mark
secret = configloader.SecretParser.fromYaml(layout, conf)
layout.addSecret(secret)
trusted_secret = configloader.SecretParser.fromYaml(layout, conf)
layout.addSecret(trusted_secret)
conf['name'] = 'untrusted-secret'
conf['_source_context'] = self.untrusted_context
untrusted_secret = configloader.SecretParser.fromYaml(layout, conf)
layout.addSecret(untrusted_secret)
base = configloader.JobParser.fromYaml(self.tenant, self.layout, {
'_source_context': self.context,
@ -395,86 +402,100 @@ class TestJob(BaseTestCase):
'timeout': 30,
})
layout.addJob(base)
pypi_upload_without_inherit = configloader.JobParser.fromYaml(
tenant, layout, {
'_source_context': self.context,
'_start_mark': self.start_mark,
'name': 'pypi-upload-without-inherit',
'parent': 'base',
'timeout': 40,
'auth': {
'secrets': [
'pypi-credentials',
]
}
})
layout.addJob(pypi_upload_without_inherit)
pypi_upload_with_inherit = configloader.JobParser.fromYaml(
trusted_secrets_job = configloader.JobParser.fromYaml(
tenant, layout, {
'_source_context': self.context,
'_start_mark': self.start_mark,
'name': 'pypi-upload-with-inherit',
'name': 'trusted-secrets',
'parent': 'base',
'timeout': 40,
'auth': {
'inherit': True,
'secrets': [
'pypi-credentials',
'trusted-secret',
]
}
})
layout.addJob(pypi_upload_with_inherit)
pypi_upload_with_inherit_false = configloader.JobParser.fromYaml(
layout.addJob(trusted_secrets_job)
untrusted_secrets_job = configloader.JobParser.fromYaml(
tenant, layout, {
'_source_context': self.context,
'_source_context': self.untrusted_context,
'_start_mark': self.start_mark,
'name': 'pypi-upload-with-inherit-false',
'name': 'untrusted-secrets',
'parent': 'base',
'timeout': 40,
'auth': {
'inherit': False,
'secrets': [
'pypi-credentials',
'untrusted-secret',
]
}
})
layout.addJob(pypi_upload_with_inherit_false)
in_repo_job_without_inherit = configloader.JobParser.fromYaml(
layout.addJob(untrusted_secrets_job)
trusted_secrets_trusted_child_job = configloader.JobParser.fromYaml(
tenant, layout, {
'_source_context': self.context,
'_start_mark': self.start_mark,
'name': 'in-repo-job-without-inherit',
'parent': 'pypi-upload-without-inherit',
'name': 'trusted-secrets-trusted-child',
'parent': 'trusted-secrets',
})
layout.addJob(in_repo_job_without_inherit)
in_repo_job_with_inherit = configloader.JobParser.fromYaml(
layout.addJob(trusted_secrets_trusted_child_job)
trusted_secrets_untrusted_child_job = configloader.JobParser.fromYaml(
tenant, layout, {
'_source_context': self.context,
'_source_context': self.untrusted_context,
'_start_mark': self.start_mark,
'name': 'in-repo-job-with-inherit',
'parent': 'pypi-upload-with-inherit',
'name': 'trusted-secrets-untrusted-child',
'parent': 'trusted-secrets',
})
layout.addJob(in_repo_job_with_inherit)
in_repo_job_with_inherit_false = configloader.JobParser.fromYaml(
layout.addJob(trusted_secrets_untrusted_child_job)
untrusted_secrets_trusted_child_job = configloader.JobParser.fromYaml(
tenant, layout, {
'_source_context': self.context,
'_start_mark': self.start_mark,
'name': 'in-repo-job-with-inherit-false',
'parent': 'pypi-upload-with-inherit-false',
'name': 'untrusted-secrets-trusted-child',
'parent': 'untrusted-secrets',
})
layout.addJob(in_repo_job_with_inherit_false)
self.assertIsNone(in_repo_job_without_inherit.auth)
self.assertEqual(1, len(in_repo_job_with_inherit.auth.secrets))
self.assertEqual(in_repo_job_with_inherit.auth.secrets[0].name,
'pypi-credentials')
self.assertIsNone(in_repo_job_with_inherit_false.auth)
self.assertEqual(in_repo_job_with_inherit.auth.secrets[0].
layout.addJob(untrusted_secrets_trusted_child_job)
untrusted_secrets_untrusted_child_job = \
configloader.JobParser.fromYaml(
tenant, layout, {
'_source_context': self.untrusted_context,
'_start_mark': self.start_mark,
'name': 'untrusted-secrets-untrusted-child',
'parent': 'untrusted-secrets',
})
layout.addJob(untrusted_secrets_untrusted_child_job)
self.assertIsNone(trusted_secrets_job.untrusted_secrets)
self.assertTrue(untrusted_secrets_job.untrusted_secrets)
self.assertIsNone(
trusted_secrets_trusted_child_job.untrusted_secrets)
self.assertIsNone(
trusted_secrets_untrusted_child_job.untrusted_secrets)
self.assertTrue(
untrusted_secrets_trusted_child_job.untrusted_secrets)
self.assertTrue(
untrusted_secrets_untrusted_child_job.untrusted_secrets)
self.assertEqual(trusted_secrets_job.implied_run[0].secrets[0].name,
'trusted-secret')
self.assertEqual(trusted_secrets_job.implied_run[0].secrets[0].
secret_data['longpassword'],
'test-passwordtest-password')
self.assertEqual(in_repo_job_with_inherit.auth.secrets[0].
self.assertEqual(trusted_secrets_job.implied_run[0].secrets[0].
secret_data['password'],
'test-password')
self.assertEqual(
len(trusted_secrets_trusted_child_job.implied_run[0].secrets), 0)
self.assertEqual(
len(trusted_secrets_untrusted_child_job.implied_run[0].secrets), 0)
self.assertEqual(untrusted_secrets_job.implied_run[0].secrets[0].name,
'untrusted-secret')
self.assertEqual(
len(untrusted_secrets_trusted_child_job.implied_run[0].secrets), 0)
self.assertEqual(
len(untrusted_secrets_untrusted_child_job.implied_run[0].secrets),
0)
def test_job_inheritance_job_tree(self):
tenant = model.Tenant('tenant')
@ -688,9 +709,7 @@ class TestJob(BaseTestCase):
'name': 'job',
'parent': None,
})
auth = model.AuthContext()
auth.secrets.append('foo')
job.auth = auth
job.untrusted_secrets = True
self.layout.addJob(job)

@ -2818,6 +2818,18 @@ class TestScheduler(ZuulTestCase):
self.assertEqual(B.data['status'], 'MERGED')
self.assertEqual(B.reported, 2)
@simple_layout('layouts/untrusted-secrets.yaml')
def test_untrusted_secrets(self):
"Test untrusted secrets"
A = self.fake_gerrit.addFakeChange('org/project1', 'master', 'A')
self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
self.waitUntilSettled()
self.assertHistory([])
self.assertEqual(A.patchsets[0]['approvals'][0]['value'], "-1")
self.assertIn('does not allow jobs with secrets',
A.messages[0])
@simple_layout('layouts/tags.yaml')
def test_tags(self):
"Test job tags"

@ -372,6 +372,7 @@ class JobParser(object):
'allowed-projects': to_list(str),
'override-branch': str,
'description': str,
'untrusted-secrets': bool
}
return vs.Schema(job)
@ -426,20 +427,6 @@ class JobParser(object):
job = model.Job(conf['name'])
job.source_context = conf.get('_source_context')
if 'auth' in conf:
job.auth = model.AuthContext()
if 'inherit' in conf['auth']:
job.auth.inherit = conf['auth']['inherit']
for secret_name in conf['auth'].get('secrets', []):
secret = layout.secrets[secret_name]
if secret.source_context != job.source_context:
raise Exception(
"Unable to use secret %s. Secrets must be "
"defined in the same project in which they "
"are used" % secret_name)
job.auth.secrets.append(secret.decrypt(
job.source_context.project.private_key))
is_variant = layout.hasJob(conf['name'])
if 'parent' in conf:
@ -462,6 +449,33 @@ class JobParser(object):
if not is_variant:
parent = layout.getJob(tenant.default_base_job)
job.inheritFrom(parent)
# Secrets are part of the playbook context so we must establish
# them earlier than playbooks.
secrets = []
if 'auth' in conf:
for secret_name in conf['auth'].get('secrets', []):
secret = layout.secrets[secret_name]
if secret.source_context != job.source_context:
raise Exception(
"Unable to use secret %s. Secrets must be "
"defined in the same project in which they "
"are used" % secret_name)
secrets.append(secret.decrypt(
job.source_context.project.private_key))
# A job in an untrusted repo that uses secrets requires
# special care. We must note this, and carry this flag
# through inheritance to ensure that we don't run this job in
# an unsafe check pipeline.
if secrets and not conf['_source_context'].trusted:
job.untrusted_secrets = True
if 'untrusted-secrets' in conf:
if conf['untrusted-secrets']:
job.untrusted_secrets = True
else:
raise Exception("Once set, the untrusted_secrets "
"attribute may not be unset")
# Roles are part of the playbook context so we must establish
# them earlier than playbooks.
@ -481,21 +495,23 @@ class JobParser(object):
for pre_run_name in as_list(conf.get('pre-run')):
pre_run = model.PlaybookContext(job.source_context,
pre_run_name, job.roles)
pre_run_name, job.roles,
secrets)
job.pre_run = job.pre_run + (pre_run,)
for post_run_name in as_list(conf.get('post-run')):
post_run = model.PlaybookContext(job.source_context,
post_run_name, job.roles)
post_run_name, job.roles,
secrets)
job.post_run = (post_run,) + job.post_run
if 'run' in conf:
run = model.PlaybookContext(job.source_context, conf['run'],
job.roles)
job.roles, secrets)
job.run = (run,)
else:
if not project_pipeline:
run_name = os.path.join('playbooks', job.name)
run = model.PlaybookContext(job.source_context, run_name,
job.roles)
job.roles, secrets)
job.implied_run = (run,) + job.implied_run
for k in JobParser.simple_attributes:

@ -158,7 +158,6 @@ class BubblewrapDriver(Driver, WrapperInterface):
'--ro-bind', '/etc/resolv.conf', '/etc/resolv.conf',
'--ro-bind', '/etc/hosts', '/etc/hosts',
'--ro-bind', '{ssh_auth_sock}', '{ssh_auth_sock}',
'--dir', '{work_dir}',
'--bind', '{work_dir}', '{work_dir}',
'--dev', '/dev',
'--chdir', '{work_dir}',

@ -228,11 +228,6 @@ class ExecutorClient(object):
params['nodes'] = nodes
params['groups'] = [group.toDict() for group in nodeset.getGroups()]
params['vars'] = copy.deepcopy(job.variables)
params['secrets'] = {}
if job.auth:
for secret in job.auth.secrets:
secret_data = copy.deepcopy(secret.secret_data)
params['secrets'][secret.name] = secret_data
params['zuul'] = zuul_params
projects = set()

@ -251,6 +251,8 @@ class JobDirPlaybook(object):
self.roles_path = []
self.ansible_config = os.path.join(self.root, 'ansible.cfg')
self.project_link = os.path.join(self.root, 'project')
self.secrets = os.path.join(self.root, 'secrets.yaml')
self.has_secrets = False
def addRole(self):
count = len(self.roles)
@ -271,18 +273,19 @@ class JobDir(object):
log streaming daemon find job logs.
'''
# root
# .ansible
# fact-cache/localhost
# ansible
# ansible (mounted in bwrap read-only)
# inventory.yaml
# playbook_0
# .ansible (mounted in bwrap read-write)
# fact-cache/localhost
# playbook_0 (mounted in bwrap for each playbook read-only)
# secrets.yaml
# project -> ../trusted/project_0/...
# role_0 -> ../trusted/project_0/...
# trusted
# trusted (mounted in bwrap read-only)
# project_0
# <git.example.com>
# <project>
# work
# work (mounted in bwrap read-write)
# .ssh
# known_hosts
# src
@ -311,8 +314,8 @@ class JobDir(object):
ssh_dir = os.path.join(self.work_root, '.ssh')
os.mkdir(ssh_dir, 0o700)
# Create ansible cache directory
ansible_cache = os.path.join(self.root, '.ansible')
self.fact_cache = os.path.join(ansible_cache, 'fact-cache')
self.ansible_cache_root = os.path.join(self.root, '.ansible')
self.fact_cache = os.path.join(self.ansible_cache_root, 'fact-cache')
os.makedirs(self.fact_cache)
localhost_facts = os.path.join(self.fact_cache, 'localhost')
# NOTE(pabelanger): We do not want to leak zuul-executor facts to other
@ -327,8 +330,6 @@ class JobDir(object):
pass
self.known_hosts = os.path.join(ssh_dir, 'known_hosts')
self.inventory = os.path.join(self.ansible_root, 'inventory.yaml')
self.secrets = os.path.join(self.ansible_root, 'secrets.yaml')
self.has_secrets = False
self.playbooks = [] # The list of candidate playbooks
self.playbook = None # A pointer to the candidate we have chosen
self.pre_playbooks = []
@ -1260,7 +1261,7 @@ class AnsibleJob(object):
for role in playbook['roles']:
self.prepareRole(jobdir_playbook, role, args)
self.writeAnsibleConfig(jobdir_playbook)
self.writeAnsibleConfig(jobdir_playbook, playbook)
def checkoutTrustedProject(self, project, branch):
root = self.jobdir.getTrustedProject(project.canonical_name,
@ -1381,25 +1382,25 @@ class AnsibleJob(object):
for key in node['host_keys']:
known_hosts.write('%s\n' % key)
secrets = args['secrets'].copy()
def writeAnsibleConfig(self, jobdir_playbook, playbook):
trusted = jobdir_playbook.trusted
secrets = playbook['secrets'].copy()
if secrets:
if 'zuul' in secrets:
raise Exception("Defining secrets named 'zuul' is not allowed")
with open(self.jobdir.secrets, 'w') as secrets_yaml:
with open(jobdir_playbook.secrets, 'w') as secrets_yaml:
secrets_yaml.write(
yaml.safe_dump(secrets, default_flow_style=False))
self.jobdir.has_secrets = True
def writeAnsibleConfig(self, jobdir_playbook):
trusted = jobdir_playbook.trusted
jobdir_playbook.has_secrets = True
with open(jobdir_playbook.ansible_config, 'w') as config:
config.write('[defaults]\n')
config.write('hostfile = %s\n' % self.jobdir.inventory)
config.write('local_tmp = %s/.ansible/local_tmp\n' %
self.jobdir.root)
config.write('remote_tmp = %s/.ansible/remote_tmp\n' %
self.jobdir.root)
config.write('local_tmp = %s/local_tmp\n' %
self.jobdir.ansible_cache_root)
config.write('remote_tmp = %s/remote_tmp\n' %
self.jobdir.ansible_cache_root)
config.write('retry_files_enabled = False\n')
config.write('gathering = smart\n')
config.write('fact_caching = jsonfile\n')
@ -1425,13 +1426,12 @@ class AnsibleJob(object):
config.write('roles_path = %s\n' % ':'.join(
jobdir_playbook.roles_path))
# On trusted jobs, we want to prevent the printing of args,
# since trusted jobs might have access to secrets that they may
# need to pass to a task or a role. On the other hand, there
# should be no sensitive data in untrusted jobs, and printing
# the args could be useful for debugging.
# On playbooks with secrets we want to prevent the
# printing of args since they may be passed to a task or a
# role. Otherwise, printing the args could be useful for
# debugging.
config.write('display_args_to_stdout = %s\n' %
str(not trusted))
str(not secrets))
config.write('[ssh_connection]\n')
# NB: when setting pipelining = True, keep_remote_files
@ -1464,7 +1464,8 @@ class AnsibleJob(object):
except Exception:
self.log.exception("Exception while killing ansible process:")
def runAnsible(self, cmd, timeout, config_file, trusted):
def runAnsible(self, cmd, timeout, playbook):
config_file = playbook.ansible_config
env_copy = os.environ.copy()
env_copy.update(self.ssh_agent.env)
env_copy['ZUUL_JOB_OUTPUT_FILE'] = self.jobdir.job_output_file
@ -1477,7 +1478,7 @@ class AnsibleJob(object):
pythonpath = [self.executor_server.ansible_dir] + pythonpath
env_copy['PYTHONPATH'] = os.path.pathsep.join(pythonpath)
if trusted:
if playbook.trusted:
opt_prefix = 'trusted'
else:
opt_prefix = 'untrusted'
@ -1489,6 +1490,11 @@ class AnsibleJob(object):
rw_paths = rw_paths.split(":") if rw_paths else []
ro_paths.append(self.executor_server.ansible_dir)
ro_paths.append(self.jobdir.ansible_root)
ro_paths.append(self.jobdir.trusted_root)
ro_paths.append(playbook.root)
rw_paths.append(self.jobdir.ansible_cache_root)
if self.executor_variables_file:
ro_paths.append(self.executor_variables_file)
@ -1497,7 +1503,7 @@ class AnsibleJob(object):
rw_paths)
popen = self.executor_server.execution_wrapper.getPopen(
work_dir=self.jobdir.root,
work_dir=self.jobdir.work_root,
ssh_auth_sock=env_copy.get('SSH_AUTH_SOCK'))
env_copy['ANSIBLE_CONFIG'] = config_file
@ -1577,8 +1583,8 @@ class AnsibleJob(object):
verbose = '-v'
cmd = ['ansible-playbook', verbose, playbook.path]
if self.jobdir.has_secrets:
cmd.extend(['-e', '@' + self.jobdir.secrets])
if playbook.has_secrets:
cmd.extend(['-e', '@' + playbook.secrets])
if success is not None:
cmd.extend(['-e', 'success=%s' % str(bool(success))])
@ -1600,9 +1606,7 @@ class AnsibleJob(object):
cmd.extend(['-e@%s' % self.executor_variables_file])
result, code = self.runAnsible(
cmd=cmd, timeout=timeout,
config_file=playbook.ansible_config,
trusted=playbook.trusted)
cmd=cmd, timeout=timeout, playbook=playbook)
self.log.debug("Ansible complete, result %s code %s" % (
self.RESULT_MAP[result], code))
return result, code

@ -651,10 +651,11 @@ class PlaybookContext(object):
"""
def __init__(self, source_context, path, roles):
def __init__(self, source_context, path, roles, secrets):
self.source_context = source_context
self.path = path
self.roles = roles
self.secrets = secrets
def __repr__(self):
return '<PlaybookContext %s %s>' % (self.source_context,
@ -668,16 +669,22 @@ class PlaybookContext(object):
return False
return (self.source_context == other.source_context and
self.path == other.path and
self.roles == other.roles)
self.roles == other.roles and
self.secrets == other.secrets)
def toDict(self):
# Render to a dict to use in passing json to the executor
secrets = {}
for secret in self.secrets:
secret_data = copy.deepcopy(secret.secret_data)
secrets[secret.name] = secret_data
return dict(
connection=self.source_context.project.connection_name,
project=self.source_context.project.name,
branch=self.source_context.branch,
trusted=self.source_context.trusted,
roles=[r.toDict() for r in self.roles],
secrets=secrets,
path=self.path)
@ -740,28 +747,6 @@ class ZuulRole(Role):
return d
class AuthContext(object):
"""The authentication information for a job.
Authentication information (both the actual data and metadata such
as whether it should be inherited) for a job is grouped together
in this object.
"""
def __init__(self, inherit=False):
self.inherit = inherit
self.secrets = []
def __ne__(self, other):
return not self.__eq__(other)
def __eq__(self, other):
if not isinstance(other, AuthContext):
return False
return (self.inherit == other.inherit and
self.secrets == other.secrets)
class Job(object):
"""A Job represents the defintion of actions to perform.
@ -804,7 +789,6 @@ class Job(object):
timeout=None,
variables={},
nodeset=NodeSet(),
auth=None,
workspace=None,
pre_run=(),
post_run=(),
@ -817,6 +801,7 @@ class Job(object):
required_projects={},
allowed_projects=None,
override_branch=None,
untrusted_secrets=None,
)
# These are generally internal attributes which are not
@ -932,13 +917,9 @@ class Job(object):
if not isinstance(other, Job):
raise Exception("Job unable to inherit from %s" % (other,))
do_not_inherit = set()
if other.auth and not other.auth.inherit:
do_not_inherit.add('auth')
# copy all attributes
for k in self.inheritable_attributes:
if (other._get(k) is not None and k not in do_not_inherit):
if (other._get(k) is not None):
setattr(self, k, copy.deepcopy(getattr(other, k)))
msg = 'inherit from %s' % (repr(other),)
@ -2322,11 +2303,6 @@ class Layout(object):
# (that is to say that it must match more than just
# the job that is defined in the tree).
continue
# If the job does not allow auth inheritance, do not allow
# the project-pipeline variants to update its execution
# attributes.
if frozen_job.auth and not frozen_job.auth.inherit:
frozen_job.final = True
# Whether the change matches any of the project pipeline
# variants
matched = False
@ -2342,8 +2318,7 @@ class Layout(object):
change.project.name not in frozen_job.allowed_projects):
raise Exception("Project %s is not allowed to run job %s" %
(change.project.name, frozen_job.name))
if ((not pipeline.allow_secrets) and frozen_job.auth and
frozen_job.auth.secrets):
if ((not pipeline.allow_secrets) and frozen_job.untrusted_secrets):
raise Exception("Pipeline %s does not allow jobs with "
"secrets (job %s)" % (
pipeline.name, frozen_job.name))

Loading…
Cancel
Save