Update our diagram to show the connections needed if running a database.

Change-Id: I67e47b1916ac1c3ad1f06b9b65c4b1e78aa6a55f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>

Show how zuul-web and zuul-fingergw need to connect to zuul-executors
for log streaming.

Change-Id: Ia985979c16d8276c13b1ba7ffbbb5a2224ccff01
Signed-off-by: Paul Belanger <pabelanger@redhat.com>

This is not documented in the javascript page.

Change-Id: I19c104edb87b50552a99d4c400219b5bd055630a

I found myself looking into source code to see which services I needed
to open firewall ports for. It seems only the executors and scheduler
send traffic to statsd today.

Change-Id: If7a02bb2658435d3ce7435e5ad061cd1224eb3da
Signed-off-by: Paul Belanger <pabelanger@redhat.com>

The docs missed the usage options of zuul-manage-ansible. This is
important as this documents how to add additional dependencies to
ansible during installation.

Change-Id: Iedb0d179a86130b72e314855a94feef98c454654

In order to make running zuul easier we want the possiblility that the
executor installs the supported ansible versions during startup. This
adds this functionality as well as a config switch to disable it and a
config option to optionally specify the install location. The default
location is <state_dir>/ansible-bin.

Change-Id: I1858e4fb40190626d001e20b48cf7e69ad35d634

Currently the default ansible version is selected by the version of
zuul itself. However we want to make this configurable per deployment
(zuul.conf), tenant and job.

Change-Id: Iccbb124ac7f7a8260c730fbc109ccfc1dec09f8b

As a first step towards supporting multiple ansible versions we need
tooling to manage ansible installations. This moves the installation
of ansible from the requirements.txt into zuul. This is called as a
setup hook to install the ansible versions into
<prefix>/lib/zuul/ansible. Further this tooling abstracts knowledge
that the executor must know in order to actually run the correct
version of ansible.

The actual usage of multiple ansible versions will be done in
follow-ups.

For better maintainability the ansible plugins live in
zuul/ansible/base where plugins can be kept in different versions if
necessary. For each supported ansible version there is a specific
folder that symlinks the according plugins.

Change-Id: I5ce1385245c76818777aa34230786a9dbaf723e5
Depends-On: https://review.openstack.org/623927

Change-Id: Ibd7cb02c659bb0de0f08845ed974d6c03043b21f

A "soft" dependency can be used to indicate that a job must run
after another completes, but only if it runs at all.  For example,
a deployment job which depends on a build job with different file
matcher criteria.

Change-Id: I4d7fc2b40942569323da273c4529fdb365a3b11a

This default is unlikely to be correct and has caused confusion
for us in the past.  Remove it (which matches the documentation).

Change-Id: I3453b0e918fb1c6783514c470f40f4e973fd683a

Change-Id: Ibc004adc67add0c14a98d9de5c508f5aa131224c

Zuul recently added zuul.message which needs to be protected against
interpretation by jinja in ansible. This was initially done by marking
it with the !unsafe tag. However this has the disadvantage that the
inventory is no longer parsable by standard yaml parsers without
teaching them the !unsafe tag.

There is a similar simple possibility that doesn't rely on this tag by
base64 encoding the commit message. Ansible has filters for decoding
this so it is still quite easy to deal with base64 encoded vars in
ansible via '{{ zuul.message | b64decode }}'.

Change-Id: I9628e2770dda120b269612e28bb6217036942b8e

The `sshkey` option is only there for usage when using the webhook
method of configuring the GitHub connection, this change documents
and clarifies that.

Change-Id: Iee5cccc12b5226aec8aadfe3bd73e8e52f98aac2

Our current approach to enforce the disk limit per job can be very
expensive by running 'du' in a loop. When having many repos in the
cache and many running jobs this can poison the cache and induce a
large amount of IO load. This can influence overall performance
especially if zuul is running on a shared storage like ceph.

Change-Id: Ic03168e30e0cba4a4adb42eebf4709ceba0d8c3e

In order to make it simple for multiple independent playbooks to return
artifacts, append the values of zuul.artifacts in zuul_return rather than
the usual method of overwriting lists.

Change-Id: I09e6076b4bb354023c5414d149d9bfa59fb3ea4b

This is a partial revert of f12453f6cb.

The use case that change was designed to address is poorly served by
that change.  The intent was to make it easier to return multiple
artifacts in multiple playbooks independently by relying on the dictionary
merge behavior of zuul_return.  However, in the entirely likely case
of artifacts with generated names, it becomes difficult because Ansible
does not run jinja on dictionary keys.

Therefore, revert to the previous list behavior.  A subsequent change
will add a feature to zuul_return to address the underlying issue of
returning multiple artifacts from different playbooks.

Change-Id: I0581aa68fcef320ab27c11ddd6338a15eef38ceb

We had the restriction that zuul_return only works on localhost but
this is no longer true.

Change-Id: I0296a40b4a6a565991e97120b7d14824d2a30fbb

A recent attempt to use the artifact return feature of zuul_return
exposed some rough edges.  These two changes should make it much
easier to use.

First, return artifacts as a dictionary instead of a list.  This
requires that they have unique names (which is no bad thing -- what
would two artifacts named "docs" mean anyway?).  But mainly it allows
the dict merging behavior of zuul_return to be used, so that one
playbook may use zuul_return with some artifacts, and another playbook
may do the same, without either needing to load in the values of
the other first (assuming, of course, that they use different artifact
names).

Second, add a metadata field.  In the database and API, this is JSON
serialized, but in zuul_return and zuul.artifacts, it is exploded into
separate fields.  This lets jobs do things like associate versions or
tags with artifacts without having to abuse the url field.

Change-Id: I228687c1bd1c74ebc33b088ffd43f30c7309990d

We're presuming, by default, that a user named "zuul" exists, which
is not the case in some environments.  Change the default to avoid
dropping privileges, and require that this be explicitly set in order
to do so.

Change-Id: Ia677d2615dd9292a809df4c8859a60b7f4df6243

The existing url does not work.

Change-Id: Ie5bef54be0e4122a10d0042c2ee2f5328f68d043

Like pre-run and post-run, allow a user to run a list of playbooks for
a job. One example would be your job workflow would be to run multiple
playbooks over using a site.yaml file with include_playbook commands.

A second use case, more related to job design. With multiple playbooks
support for job.run, the first playbook would be use deploy your server
and the second playbook to validate the server was provisioned properly.
Today, this can be done using a single run and post-run playbooks,
however if post-run fails, zuul will return POST_FAILURE, not FAILURE.
Not a large issue, but could be confusing to users when POST_FAILURE is
returned.

While it is possible a user could create a single site.yaml playbook,
and use multiple include_playbook statements to get this functionality,
there are downsides to this approach (mostly with the leaking of
variables).  Today, operators simply run ansible-playbook multiple times
with the specific playbooks they only want.

Story: 2002543
Task: 22101

Change-Id: I6268d9944e745cc07407ea7dd040fbfeb79dad4d
Related-To: https://review.openstack.org/519596
Signed-off-by: Paul Belanger <pabelanger@redhat.com>

The Service Workers seem to be consistently causing issues for people
that are strange, meaning many of our deployers are disabling them.

Since they aren't super necessary for the Zuul use case, change the
default behavior to be to disable them instead of enable them.

Change-Id: Iea8348a3b007badaae74fc1837b55bb0b076ac65

So that people can re-use the Dockerfiles to build zuul images
but with different flags set, plumb the env vars through here
as ARG entries.

Also, fix 2 doc references that were misspelled.

Change-Id: I320a496eadf4132fc0583dd48a87024a2ff61a07

Currently zuul only supports one specific ansible version at
once. This complicates upgrading ansible because ansible often breaks
backwards compatibility and so we need to synchronize the upgrade on
the complete deployment which is often not possible.

Instead we want to support multiple ansible versions at once so we can
handle the lifecycle of ansible versions by adding new versions and
deprecating old ones.

Change-Id: If00c971cec3b90406ec1bf9b37e9fecf38c0c03f
Story: 2004572
Task: 28345

When dealing with large repos or slow connections to the scm the
default clone timeout of 5 minutes may not be sufficient. Thus a
configurable clone/fetch timeout can make it possible to handle those
repos.

Change-Id: I0711895806b7cbcc8b9fa3ba085bcf79d7fb6665

Adds support for expressing artifact dependencies between jobs
which may run in different projects.

Change-Id: If8cce8750d296d607841800e4bbf688a24c40e08

This is already mentioned in Nodepool's operator guide when speaking
about clouds. Let's make sure that the same thing is mentioned when
speaking about a statically preconfigured set of nodes.

Change-Id: Icc7786d54a0a4c1f499704bcb0fb59199a30f615

In some environments the setup playbook can sometimes take more than
60 seconds. In order to avoid retry limits in these cases make this
timeout configurable.

Change-Id: Ib45957df12b34ddaeec79eb10f7b2e99091dcad1

ZooKeeper is mentioned under the Nodepool external dependency
section, but really needs to be in its own section to make
it more easily identifiable as an external system.

Change-Id: I8dea96eee7062769f0ce72c463c48fc8dfa329ad

It is possible to circumvent the use of `allowed-projects` in
untrusted projects by creating a change which `Depends-On` a
change which alters a project definition.  This behavior may be
unexpected, so documentation has been updated with warnings to
avoid relying on it in sensitive cases.

It may have been possible to expose a secret, or use resources
protected by a secret, if a job using a secret was defined in an
untrusted project on a system with an independent pre-merge
post-review pipeline -- that is, a pipeline with `post-review` set
to true, `manager` set to `independent`, and which operated on
changes before they merged.

To prevent disclosure or use in this situation, `allowed-projects`
is now automatically set to the current project when a secret is
used in a job defined in an untrusted project, and it can not be
overridden.

The test_trusted_secret_inheritance_gate test is removed because
it only tested that jobs with secrets in an untrusted repo were
able to run in a trusted repo.  That is no longer possible.

Change-Id: I77f6a011bca08a2433137dc29597b7cc2757adb1
Story: 2004837
Task: 29037

So that folks who would like to examine the config files without
running it (or, while running it) can do so easily in their browser.

Change-Id: I73396a4d63ffc2f1aea19569907d300757acd3ec

This allows jobs to pass secrets to parent jobs.  That will allow
for jobs which are designed to be used with secrets with the actual
secrets supplied by child jobs.

Change-Id: I544aec77c86f4cb1c11bf6e14a7d93efea0421d6

Having the change message available via the Zuul vars simplifies cases
where a job e.g. needs to update a GitHub/Jira/... ticket.

Those ticket numbers are usually referenced in the commit/PR message.

This avoids having to deal with secrets etc. to get this information
'out-of-band'.

Change-Id: Ib88db7f724dadfb8a4f86e76692f3e1c2c63a258

Using 'sudo -i' changes the working directory and so docker-compose
doesn't find the config file. Use 'sudo -E' instead to preserve the
environment variables in case http_proxy vars are needed.

Change-Id: Ib1da4a5c42029f3c0cf20b003fad09bc418b62ad

This adds a reno note for upgrading to 3.4.0 about zuul-web needing
access to zookeeper now.  Also update our components diagram too.

Change-Id: I60e9eaa6cc78306e71869602e330b4bec435d158
Signed-off-by: Paul Belanger <pabelanger@redhat.com>

sudo -i is needed in case you have environment variables defined like
http_proxy, without it docker will not find them.

Change-Id: Ic3b728fe7f81278d266ec7945a1cfd709c6d3ca4

The plan for the idea of a "promote" pipeline is to fetch
previously uploaded artifacts from the build log server
and move them to the final publication location.  However,
jobs which store data (such as documentation builds,
tarballs, or container images) on the log server should not
need to know the configuration of the log server in order
to return the artifact URL to zuul.  To support this, if
the job returns a relative URL for an artifact, assume it
is relative to the log URL for the build and combine the
two when storing the artifact info.

Change-Id: I4bce2401c9e59fd469e3b3da2973514c07faecf2

The stats zuul.executor.<name>.pause and
zuul.executor.<name>.paused_builds are undocumented. While at it fix
the indentation of this section.

Change-Id: I5d5bdc1fe748ec2c545c8b7e8ec2674d50208f9f

We currently have a gauge for starting_builds but actually have no
knowledge about how long jobs are in the starting state. This adds a
metric for this so we can see changes in the job startup time after
changes in the system.

Change-Id: I261f8bdc8de336967b9c8ecd6eafc68f0bfe6b78

This is a follow-up to fix typo introduced by
https://review.openstack.org/570668.

Change-Id: I69e42542761bbbd5cd47976ea437911b28be27e7

When running within k8s the system memory statistics are useless as
soon there are configured limits (which is strongly advised). In this
case we additionally need to check the cgroups.

Change-Id: Idebe5d7e60dc862e89d012594ab362a19f18708d

This is an attempt to formalize our governance structure, but it's
not really an attempt to change it.  The goal has been to create
a governance structure which most closely resembles how we have
been operating for the past several years.

Change-Id: Ia692631353b5d2ac90ee4f9b9c615e4f7244a24d

The following error 
exectuor to executor
formated to formatted
overidden to overridden

Change-Id: Ie80e1632624c65adaf6aad86a2c7aae93da688ff

This change adds executor support for generic build resources.
Using the connection_port, a kube config file is generated to enable ansible
tasks to use the resource. Context and pod name is also exported in a
zuul.resources variable to enable job direct access to the build resource.

Currently this change supports 'kubectl' resource for containers that behave
like a machine, and 'project'/'namespace' resource for native containers.

Change-Id: Icdb9e800177dc770c58f65b02456a6b904be0666

The current stats set a counter zuul.nodepool.<status> but then tries
to set more counters like zuul.nodepool.<status>.label.

This doesn't work because zuul.nodepool.<status> is already a counter
value; it can't also be an intermediate key.  Note this *does* work
with the timer values, but that's because statsd is turning the timer
into individual values
(e.g. zuul.nodepool.<status>.<mean|count|std...>) as it flushes each
interval.

Thus we need to rethink these stats.  This puts them under a new
intermeidate key "requests" and adds a "total" count; thus
zuul.nodepool.<status> == zuul.nodepool.requests.<status>.total

The other stats, showing requests by-label and by-size will now live
under the zuul.nodepool.requests parent.

While we're here, use a statsd pipeline to send the status update as
it works better when sending lots of stats quickly over UDP.  This
isn't handled by the current debug log below; move this into the
test-case framework.

The documentation has been clarified to match the code.

Change-Id: I127e8b6d08ab86e0f24018fd4b33c626682c76c7

Change-Id: Ic96a00618e1c98ee1b0339d360351be719aa676d

Change-Id: Ief2c7f7db59c32f412a331dea430272ae1b63341

When calculating relative_priority for independent pipelines,
use shared change queues just as is done for dependent pipelines.

To implement this, we now calculate shared change queues for all
pipelines, not just dependent ones, though we don't use those
queues for any purpose other than this.

Change-Id: I59b1090ca1f4fcc72276445e6ff4c5cf4f2f5030

Prominently in the Zuul User Guide, include a brief overview of
preferred methods for reporting suspected security vulnerabilities.
Also link to it from the README in such a way that the same
reference can be reused in other related Zuul repositories following
the same policy.

Change-Id: I2bd13bd13372f26c328cd7d6b5618ee8edffe490