Change-Id: I364ff19916cede6af21db37a52f4a72e4e2bf5b5

The OpenAPI spec notes that it is currently incomplete in its
description. However, the 'sphinxcontrib-openapi' extension doesn't
output this. Manually add one instead.

Change-Id: I6a0cde5b0533c121ed8c56261ebf9e97c5855ff8
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

Change I1fb21c7c7a2e2ae7fb02fb416e316b39714d403e caused cleanup
playbooks to run when a build is aborted, however, it was run
as part of the stop job.  This causes several problems:

* The stop job is meant to be fast.  The executor may need to
  process many stop jobs nearly simultaneously, but the job
  dispatcher is single-threaded.  The stop job should do the
  minimum necessary to stop the job and leave the bulk of the
  work to the job thread.
* The job thread assumes that it is the only thing running
  ansible processes for a job.  If another thread is doing so,
  then internal state attributes may get out of sync.

The latter is what brought this to our attention -- we were
seeing exceptions in unit tests indicating that multiple ansible
processes were setting the self.proc attribute of the AnsibleJob
object.

To correct this, simply do not run the cleanup playbook in the
stop job, and let the main job process run it.  It already handles
running the cleanup playbook regardless of the results of previous
playbooks.  It did have an extra check that the main playbook ran;
this would have produced different outcomes depending on whether
the job was aborted or not (if aborted during pre-run, the cleanup
would have run; if pre-run failed it would not have run).  This
check has been removed so that the cleanup playbooks run
unconditionally (which matches the documentation).

A mutex which attempted to prevent multiple cleanup playbook
invocations from running at the same time (which was only partially
successful) is no longer necessary since only the job thread will
run them now, so it is removed.

Change-Id: I529ef185b04764e6c40fa1bdf56ff561dc8c621a

CentOS 7.x lacks python3, so just use the default python interpreter
when invoking pip to install git-review from PyPI. It also lacks
pip, and needs EPEL enabled to be able to install it.

Change-Id: I69b857eb2781e2a8a1df9bcaea4da9ad9396d17c

The zuul logs contain start and end of reconfiguration events.
Add the reconfiguration duration to the ending part to make it easier to monitor.

Change-Id: Iea81888083c908035c77198eb2a59b609f817150

We currently have five gearman worker in the system which are all
similar but different. In preparation of adding a sixth worker
refactor them to all re-use a central class and the same config and
dispatch mechanism.

Change-Id: Ifbb4c5aec28fe5b044569d365a4e3fe31150eb3b

If Zuul was unable to freeze the job graph of the previous layout,
then it would throw an exception when calculating whether jobs have
changed.  This is a problem if it does so on a change which fixes
an error in the current tenant config, as it means the fix can not
merge.

To handle this case, we need to catch that exception, and then provide
a default value indicating whether the job config has been updated
or not.  Because returning True could cause every job with a file
matcher to run unecessarily, the safer default is False (so the job
relies on the file matcher alone).

Change-Id: I8a72e57e068b073e37274cdabeacffc9daee2e94

The recent change to diff job configs in order to run modified jobs
even if file matchers don't match compares the job config in the
change under test to the most recent layout ahead in the queue, or
the running tenant layout if none, in order to determine if a job's
config has changed.  In an independent pipeline with multiple
dependent changes, we did not generate layouts for non-live items,
therefore the only layout available to diff was the running tenant
layout.  This would cause changes to run too many jobs in that
situation (while they would run the correct jobs in a dependent
pipeline since layouts of items ahead were available).

To correct this, always generate layouts for items with config updates
regardless of whether they are live, and ensure that every item in
the queue has a pointer to a layout (whether that is its own layout
or the layout of the item ahead, or the tenant layout) so that the
next item can diff against it.

This means we are potentially running more merge operations than before.
The TestNonLiveMerges test is a poor match for the new system, since
every change in it contained a configuration update.  So a test which
previously was used to verify that we did not perform a merge for every
item now correctly performs a merge for every item.  Since that is
no longer a useful test, it is split into two simpler tests; one which
verifies that non-live items without config updates do not perform merges,
and one that repeats the same scenario with config updates and verifies
that we do perform merges for non-live items in that case (as a sort
of control for the other test).

This also changes how errors are reported for a series of changes with
a config error.  Previously in the case of a non-live item with a
config error followed by a live item without a config error, we would
report the config error of the non-live item on the live item, because
Zuul was unable to determine which change was responsible for the error
(since only the live item got a merge and layout).  Now that both items
have layouts, Zuul can determine that it does not need to report the
non-live item's error on the live item.  However, we still report an
error on the live item since it can not merge as-is.  We now report
an abbreviated error: "This change depends on a change with an invalid
configuration."

Change-Id: Id533772f35ebbc76910398e0e0fa50a3abfceb52

This helps with debugging from logs in case something doesn't enter a
gate as expected.

Change-Id: Ia0c7e84812d479c455d72f8e4c367975ea0bd709

We no longer need the zuul-specific release jobs, the opendev
job will do.  Also, we haven't needed it in the post pipeline
for a while since we added the promote release for dev releases.
We must have missed that.

Also switch to the new promote-based docs jobs.

Change-Id: I826aa3fc3c8e83cf6f2c6d3896b602d164275ba0

This is a simple typo fix.

Change-Id: Ibe5eef419c45886a0cc92a7988a01448c152a400

Add the "create-auth-token" subcommand to the zuul CLI; this subcommand
allows an operator to create an authentication token for a user with
customized authorizations.

This requires at least one auth section with a signing key to be specified in
Zuul's configuration file.

This is meant as a way to provide authorizations "manually" on test
deployments, until a proper authorization engine is plugged into Zuul,
in a subsequent patch.

Change-Id: I039e70cd8d5e502795772af0ea2a336c08316f2c

A user with the right JSON Web Token (JWT) can trigger a autohold,
reenqueue or dequeue a buildset from the web API.

The Token is expected to include a key called "zuul.admin" that
contains a list of the tenants the user is allowed to perform
these actions on.

The Token must be passed as a bearer token in an Authorization header.

The Token is validated thanks to authenticator declarations in Zuul's
configuration file.

Change-Id: Ief9088812f44368f14234ddfa25ba872526b8735

This causes file matchers to automatically match when the
configuration of the job itself changes.  This can be used instead
of matching "^.zuul.yaml$" which may cause too many jobs to run
in larger repos.

Change-Id: Ieddaead91b597282c5674ba99b0c0f387843c722

The cache of unparsed config data holds all of the config objects
on a project-branch, for all of the tenants.  That works as long
as all of the tenants load the same config files.  With the addition
of the extra-config-paths tenant option, different tenants may now
see different configuration files for the same project-branch.
Because the cache did not account for that, the data for the last
tenant to load a project-branch is what would be cached.

To correct this, make the global unparsed config cache aware of
which paths have been searched in constructing it, and cache each
file individually.  This allows the cache to continue to be
efficient across systems with large tenants where the same repos
appear multiple times, in that each configuration object will
still be stored only once.  However, by storing them along with
their files, the unparsed config for each tenant can be created
from the cache, even if the cache contains more files than that
tenant should see.  Because the cache remembers which files have
been searched, we can determine whether the cache results from a
prior tenant are valid for a later one, or if the later one is
meant to search more files, we can run another cat job and fetch
the larger set of files and expand the cache.

A unit test is adjusted to add a tenant which loads fewer files
before one which loads more to verify that the cache expands
appropriately when loading the second tenant.

Change-Id: I268d89dfaaaf65af2237ec852d706a2752535c4a

The change to add extra-config-paths neclected to update the updatesConfig()
method to account for the potential new paths; this could Zuul to fail to
reload its configuration on changes which touched only the extra files.

This corrects that and adds a test case.

Change-Id: I79fa02a482a0bbebff0de7f3081bb7ed04f1214c

This is intended to address the case of the zuul-jobs repo, where
it would be convenient to include a project definition and job
definitions which test the roles in the zuul-jobs repo.  However,
we want the repo te be consumable by anyone, which means we can
not encode nodesets or required-projects in any jobs which may
be loaded by another Zuul.

To address this, this commit adds a feature which will allow us
to put those job and project definitions in a separate file or
directory which will not be loaded by default.  But in the Zuul
tenant of the OpenDev Zuul installation, we will configure the
system to load this secondary configuration location, so the
self-test jobs will be available.

Change-Id: Ic205d1f93f583514757a100471c47688d6641c53

If you define pragma.implied-branches on a branch, it needs to include
that branch name or changes will not be applied. Let's document this so
that other don't wonder why their zuul layout changes on a branch don't
actually trigger anything.

Change-Id: Ia9bbc54a199cd759580973f756956a9cf07d6b3e

When working with GitHub Enterprise the recommended working model is
branch&pull within the same repo. This is especially necessary for
workflows that combine multiple repos in a single workspace. This has
the side effect that those repos can contain a large number of
branches that never will be part of a job. Having many branches in a
repo can have a large impact on the executor performance so exclude
them from the repo state if we exclude them in the tenant config. This
change only affects branches, not tags or other references.

Change-Id: Ic8e75fa8bf76d2e5a0b1779fa3538ee9a5c43411