The "initial comment" is the first comment of a PR on Pagure. It
is used to provide the Depends-on stanza. Recently Pagure added the
capability to send an event when that initial comment is changed:
https://pagure.io/pagure/issue/4398

This change handles the event as a PR changed to retrigger the attached
jobs.

Change-Id: I62d4e783e94528126cd4a7d85b3e664e84758bf1

The gitweb url template was wrong and this patch fixes it.

Change-Id: Ic4ead74ddfe09b2a4a90cb7ffca746dc1e132430

send() requires a bytes-like object in Python 3, ensure the error
message is encoded correctly.

---

Some debugging notes might come in handy for the future here.  This
problem appeared in a fairly specific part of the test cases when
setting "ansible_python_interpreter" to /usr/bin/python3.  The remote
streaming test has a task that is designed to fail [1]:

 - hosts: all
   tasks:
     - name: Remote shell task with python exception
       command: echo foo
       args:
         chdir: /remote-shelltask/somewhere/that/does/not/exist
       failed_when: false

We see that Ansible ships over a payload and tries to run it, but it
raises an exception very early.

 <192.168.122.1> SSH: EXEC ssh -C ...  '/bin/sh -c '"'"'/usr/bin/python3 && sleep 0'"'"''
 <192.168.122.1> Failed to connect to the host via ssh:
 Traceback (most recent call last):
   File "<stdin>", line 114, in <module>
   File "<stdin>", line 106, in _ansiballz_main
   ...
   File "/tmp/ansible_command_payload_tieedyzs/__main__.py", line 263, in main
 FileNotFoundError: [Errno 2] No such file or directory: '/remote-shelltask/somewhere/that/does/not/exist'

When this task started, the Ansible task callbacks in the zuul_stream
callback plugin have setup a thread that listens for the console
output being sent by the remote zuul_console daemon started earlier in
the playbook [2].  This listening thread is sitting in a recv()
waiting for some streaming data to log [3].

There will be no remote log file for zuul_console to stream back,
because this task failed before it even got started.  What should
happen is the "[Zuul] Log not found" message should be sent back and
logic in [4] will match this and stop this thread.

When this does *not* happen, such as when this send() raises an
exception because of wrong data type, the task ends anyway and Ansible
moves on to make the end-of-task callbacks in zuul_stream (actually
there's a bunch of looping happening, but let's ignore those details).
This ends up in _stop_streamers() [5] which attempts to join(30) the
streaming thread.  Under normal circumstances, this thread should be
finished and the join() successful.  However, because the target
thread is stuck in a recv(), the 30-second timeout begins.  The clue
to this is in the logs you eventually get:

 [Zuul] Log Stream did not terminate

So eventually, Zuul would have made progress here and given up on
waiting for the thread to finish properly.  However, 30 seconds is a
long time to the unit-test and pushes the job over it's timeout.

Thus your end result is that when using Python 3 Zuul aborts the job,
and the test rather mysteriously fails!

[1] 3f8b36aa0b/tests/fixtures/config/remote-zuul-stream/git/org_project/playbooks/command.yaml (L93)
[2] 3f8b36aa0b/tests/fixtures/config/remote-zuul-stream/git/org_project/playbooks/command.yaml (L93)
[3] 3f8b36aa0b/zuul/ansible/base/callback/zuul_stream.py (L14)
[4] 3f8b36aa0b/zuul/ansible/base/callback/zuul_stream.py (L174)
[5] 3f8b36aa0b/zuul/ansible/base/callback/zuul_stream.py (L271)

This is tested in the follow-on I2b3bc6d4f873b7d653cfaccd1598464583c561e7

Change-Id: I7cdcfc760975871f7fa9949da1015d7cec92ee67

The current location of the reference pipelines is within the
quick-start repo, which causes all sorts of configuration errors
to be logged (but since Zuul is so tolerant of them, the job
doesn't actually fail).

Move them outside of one of the demo project directories into their
own.

Change-Id: I9cb1ddd803d938fc154a32308cae99dbab9392e2

When leaving file or line comments, leave them as robot comments
to enable more sophisticated processing/filtering.

Change-Id: Ib9e326d8a87639b06c1bc8b7f85425d98da9c003

When reporting a Gerrit review, add a tag to the ReviewInput
structure so that review messages and approvals carry the
a tag with the prefix "autogenerated:".  The UI (especially in
later versions of Gerrit) can support handling automated messages
specially (this is used by the "Only comments" switch in
polygerrit).  This is available as far back as 2.13 (possibly
older, I haven't checked).

Change-Id: I8c693d7bcd38be4ac0301fcc9a3e97748ead6d4d

This adds support for performing queries and git clones over HTTP
(reporting over HTTP was already supported).  This will happen
automatically for any connection with a password configured.
Otherwise, the SSH connection will be used as before.

Change-Id: I11920a13615de103eb3d8fb305eacbbcb30e5e40

These fixtures are missing the data returned by the "--current-patch-set"
and "--commit-message" arguments, which we do pass when performing the
queries.  A future change alters where some of these data are parsed
and will error because of this omission.

This updates the data to match what is currently returned for these
changes (with the old sortKeys manually added back in where appropriate).

Change-Id: I11e983d22c4da818b3eeab1632ca7102aab087d5

This adds initial support for the Gerrit checks plugin.
Development of that plugin is still in progress, and hopefully it
(and our support for it) will change over time.  Because we expect
to change how we interact with it in the near future, this is
documented as experimental support for now.  A release note is
intentionally omitted -- that's more appropriate when we remove
the 'experimental' label.

Change-Id: Ida0cdef682ca2ce117617eacfb67f371426a3131

This method can be used by reporters to generate the best "status"
url to report for an item.  It will report the buildset page if
that is available, or the per-change status page if it is not.

Eventually we plan on making the database required, at which point
we can insert buildsets in the db earlier.  Then we can report the
buildset page in all cases.  By beginning to use this method now,
we can seamlessly upgrade reporters in the future.

Test coverage for this is added in change
Ida0cdef682ca2ce117617eacfb67f371426a3131.

Change-Id: Ib0c2ca84f6c4d30f233382048c8885fb73edfeec

This allows reporters to include the enqueue/dequeue time of an
item.  The item model already hase enqueue/dequeue times which we
use when reporting status, however, a reporter runs right before
the item is dequeued.  So we need one more time value which
corresponds to the start of the reporting phase -- thus report_time
in this patch.

Test coverage for this is added in change
Ida0cdef682ca2ce117617eacfb67f371426a3131.

Change-Id: I093626e098b7ce2deea2b0c25265cb48d38712ad

This facilitates integration with the gerrit checks API (and may
prove useful for other similar APIs).  It will allow us to report
that a change has no jobs in a particular pipeline.  A Zuul
pipeline will correspond to a Gerrit check, which means we can
update the status for that check from "SCHEDULED" to "NOT_RELEVANT"
if we determine that no jobs should run for the change.  This
closes out the status of the check in Gerrit when a project is
configured to participate in a check/pipeline but no jobs are
actually configured.

Test coverage for this will be added in change
Ida0cdef682ca2ce117617eacfb67f371426a3131.

Change-Id: Ide2a332b294d7efe23601d80eeb92b5af1d4c21b

This facilitates integration with the gerrit checks API (and may
prove useful for other similar APIs).  It will allow us to report
that a change has been enqueued in a particular pipeline.  A Zuul
pipeline will correspond to a Gerrit check, which means we can
update the status for that check from "NOT_STARTED" to "SCHEDULED"
when it enters the pipeline.  This is important for our check
polling loop, and it will cause that check to stop appearing in
the list of pending checks.

Test coverage for this is added in change
Ida0cdef682ca2ce117617eacfb67f371426a3131.

Change-Id: I9ec329b446fa51e0911d4d9ff67eea7ddd55ab5d

This change prevents executor host command execution through a
malicious rsh synchronize rsync_opts.

Fixes bug #2006526

Change-Id: I3cd17ca91410394f164d8ea7cd91a1ea5890f998

This expands the discussion of executor-only jobs with some additional
notes.

Additionally a unit test is added to explicitly test executor-only
(i.e. blank nodeset) jobs.

Change-Id: I8fd2f932290e49da5a3605737e8940425cd092f4

Unit test playbooks are generally written as

 - hosts: all
   tasks:
     ...

However, many of the unit tests don't specify any nodes for their
jobs.  With no nodes specified, Ansible gets a empty host list and
thus the only host available is the special "implicit localhost".
Since "all" doesn't match the implicit localhost, under normal
circumstances Ansible does not match anything and doesn't run any of
the playbooks.

To get around this, the extant code in
RecordingAnsibleJob:getHostList() (tests/base.py) overrides the host
list and explicitly adds a host named "localhost".  This is put into
the Ansible inventory and now the "all" matcher has something to match
against and the playbooks run.  This work-around was initially added
with I5e23f330476f064acf3cb87f746c5d3193cce274.

The situation became a bit more confused with
Iacf670d992bb051560a0c46c313beaa6721489c4 where the "localhost" fake
node is only added if other nodes are *not* specified.  Several tests
rely on this now as they specify various forms of nodes explicitly and
don't want this fake node added.

This change removes the automatic addition of "localhost" in
unit-tests all together.  I believe this is the correct direction to
move in, because it's a fairly confusing anti-feature if, for example,
you write a unit test that *is* explicitly executor-only (i.e. a blank
node list).  Such a test fails because the unit-test framework adds a
host for you; something that does not happen in production.  It's also
a bit confusing if you're reading the config files and thinking
"hosts: all shouldn't match anything here" without digging into the
test framework.

There are two ways this could be fixed.  The playbooks that are part
of jobs that have no nodes defined could be re-written to "hosts:
localhost" so that they match the "implicit localhost" and always run.
This does not really seem to be their intent, however.  The other
option, which is taken here, is to always add nodes to the job.  I
believe this is a better approach, as it more closely matches what you
would see in actual jobs.

Change-Id: I6b52b7e4bc591c09034461b534ca5225945f76cf

You might want to take action only if a job fails or succeeds.

Change-Id: I45c1d3d22d3c49cd100552f6d4606b0c560fab10

 Rename typo in hashi_vault lookup files

Change-Id: Ie3e1d46dce222d2c0ced50cf3437dfb3ce787e51

Since the subprocess is started before the reference timestamp is
created, it can happen that the check for the expiration field fails.

Traceback (most recent call last):
  File "/tmp/zuul/tests/unit/test_client.py", line 151, in test_token_generation
    (token['exp'], now))
  File "/tmp/zuul/.tox/py36/lib/python3.6/site-packages/unittest2/case.py", line 702, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : (1568016146.9831738, 1568015546.1448617)

Change-Id: I9ef56c12ed1be2a6ec168c4a9363125919be44e9

There's a separate callback method for handler tasks.  This is what
we do in the text stream callback plugin; we should do the same in
the json output.

Change-Id: I48273ec182032f198b8886be95b0cba5c6f4843e

Now that the job-output.json file has attributes for 'failed' or
'skipped' tasks, rely on those rather than hueristics for determining
the result of a task.

Change-Id: I2714c5a1776970d5339aafe2ca5e17abf5a3bb0b

This change was ineffective because it was modifying the wrong data.
Correct that and add a test.

Change-Id: I26cfba0ff9a3eca05314e12f3f1d66b934c505b6

Log a proper error when the admin API token is invalid or
expired. Also handle API call errors.

Change-Id: Ide7a6ff0266daaac53f40d42a545d76ca76a6ff2

The toDict() method of the github connection should use base_url instead
of baseurl to prevent an AttributeError exception being through as
baseurl doesn't exists in the GithubConnection object.

Change-Id: I2fa7a2bfde781371816da533530beb81c188bc9c