Commit Graph

138 Commits (247cab7f2c0ef7569851aec0e7b447679a71a47b)

Author SHA1 Message Date
Monty Taylor c1229468ad
Log items in loops better
If the loop item has a result msg - we should display it.

If the loop item doens't have a result msg - there is no other
identifying feature than the loop variable - which may be a full dict or
other json-object. In that case, pretty print it rather than one-lining
it. This may not be perfect - but let's give it a try.

Change-Id: Ib9a4dc0ac7783559151f12996144941e534633ee
6 years ago
Monty Taylor e60af64c1b
Improve display of simple error messages
Fact gathering does not have a variable_manager yet, so we need to put
in an if condition for localhost fact gathering.

Additionally, simple errors, such as our local code is prohibited
message, wind up just being {'msg': 'Error Message', 'failed': True}. We
don't need to emit that as a json dict- we can TOTALLY emit that as, you
know, a string after ERROR. So it'll look like:

  2017-07-21 06:38:28.982788 | localhost | ERROR: Executing local code is prohibited

Story: 2001129
Task: 4838
Change-Id: Ie9f3bd917bf75f5a566a8a1a09ccf9f29786d82f
6 years ago
Monty Taylor 995df73d47
Add spacer after playbook stats rather than before playbook
We were trying to be clever and only emit a space before playbook start
if it was not the first. WHOOPS. Instance variables to not persist
across process boundaries - so the stats call cannot set a variable for
the playbook_on_start to read.

Change-Id: I5fb19102d4e5903f3a49c16df5701a70b93b0b1f
6 years ago
Monty Taylor 1c39ec7361
Make playbook run meta info less fragile
Pass in and use info from the JobDirPlaybook rather than trying to strip
path elements from the playbook name.

Change-Id: Ifcd6f05e27c987d40db23b3dcec344c2eb786d7c
6 years ago
Monty Taylor 3e4a3b11a4
Remove noise both vertical and horizontal
We have spacer lines after task banners that we don't need.

We don't have spacer lines between a PLAY RECAP and the next PLAY which
we do need.

We have a spacer line before a PLAY RECAP which doubles with the
post-task spacer.

We don't need args printed in task banner lines.

Change-Id: I678690bbb4fc41d6028b5e86f43574989c24f8b6
6 years ago
Monty Taylor b0dc141de4
Emit phase banner separate from play banner
The play banner is very busy. Let's just emit the information about the
execution phase on it's own line. ALSO - let's use PRE-RUN, RUN and
POST-RUN, since those are actually the terms from zuul.yaml. Ignore the
index count in the text log - it's not super interesting. It IS
interesting in the json - but we'll deal with that later.

There is now an annoying pile of copy-pasta between zuul_stream and
zuul_json that we should put into a shared library location. But let's
do that as a follow up.

Change-Id: I6e3abd4c2c7c1967435815fab4f9aaa333103397
6 years ago
Monty Taylor f482a5bc14
Fix json output appending
The json output wasn't doing its read/append/write cycle properly,
leading to:

    "plays": {
        "plays": {
            "plays": {
                "plays": {
                    "plays": {

Which, while amusing, isn't really what we wanted.

Add a wrapping layer that contains playbook info. The original format
has a list of play results but only one stats section, so it's not
actually suitable for appending a list of plays anyway.

Change-Id: I49394e9faa8027a21e5ef6919c0f75a4473f51a9
6 years ago
Monty Taylor 5ddb26a0ca
Only output result details on error
We have the full json output if we need it which contains all of the
results - but in most cases we do not need the full result dict on
success and it makes reading logs quite noisy. (hi stat)

Change-Id: I153bcdf0ff4826d34bc83c68ac1cefc4d264da01
6 years ago
Monty Taylor 83509426c8
Add callback plugin to emit json
Tried first with the upstream callback plugin, but it is a stdout
plugin, so needs to take over stdout to work. We need stdout for
executor communication. Then tried subclassing- but the magical ansible
module plugin loading fun happened again. Just copy it in and modify it
slightly for now.

We add playbook, phase and index information. We also read the previous
file back in and append to it on subsequent runs. This may be a memory
issue. HOWEVER - the current construction will hold all of an individual
play in memory anyway. Most of our content size concerns are around
devstack jobs where the bulk of the content will be in a single playbook
anyway - so although ram pressure may be a real thing - we may need to
solve it on the single playbook level anyway. But for now, this should
get us the data.

Change-Id: Ic1becaf2f3ab345da22fa62314f1296d76777fec
6 years ago
Monty Taylor d8a7dcc2e9
Filter and print results more comprehensively
Don't emit results when no_log is set. That's impotant.

If the result is a message and a rc, emit the message - since
that's the important message for the user.

If messages are multi-line, split them and show them multi-line.

If the results have an "invocation" in them, suppress it.

This will still result in synchronize calls potentially showing a lot of
output - but it should be in a readable format. If output from a
synchronize call is not desired or two large, it can be marked no_log.

Change-Id: I90cf5de8eb4479773aee97b5590ac125c06dc00e
6 years ago
Tobias Henkel 27d51685c9 Fix dictionary change during iteration
During dumping the result_dict it we iterate over a copy of it and
filter out some variables. This results in the error below [1]. The
reason is the result_dict.keys() doesn't return a new set of the
existing keys but a view of them which is changed then during
iteration. This can be fixed by creating a list of the keys and
iterating over that.

[1] Log:

 [WARNING]: Failure using method (v2_runner_on_ok) in callback plugin
  (<ansible.plugins.callback.zuul_stream.CallbackModule object at
  0x7f5d41b73b70>): dictionary changed size during iteration

Change-Id: I2ad148002dbe471296b4c1b0e448f2cc03ce2913
6 years ago
Tobias Henkel 5b7a042e3b Fix KeysView object does not support indexing
The KeysView we get from the hostvars is not indexable and throws an
error [1] when trying to access via an index. This can be fixed by
using an iterator and getting one item.

[1] Log:

  [WARNING]: Failure using method (v2_playbook_on_play_start) in callback plugin
  (<ansible.plugins.callback.zuul_stream.CallbackModule object at
  0x7f5d41b73b70>): 'KeysView' object does not support indexing

Change-Id: I0c95b55a3b26856e9d101db95682efa0e0694523
6 years ago
Jenkins e391ec8df8 Merge "Log execution phase and include information" into feature/zuulv3 6 years ago
Jenkins 6c53761e6a Merge "Add play recap back in to the end of plays" into feature/zuulv3 6 years ago
Jenkins 34bafd22eb Merge "Add spacing lines and formatting to result dicts" into feature/zuulv3 6 years ago
Monty Taylor 393ee76e59
Log execution phase and include information
We pass the execution phase and the index of the phase now, so emit it
into the play banners in the log. This will allow us to post-process the
logs and put in smart things like "collapse pre tasks".

Also include information about include statements.

Also rename zuul_execution_phase_count to zuul_execution_phase_index -
mainly just because it's an index not a count. My shed is red.

Change-Id: I975ed9547bbcdbb70d5a25c9be398888bdcdb07a
6 years ago
Monty Taylor b09cb00b85
Add play recap back in to the end of plays
We aren't showing this and we're thinking it'll be nice to.

Change-Id: Ie184a5562ee5168592f81ed719b6158adb62776a
6 years ago
Monty Taylor 3a5db0a41e
Add spacing lines and formatting to result dicts
Based on a suggestion from pabelanger to try out some of the formatting
ideas from here:

add blank lines for spacing at the start of tasks and plays, and format
the results dict when we have one.

Change-Id: I29945d5451c6ac69c90a0140e1a686ae3afea340
6 years ago
James E. Blair 196f61a552 Pass result data back from ansible
This loads a json file (work/results.json) that the job can write
to.  It will be loaded by the executor after the job completes and
returned to the scheduler.

We can use the data in this file as the reported log URL for the
build.  Later we can use it to supply file/line comments in

Change-Id: Ib4eb743405f337c5bd541dd147e687fd44699713
6 years ago
Tobias Henkel 542f948952 Check ret for None in zuul_run_command
The check 'if not ret' not only matches a missing return code but also
the value 0 which is actually a successful run. Thus successful
commands end with an error 'Something went horribly wrong during task
execution'. This can be fixed by explicitly checking for None.

Also adds a successful shell task to the test_playbook test case which
fails without this patch.

Change-Id: If1d0721574a82e247659ab0f865ae6acfe12a6be
6 years ago
Monty Taylor d635d2a31c
Make sure we always log the exit line
If there's an exception, we might not write the final lines to the log.
Put those in a finally block.

Change-Id: I0a294af3874c53543176a7bd9c4b89253717b83c
6 years ago
Monty Taylor 37338d33bf
Add support for marking module results uninteresting
Using python to collect data for later display is useful, as it allows
us to control what's output and what isn't. That's defeated by the
default behavior of logging all of the result parameters. Add a flag we
can set in result dicts that will cause us to discard the parameters
from the log on success. On failure we don't want to do this because on
failure all the debugging we can get is great.

Change-Id: Ic2295c7a8be30144fe47c5bd385e827bfd8095f2
6 years ago
Monty Taylor 9db79043f7
Don't look for ansible_host unconditionally
We do this same fallback in other places, and see warnings in the
executor log about hitting this.

Change-Id: Ic9d27a7d3299c8f32fb6d1104dbda4bf5b76c54a
6 years ago
Monty Taylor 1d0bcea15c
Allow file manipulation in the work dir
Change-Id: I9f03ea408085e3da3ddeb231264e45b1c1fb6896
6 years ago
Jenkins 08fe120a9e Merge "Don't log starting to log messages to build log" into feature/zuulv3 6 years ago
Jenkins 6bc671a4ec Merge "Fix py3 issue with command module" into feature/zuulv3 6 years ago
Monty Taylor 2e8052b40a
Apply localhost workaround if ansible_host is localhost
This shouldn't really ever happen for real in zuul, but just in case we
do something in a trusted playbook somewhere like setting ansible_host
to localhost, the code should at least do the right thing. (This comes
up more in doing local testing of plugins)

Change-Id: I638fa5f8cd69cd46823ea42bc4e257972d125945
6 years ago
Monty Taylor 9de422f55b
Handle item loops in zuul_stream
Items in loops get their own callback events, so we need to handle them.

Additionally, the final item and the event finished event come out of
sequence - so we need to save a note in the event finished event and
then emit it after the following item.

Finally, we'd have a todo list item from ansiblefest to copy not
directly modify the result objects we get because they're shared. Turns
out this is immediately necessary with items, because popping
stdout_lines in one item totally borks the further items (whoops)

Change-Id: Ia70239f240276b10ad09d4bda57310ebcbdb1f4d
6 years ago
Monty Taylor 9fa148291a
Don't print results from explicit gather facts
When gather_facts is explicit and then is run, it shows up as a 'setup'
task in the execution. It's noise in the log.

Change-Id: I1a9776dcb4c832827d394f5e8f425bd21efdd089
6 years ago
James E. Blair 3fabd44226 Fix py3 issue with command module
This caused us to fail to log lines when running commands on
localhost (because in that situation, we use python3).

Change-Id: If9f5cf6ed62cb24a6fc16ebf189d36a752fa66d5
6 years ago
Monty Taylor 3060f9f4f5
Don't log starting to log messages to build log
These are for the debug log, not for the job log. They are noise in the
job log.

Change-Id: Icdc861d7f9d22a2e4ea9e9c01203d615420b67df
6 years ago
Jenkins 2ee8b01cd4 Merge "Read the log file as binary in zuul_console" into feature/zuulv3 6 years ago
Jenkins 0d391caa5f Merge "Carve out for stat" into feature/zuulv3 6 years ago
Jenkins 6ea7dc7ecd Merge "Don't try to split localhost log lines" into feature/zuulv3 6 years ago
Monty Taylor 6a4d0aa745
Don't try to split localhost log lines
They don't come with timestamps.

Change-Id: I20f84929a5ca7b075b88db3029c51c6bbba6f8d0
6 years ago
Jenkins 547af8e8fd Merge "Write logfile as binary encoded utf-8" into feature/zuulv3 6 years ago
Monty Taylor 8dce6aacaa
Read the log file as binary in zuul_console
It gets decoded as utf-8 on the other side of the socket, and it's being
written to the file as utf-8 - so there's not reason to read it in to a
string before sending it over the wire.

Change-Id: Iad3d33108835fad3ceae0eec38985659449a6452
6 years ago
Monty Taylor 1a3bb28e47
Write logfile as binary encoded utf-8
Change-Id: I08e3e2fbfb629a6b68b80ddaadbef71038b3eda0
6 years ago
James E. Blair c9003740b8 Fix exception handler in command module
Change-Id: Ie7f860e25a533620b263d3f1f7f127605991fa11
6 years ago
Monty Taylor 3545a6fda9
Carve out for stat
Change-Id: I30e4fcc9697f8c50e7419d65f01bd50fe1d4b7d4
6 years ago
Jenkins 3515c5cac2 Merge "Fix typo in localhost check in zuul streamer" into feature/zuulv3 6 years ago
Tobias Henkel 7dc938173e Fix typo in localhost check in zuul streamer
There is a typo causing the match against localhost to fail

Change-Id: I9f8375a8dd306dbc7ac9019d227514d2f6ea3bd2
6 years ago
Tobias Henkel 9ca3f55664 Fix access before assignment in zuul streamer
For local non-command and non-shell tasks there is an access to
stdout_lines which is not assigned before.

Change-Id: Ifaa88a5b7399514afd1a9babe195a70a4a0e9fae
6 years ago
Clint Byrum b159de7881 Revert "Revert sync from latest command from ansible"
It also fixes the bug of executing the command twice.

This reverts commit 1cfc946297.

Change-Id: Ia0b4f0b7d1bea5bba74f518a1b2cb7b9d84eba50
6 years ago
Monty Taylor 1cfc946297
Revert sync from latest command from ansible
These are causing WILDLY strange issues on ze01. Revert them until we

Revert "Update run_command to latest ansible"

This reverts commit 1dbf5f96b5.

Revert "Sync command from ansible"

This reverts commit 429428c020.

Change-Id: Ib481160312216a4dbc9c3184d31cbc35b5b36371
6 years ago
Jenkins c3a56cc37e Merge "Handle lists of streamers" into feature/zuulv3 6 years ago
Jenkins 8c48ca580d Merge "Special case shell logging on localhost" into feature/zuulv3 6 years ago
Jenkins a591e73f77 Merge "Direct streaming at delegated_to target" into feature/zuulv3 6 years ago
Jenkins c819ae14f9 Merge "Extract get_playhosts listing in zuul_stream to a method" into feature/zuulv3 6 years ago
Jenkins df58192624 Merge "Make logging helper method in zuul_stream" into feature/zuulv3 6 years ago