26 Commits

Author SHA1 Message Date
James E. Blair
f0d7c980b5 Handle no-sudo in stage-output
The stage-output role had two assumptions:

1) The zuul_copy_output variable would always be defined
2) The role would be able to sudo on the remote node

To make it easier for users who want to use this in a base job,
remove both of those assumptions.

1) We now supply a default empty dict if the variable is not defined
so that the role does not fail with an error.

2) We check to see if we can sudo on the remote node, and if we can,
we do when copying files; otherwise we don't, and assume that the
user will only specify files they have access to (if they don't then
the copy will fail).

Change-Id: I5428c44adfafac4872342b59a92de311f41687b6
2021-05-31 15:47:57 -07:00
Sorin Sbarnea
90d263941a More E208
Change-Id: I8c3d224682d7d6780d6ad7823e390c47114e702b
2020-11-05 08:33:28 +00:00
Jens Harbott
3d86c71723 Fix with_* loop for ansible 2.9
Ansible changed handling loop variables in 2.9 [0]. Use explicit
loop statements instead of with_*.

[0] https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#migrating-to-loop

Change-Id: Ic063adbb672c3ff6bb4cca346d689630a7a37f34
2020-06-18 15:33:12 +02:00
Sorin Sbarnea
4af438d136 Made sequence indent consistent
Change-Id: I5ea4232ca4fd6e03d5b5f72eb6704bee84d04ea5
2020-05-07 14:13:55 +01:00
Albin Vass
dd8e831379 use zj_log_file instead of item as loop_var
Change-Id: Ife5560ad028e293f8bfa71756fe6825c1a3f43ce
2020-05-05 18:43:30 +00:00
Albin Vass
bee0c6ae2f ansible-lint: use matchplay instead of matchtask
For some reason matchtask doesn't match includes, matchplay does so use
that instead.

Change-Id: I040f7f3394503e92d06c05e8ff671a43b14baebc
2020-05-05 20:42:38 +02:00
Albin Vass
61c4eeaffd Distinguish zj_source and zj_result and item
The inner item needs to be zj_source. Call the outer zj_result
for clarity.

Change-Id: Ic5a7b18e1dba71d8dffa171246b52c81e30c6d8f
2020-05-01 15:39:21 -05:00
Albin Vass
9062289151 Check for loop_control in with_ type loops
Change-Id: I191265df7709a6262b44a428d78fe28ffaeb4b75
2020-05-01 13:45:34 +02:00
Jeremy Stanley
2711d10186 Be consistent about spaces before and after vars
With the arrival of ansible-lint 4, Jinja2 variable expansions must
include spaces before and after the variable name inside the
brackets.

Adjust the new violations accordingly and remove the rule
206 exclusion.

Change-Id: Ib3ff7b0233a5d5cf99772f9c2adc81861cf34ffa
2019-08-07 20:17:41 +01:00
Luigi Toscano
d89f67aadf stage-output: fix the archiving of all files
The archival task currently relies on the content of all_sources,
but some files are renamed afterwards if their extensions match
one of the special extension of the list.
This means that few files from all_sources are not found and then
not compressed.
Change then the logic: simply discover all files inside logs
and compress all of them.

Change-Id: I7d34d7d90849736b7b842c0bdd67492816f98ebc
2019-03-22 16:05:43 +01:00
Paul Belanger
a5dbe86b4b Move chdir under args for shell task
This should fix a syntax issue on ansible 2.7.

Change-Id: I006ff64c40875b8f1edd1e9f9bdfe509e2bdb295
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-03-19 19:17:01 -04:00
James E. Blair
dd8c88354e Fix ownership change in stage-output
If the stage_dir/logs directory was a symlink, Ansible would refuse
to perform the chmod.  However, if we tell Ansible that it is a
directory, it's fine.  This (perhaps curiously) does not tell
Ansible to turn the symlink into a directory.  It just tells it
that everything is okay.

Change-Id: Idbd78b97b95ce50b23dda8f384e32359d31013e3
2019-02-05 09:32:02 -08:00
James E. Blair
98628f718a stage-output: Add the ability to add .txt extensions to files
So that this can more easily work with files like '/var/log/syslog',
make it so that if the user specifies "logs_txt" then the file gets
a .txt extension before being placed in the logs directory.

The existing extensions_to_txt variable doesn't work for this because
it requires an existing extension.

Change-Id: I34fea0c44030c04a5540d6b62976557143289196
2019-01-31 10:48:58 -08:00
James E. Blair
2abb8a1302 Allow stage-output to copy root-owned files
This makes it easier to tell stage-output to copy files such
as syslog from remote nodes.

Change-Id: Ica2397a50de1054f5d5ec81b54ab36fda9bdc333
2019-01-30 15:38:58 -08:00
Andrea Frittoli
b3615c0d97 Remove support for extensions as lists
With tempest devstack and networking-odl switched to using dict,
we can now remove support for using lists for the list of extensions.

Depends-on: https://review.openstack.org/539686/
Depends-on: https://review.openstack.org/539708/
Depends-on: https://review.openstack.org/539854/
Change-Id: I62aaa737300c3b24e416f18e4bc469e7bd46ea29
2018-04-03 16:13:52 +01:00
Colleen Murphy
d1956bd9f7 Fix ansible-lint job
The 'linters' tox environment was not running on roles, it was only
running on playbooks. This change adds a command to the linters
environment to ensure all roles are linted.

Since these weren't being linted, there were some problems with them.
The first was a warning about usage of the shell module. Both of these
usages seemed appropriate, so this patch adds a skip_ansible_lint tag to
each task that was failing. The second is a warning "no action detected
in task" for zuul modules. This is due to the fact that the linter
cannot find the custom module. One option is to set the ANSIBLE_LIBRARY
path to point to the zuul ansible library directory[1], but the linter
virtualenv does not actually have zuul installed. Instead, we just
disable the linter for the failing tasks.

This also cleans up a comment in the tox file that was referring to a
nonexistent zuul job.

[1] https://github.com/willthames/ansible-review/issues/16

Change-Id: Ie49da9a09733b623bb25c5a4c8aa07eacacf4b33
2018-03-28 23:10:22 +02:00
Andrea Frittoli (andreaf)
ac60bc714b Improve file renaming in stage-output
We only rename the extensions for files specified in zuul_copy_output
but we should rename files in folders specified in zuul_copy_output as
well. In addition we need to rename files or folders that start with a
'.' which would otherwise not be visible once uploaded to logs.

Change-Id: Id55ab51019d10d5ccbc6531549758ac71ad276be
2018-02-16 08:03:56 +00:00
Zuul
46fb37f8bb Merge "Change the list of extensions to a dict" 2018-02-09 07:47:46 +00:00
Andrea Frittoli (andreaf)
f7fd2fce61 Change the list of extensions to a dict
The stage-output role accepts a list of extensions to be replaced as txt
so that they can be seen directly from within a web browser.
Using a list means that children jobs cannot add to the parent setting,
only overwrite. Adding support for dicts while preserving, at least for
now, support for lists - to avoid breaking existing implementations.

This reverts change 844b3333aae02d596061386b5a1a2fbc821950be.

Change-Id: I5052c873f0a5da7ecaa38627e8900af86bbec2f9
Needed-By: https://review.openstack.org/539686
Needed-By: https://review.openstack.org/539708
Needed-By: https://review.openstack.org/539854
2018-02-08 18:52:13 +00:00
Zuul
b16399ff4b Merge "Revert "Change the list of extensions to a dict"" 2018-02-02 19:28:48 +00:00
Andreas Jaeger
844b3333aa Revert "Change the list of extensions to a dict"
This reverts commit 146fa7b2e616ec4f2ba15e8445a666a086549cc3.

This does not work yet as planned. Let's revert for now and test properly again.

Change-Id: Idac7e398fe4854769589411f22f7c4ce7cb363f6
2018-02-02 18:33:12 +00:00
Andrea Frittoli
36aeb95399 Add no_log to verbouse tasks
Two tasks in stage-output "Register sources" and "Set source and
destination for files and folders" spam the whole output of "stat"
to be printed to console for each and every file to be staged.
The info is printed twice and it's hardly valuable for debugging purposes.

The following task, "Build a list of source, dest dictionaries for text
file", prints a more compact list of all files and folders that are going
to be staged, which I believe is enough for debugging.

To solve this add no_log to the two tasks mentioned above.

Change-Id: Id52047c429233c6684f2fca5537674f91fb2603d
2018-02-02 14:31:22 +00:00
Andrea Frittoli (andreaf)
146fa7b2e6 Change the list of extensions to a dict
The stage-output role accepts a list of extensions to be replaced as txt
so that they can be seen directly from within a web browser.
Using a list means that children jobs cannot add to the parent setting,
only overwrite. Adding support for dicts while preserving, at least for
now, support for lists - to avoid breaking existing implementations.

Change-Id: Ifc4b2070c5844e51acb9ad5d512e9f25f5bdd7ef
2018-01-31 20:58:38 +00:00
Andrea Frittoli (andreaf)
68dec9ecc5 Dereference symlinks when staging artefacts
If an artefact is a symlink, dereference it before coying it as
just copying the link may render it broken one the staged folder
is synced to the executor.

Change-Id: I5ad4dfccbc1389da3afc53f3c866d3475e006db6
2017-12-18 16:06:41 +00:00
Andrea Frittoli (andreaf)
2ea8661801 Add compress capabilities to stage artifacts
Compress files and files in folders. Use gz by default.

Change-Id: I72796e64a65fe88d85168734ac881ee3ec2ce100
2017-11-16 12:02:00 +00:00
Andrea Frittoli (andreaf)
786c48f051 Add a generic stage-output role
This role does roughly what save_dir and save_file do in d-g
functions, only it can be used for a list of files and/or
folders. Jobs can use this role to rename artifacts and place
them in a staging folder which is then used for mass sync to
the executor.

This is typically used in post - the zuul user won't necessarily
have write access to the folders with the artifacts - so it's
easier to stage everything to a folder owned by the zuul user.

Change-Id: Id45a50821539985710da7172a9185fd91e803603
2017-11-02 07:22:26 +00:00