This updates to ansible-lint 6.8.2.
The prior changes have updated various things for new things the
linter picked up.
As noted inline, the jinja2 parsing has not been working well with
this repository. I've been keeping an eye on it over several
releases, but I think at this point it's not going to work well for
us. I've left discussion in .ansible-lint.
The var-spacing in stage-output is the old name for the jinja2
filtering; since that is now ignored in .ansible-lint it can be
removed.
Change-Id: Ia2c9392eeb232b9b2b1d4febce8493d71de64482
Newer ansbile-lint finds "when" or "become" statements that are at the
end of blocks. Ordering these before the block seems like a very
logical thing to do, as we read from top-to-bottom so it's good to see
if the block will execute or not.
This is a no-op, and just moves the places the newer linter found.
Change-Id: If4d1dc4343ea2575c64510e1829c3fe02d6c273f
Using "For Example::" marks the next block as code, so we end up
emiting ".. code-block:: yaml"[1] as a code block which we don't really
want. Remove the additional ':' so we emit the expected HTML.
[1] Check https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-stage-output
before merge
Change-Id: Ic2e1fb9acb6a6b4ec77bf1ee0ec9ac5d809dfb7c
The stage-output role's README indicates that it stages to the
zuul_output_dir on the remote node, but this isn't true. It actually
stages to {{ stage_dir }} which has a different default than
{{ zuul_output_dir }} in fetch-output.
Update the readme to make this more clear and reduce confusion.
Change-Id: I7a7ef801db8194a7101a8dc0a6d89e1292b3fa86
This commit in Ansible:
9142be2f6c
now allows Python modules to specify their interpreter with the shebang.
We expect our roles to use the discovered python interpreter on remote
nodes, and on the executor, we need them to use the virtualenv. Removing
the specific shebang accomplishes this under Ansible 6, and has no effect
under older versions of Ansible.
Without this, for example, the log upload roles would not have access to
their cloud libraries.
Also update our ansible/cli check in our module files. Many of our modules
can be run from the command line for ease of testing, but the check that we
perform to determine if the module is being invoked from the command line
or Ansible fails on Ansible 5. Update it to a check that should work in
all 4 versions of Ansible that Zuul uses.
Change-Id: I4e6e85156459cca032e6c3e1d8a9284be919ccca
This replaces ansible loop tasks which rename log files to have a .txt
suffix with an ansible module task. The reason for this is each ansible
loop iteration can be very slow; up to 3 seconds per task. When
many log files need to be renamed this adds up quickly. The module can
do all of the renames in a single python process that should be much
quicker.
Note the old interface expected a list of extensions to rename. It then
converted that list of extensions into a regex used to find the files to
rename. The new code does not use regexes. It is possible someone abused
the old interface to hijack the regex and do something extra fun, but
that was not documented nor likely to be safe.
Change-Id: I7033a862d7e42b3acd8ad264aefca50685b317ff
This updates to ansible-lint 6. Some prior changes have updated for
the bulk of the rules, a couple of noqa points are added here. Some
updates to the skipped rules are added. This should have no
operational change, the only updates are cosmetic.
Change-Id: I165677bbb904f92292df00f7b9b27f8f3573aeb0
This is preparation for a later version of ansbile-lint, which finds
missing names on blocks. This seems a reasonable rule, and the
Ansible manual says [1]
Names for blocks have been available since Ansible 2.3. We recommend
using names in all tasks, within blocks or elsewhere, for better
visibility into the tasks being executed when you run the playbook.
This simply adds a name tag for blocks that are missing it. This
should have no operational change, but allows us to update the linter
in a follow-on change.
[1] https://docs.ansible.com/ansible/latest/user_guide/playbooks_blocks.html
Change-Id: I92ed4616775650aced352bc9088a07e919f1a25f
This fixes a number of places where we do not have spaces between
filters. I think that this is a reasonable rule for readability (I
also think it probably was enforced, but maybe later versions got
better at detecting it?).
These are detected by a later version of Ansible lint; this change
should have no operational change to any roles but prepares us to
update in a follow-on change.
Change-Id: I07e1a109b87adce86f483d14d7e02fcecb8313d5
The stage-output role had two assumptions:
1) The zuul_copy_output variable would always be defined
2) The role would be able to sudo on the remote node
To make it easier for users who want to use this in a base job,
remove both of those assumptions.
1) We now supply a default empty dict if the variable is not defined
so that the role does not fail with an error.
2) We check to see if we can sudo on the remote node, and if we can,
we do when copying files; otherwise we don't, and assume that the
user will only specify files they have access to (if they don't then
the copy will fail).
Change-Id: I5428c44adfafac4872342b59a92de311f41687b6
Stop compressing files that would otherwise be handled uncompressed in
intermediate log handling. The reason for this is final log/artifact
upload should decide if a file need to be compressed or not as some file
hosting services (like swift) may not return the expected data if we get
this wrong.
This was noticed when we used swift to store intermediate tar.gz
tarballs before final upload to permanent storage. When retrieving the
tar.gz files from swift we got them back as uncompressed .tar files
(because swift was being helpful) and then later when we try to
uncompress these files we break.
Instead we'll let the upload to swift (or other storage) decide if files
should be compressed further than their existing state.
This is the second change in the stack and update the stage-output role.
Change-Id: I2fa6f123d23568ba1537a7c26d2672f04110c35b
With the arrival of ansible-lint 4, Jinja2 variable expansions must
include spaces before and after the variable name inside the
brackets.
Adjust the new violations accordingly and remove the rule
206 exclusion.
Change-Id: Ib3ff7b0233a5d5cf99772f9c2adc81861cf34ffa
The archival task currently relies on the content of all_sources,
but some files are renamed afterwards if their extensions match
one of the special extension of the list.
This means that few files from all_sources are not found and then
not compressed.
Change then the logic: simply discover all files inside logs
and compress all of them.
Change-Id: I7d34d7d90849736b7b842c0bdd67492816f98ebc
This should fix a syntax issue on ansible 2.7.
Change-Id: I006ff64c40875b8f1edd1e9f9bdfe509e2bdb295
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
If the stage_dir/logs directory was a symlink, Ansible would refuse
to perform the chmod. However, if we tell Ansible that it is a
directory, it's fine. This (perhaps curiously) does not tell
Ansible to turn the symlink into a directory. It just tells it
that everything is okay.
Change-Id: Idbd78b97b95ce50b23dda8f384e32359d31013e3
So that this can more easily work with files like '/var/log/syslog',
make it so that if the user specifies "logs_txt" then the file gets
a .txt extension before being placed in the logs directory.
The existing extensions_to_txt variable doesn't work for this because
it requires an existing extension.
Change-Id: I34fea0c44030c04a5540d6b62976557143289196
Remove whitespaces from build-python-release which make
the sphinx display as "** Role Variable
**" instead of bold "Role Variables".
Fix buildset-artifacts-location to display items as list.
Fix stage-output to display the example as example - and not wrap
around.
Change-Id: I8da3ec0c507d26f47ee9bb71dc9d1b3ed9ca23a5
The 'linters' tox environment was not running on roles, it was only
running on playbooks. This change adds a command to the linters
environment to ensure all roles are linted.
Since these weren't being linted, there were some problems with them.
The first was a warning about usage of the shell module. Both of these
usages seemed appropriate, so this patch adds a skip_ansible_lint tag to
each task that was failing. The second is a warning "no action detected
in task" for zuul modules. This is due to the fact that the linter
cannot find the custom module. One option is to set the ANSIBLE_LIBRARY
path to point to the zuul ansible library directory[1], but the linter
virtualenv does not actually have zuul installed. Instead, we just
disable the linter for the failing tasks.
This also cleans up a comment in the tox file that was referring to a
nonexistent zuul job.
[1] https://github.com/willthames/ansible-review/issues/16
Change-Id: Ie49da9a09733b623bb25c5a4c8aa07eacacf4b33
We only rename the extensions for files specified in zuul_copy_output
but we should rename files in folders specified in zuul_copy_output as
well. In addition we need to rename files or folders that start with a
'.' which would otherwise not be visible once uploaded to logs.
Change-Id: Id55ab51019d10d5ccbc6531549758ac71ad276be
The stage-output role accepts a list of extensions to be replaced as txt
so that they can be seen directly from within a web browser.
Using a list means that children jobs cannot add to the parent setting,
only overwrite. Adding support for dicts while preserving, at least for
now, support for lists - to avoid breaking existing implementations.
This reverts change 844b3333aae02d596061386b5a1a2fbc821950be.
Change-Id: I5052c873f0a5da7ecaa38627e8900af86bbec2f9
Needed-By: https://review.openstack.org/539686
Needed-By: https://review.openstack.org/539708
Needed-By: https://review.openstack.org/539854
This reverts commit 146fa7b2e616ec4f2ba15e8445a666a086549cc3.
This does not work yet as planned. Let's revert for now and test properly again.
Change-Id: Idac7e398fe4854769589411f22f7c4ce7cb363f6
Two tasks in stage-output "Register sources" and "Set source and
destination for files and folders" spam the whole output of "stat"
to be printed to console for each and every file to be staged.
The info is printed twice and it's hardly valuable for debugging purposes.
The following task, "Build a list of source, dest dictionaries for text
file", prints a more compact list of all files and folders that are going
to be staged, which I believe is enough for debugging.
To solve this add no_log to the two tasks mentioned above.
Change-Id: Id52047c429233c6684f2fca5537674f91fb2603d
The stage-output role accepts a list of extensions to be replaced as txt
so that they can be seen directly from within a web browser.
Using a list means that children jobs cannot add to the parent setting,
only overwrite. Adding support for dicts while preserving, at least for
now, support for lists - to avoid breaking existing implementations.
Change-Id: Ifc4b2070c5844e51acb9ad5d512e9f25f5bdd7ef
If an artefact is a symlink, dereference it before coying it as
just copying the link may render it broken one the staged folder
is synced to the executor.
Change-Id: I5ad4dfccbc1389da3afc53f3c866d3475e006db6
This role does roughly what save_dir and save_file do in d-g
functions, only it can be used for a list of files and/or
folders. Jobs can use this role to rename artifacts and place
them in a staging folder which is then used for mass sync to
the executor.
This is typically used in post - the zuul user won't necessarily
have write access to the folders with the artifacts - so it's
easier to stage everything to a folder owned by the zuul user.
Change-Id: Id45a50821539985710da7172a9185fd91e803603
