There was a regression introduced in ansible-compat v4 which stopped
ansible-lint from working. This was fixed in ansible-lint 6.16.0 but
we've pinned it for so long now that there are a lot of rule
violations to fix or mark for skipping. In the interim, pin back all
new versions of ansible-compat until someone has time to fix things
or rip out ansible-lint entirely.
Change-Id: I68e06bbfaf0b38582320f09659dfc462016eacf8
Users struggle with Gerrit ACL formatting, often going through 4 or
5 revisions to get things like indents, trailing whitespace,
alphabetical ordering and final newlines correct. With this testenv,
users can run something like this to auto-reformat their Gerrit
ACLs...
tox -e normalize-gerrit-acl -- gerrit/acls/foo/bar.config
We'll include this as a suggestion in the Infra Manual section on
adding projects as well.
Change-Id: I5f0903849ea72f9d1c81164bed4c651c9423f1fc
This avoids another broken release of ansible-compat per the linked
bug (assuming the next version of ansible-lint will fix it, so
optimistically not using <), and is a continuation of the workaround
we merged for the prior version.
Change-Id: Id2369dc9a16a85981fd9b7122976f3b6120891f2
This avoids a broken release of ansible-compat per the linked bug
(assuming the next version will fix it, so optimistically not using
<), and is the same workaround we merged to the
openstack/openstack-zuul-jobs project in
https://review.opendev.org/881938 earlier today.
Change-Id: Ia43e8e481e34320bca2bed4a32b780f997c24c85
With tox4 in use for testing, tox
run failing with passenv value and
external commands run.
failed with pass_env values cannot contain
whitespace, use comma to have multiple values in a single
line, invalid values found 'http_proxy HTTP_PROXY https_proxy
HTTPS_PROXY no_proxy NO_PROXY' [tox/session/cmd/run/single.py:54]
Fixing passenv value and adding external commands
via allowlist_externals.
Change-Id: I05d36dab6895dfc14d026230592def51b78bb755
This convers whitelist_externals to allowlist_externals to make tox v4
happy. Note that this may not be a complete fix for tox v4 as
project-config isn't tested speculatively which makes it harder to find
all the cases.
Change-Id: Ia44a2ef83cdc57d0bb74c72ea5c95078b53ccaec
It's https://github.com/ansible/ansible-lint/issues/2320 which is
blocking our use of newer ansible-lint (though it's worth revisiting
whether testing with ansible-lint is still of any real value).
Change-Id: Ie3e42df7f90e5349bf1513e8896b1d5e16125b42
We try to test our job content with the same ansible-lint versions
used by the zuul/zuul-jobs repository. Adjust our exclusions to
match what it's using, for consistency and ease of future
maintenance.
Change-Id: I2300e8950780245a38f68ce9cc5aa5ff62e65e40
For local runs of testenv:linters we cache all the external Ansible
playbooks and roles ansible-lint will need to reference in order to
validate content in this repository. We build that cache in a .cache
directory, but not everything in the cached repositories is suitable
for flake8 checking, so add that directory to its exclusion list.
Also add the same directory to .gitignore, so Git will stop
complaining about it being untracked.
Change-Id: I1f740edd75cc7f6ba0f4a73f0a409a620b437b16
ansible-lint introduced a regression as described in
"Don't use ansible-lint 6.5.0" I1770e034772b55b993d276d580c5a75d059bd822
the fix is still in draft and 6.5.1 has been released so lets skip that
too
Change-Id: I641a1246f870869153df782381fd9dafa180da26
We have some skips in our playbooks which 6.5.0 ignores because of
https://github.com/ansible/ansible-lint/issues/2320 so block that
version and hope the fix makes it into the next release.
Change-Id: I1770e034772b55b993d276d580c5a75d059bd822
This updates ansible-lint to the 6.x releases. It also updates the
ansible installed to our current zuul version.
It cleans up the config file and marks it as yaml. A few new
exceptions are added with explainations.
We don't need to fake the zuul_return by telling ansible-lint to mock
it. This is added to the config, and we can remove the stub file.
A constant source of problems running this locally is that you have to
have the other role repos checked out so ansible-lint can do its
checks. Add a smaller helper script for doing this locally. In the
gate, ANSIBLE_ROLES_PATH is set to the Zuul checkouts of these
projects. Locally, add a smaller helper "ansible-lint-roles-cache.sh"
that pulls the projects into a .cache directory. If they are already
there, they get updated. By default locally we will use these
checkouts. This way, "tox -e linters" just works without having to do
anything else.
This also modifies the xargs to run the check all at once, instead of
fork for each file. I did try autodetection but it seems like other
yaml files in the roles/playbook directories still confuse
ansible-lint.
Also I don't think we need a ansible-playbook --syntax-check step;
ansible-lint covers that.
Change-Id: I972f73037b9f904a555b81f3835ca5261639ed01
grafyaml requires slugify which had a release yesterday. The py3 wheel
is being installed in the tox env, leading to Syntax errors
Change-Id: I2d577d7cc78e752ad29001e09d7e5134cfc106f7
ansible-lint is currently broken when older than a recent release. This
is due to to the 'rich' dependency updating without a pin and not
working with older ansible.
Change-Id: Id93a2bc0b452bafca1dca8550ffc825d72533cdc
This is a combination of 2 commits:
Fix the renaming of ansible-role-refstack-client
In [0] that project was moved from x/ to openinfra/, but the patch
missed that at the same time refstack was also moved, changing the
path for the ACL.[1]
[0] https://review.opendev.org/c/openstack/project-config/+/765787
[1] https://review.opendev.org/c/openstack/project-config/+/808479
Fix use of yaml.load()
The use of this function has been deprecated for a long time[2]. With
PyYAML==6.0 the call is now failing, so replace it with the safe
version.
[2] https://msg.pyyaml.org/load
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I439d1f167b5893a134a5b628ebc00fd6617e0cbd
When upgrading from Gerrit 2.13 to 3.2 we stopped relying on a local
fork of its source and have been building from (more recently
completely unmolested) upstream source. This fork was the only place
we were relying on jeepyb's track-upstream feature, so we can stop
looking for it in our checks and normalization as well.
Remove the check_gerrit_projects_changed scripts as well, as they
only existed to run things where track-upstream was enabled.
Change-Id: I597c1a577c53e2db61413d9ec531378667691d2a
Make some adjustments to the IRC access check script so that it
works in the OFTC network now. Also update the channel config to
reflect the new ACL paradigms there. Remove our volunteer operators
temporarily until we can confirm their nicks there. Also rip out the
channel forwarding for unregistered nicks, we can work on readding
it there later if we determine it's necessary after all, but it will
need implementing differently anyway if so.
Change-Id: Ib3c43ef5ba22191d869629cd01d3800f3e235ea4
ansible-lint is incompatible with Ansible 2.7 which is now causing
failures. Update to current Zuul versions.
Change-Id: I65878a896281ad67d512f0bd9219aa2058e4953a
Also, install yamllint in the dib env, as it's a requirement
of dib-lint now but is only in test-requirements for dib.
Change-Id: I083bca901ca51438099d1d3bbbd0076ac3d7da07
Zuul-jobs tox roles wants to be able to run tox --showconfig -e linters.
Unfortunately our linters target has an unescaped % symbol in it which
causes python3 + tox to fail with an interpolation error. Escaping the %
to avoid interpolation addresses this.
Change-Id: I7e23595495909fc94a7ea5a9e461e67e25ddeae9
- Use zuul_return fake module to make ansible lint happy, this allows
to remove Zuul import.
- While ansible-lint 4.2.0 is now able to detect playbooks/roles, this
is broken, so don't use it
- move its config out of tox.ini so it can be used by any tools, or
without tox
Change-Id: Ie8935f47db855647e19ae091044e5ac1871f1551
Co-Authored-By: Sorin Sbarnea <ssbarnea@redhat.com>
Co-Authored-By: Andreas Jaeger <aj@suse.com>
This version of hacking doesn't understand f-strings as usable in
Python 3. Update to the latest and fix current issues, which are all
just formatting fixes.
Change-Id: I0a7d6f93f07477b6dd29ab143130dd9064c250be
For using opendev-tox-docs, jobs need to publish to a "html"
subdirectory, update the docs-site and specs jobs.
Update .gitignore for this.
Update python files to use yaml.safe_load to avoid a warning.
Remove unused docs environment from tox.ini.
Depends-On: https://review.opendev.org/682213
Change-Id: I8d7032a6e836ec26f347d16abd600514089a677b
Remove hardcoded list of namespaces with iterating over all directories
in the check of valid ACLs.
Remove parameter, we don't need to pass it in.
Be more verbose and report directories scanned and number of ACL files
read.
Change-Id: I4193ac59b431d31f3ce835c93950745e5b742418
libyaml now needs to Loader keyword to not throw a warning, use
safe_load instead everywhere.
sort_keys isn't supported for reasons surpassing understanding, so
switch to ruamel.yaml for the places where we write contents back
out.
Our ruamel.yaml wrapper strips extra leading space, which is needed
when the root object is a list. When the root object is NOT a list,
it's not needed. Add a flag to the constructor to toggle the
behavior.
For gerrit/projects.yaml this leads to a different wrapping of
description fields, which is included here.
Let ruamel.yaml handle comments
Since we've switched to ruamel.yaml, we don't need to strip
comments. Stop stripping them. Intra-list comments aren't
supported, so just remove them.
Please don't ask me why we can't have a space before openstack-merlin.
Change-Id: I72776774eabd4e397ee5e8bdd58b3632be8fd3a1
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.
This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.
This update should result in no functional change.
For more information see the thread at
http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html
Change-Id: I0cc16c32dcea3b6036553c9f9e8df45a241254e0
With multi-ansible support, the location of the zuul plugins/modules
has moved.
Also, handle the sort_keys argument which is new in pyyaml 5.1.
Set it to False to achieve the old behavior.
Also use python3 by default for tox tests on this repo (note,
one env still explicitly requests 2.7).
Change-Id: I8e2cf1f2276bca5a912fed52062628b3b9d0d6d9
The zuul_return plugin was recently converted to an action plugin, make
sure we also export that path for ansible-lint.
Change-Id: Id74ccb58cb436ce40bd643b6def7b1db4a280c46
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We only need it in the nodepool env. Also, install from pip and let
tox-siblings take care of updating from source for us.
Depends-On: https://review.openstack.org/611629
Change-Id: I0ece7b751662b4198276e88adcf263634fd792d6
We install zuul so that we can pick up library entries. The proper path
for this is envsitepackagesdir. Also, just list zuul in the requirements
so that we're not cloning remotely. tox-siblings will take care of it in
the gate for us, because we add zuul to required-projects in the
openstack-zuul-jobs-linters job.
This adds a setup.py and setup.cfg so that tox-siblings can do its
thing.
Depends-On: https://review.openstack.org/611607
Change-Id: I864308af3424a0b0505a252d0d5cf44277172fbf
It's probably reasonable to assume that for interactive use, the
required roles for linting in zuul-jobs and openstack-zuul-jobs are in
sibling directories of project-config. Set this as the default
environment value.
Change-Id: I388fdae998ac3cb87c30616ed761c888e1e429f0
I noticed this when I tried to use Ansible 2.5 "loop:" constructs.
Unfortunately, we can't just rely on bringing in zuul to pull the
right version of ansible. ansible-lint being uncapped just takes it
over, and we end up with the latest version. Pin it to the current
zuul requirements and add a note (maybe one day
https://github.com/pypa/pip/issues/988 will get a fix ...)
Change-Id: Iaf8f0f5cdc46df41fec7c436c1179c80bb5c368e
A YAML syntax error in site-variables was merged recently. Errors in
that file break basically everything, so add a check that the file at
least parses.
Change-Id: I9b2c47a0461275e1d53aabe387f0309719d46b77
Currently this calls out to grafyaml-validate.sh which tries to
install zuul-cloner and then pull grafyaml from that.
Since zuulv3 doesn't *have* zuul-cloner, this means the job is broken
for local testing. It works in the gate only because zuul-cloner
compatability shims are installed.
To maintain the testing-against-master behaviour, the dependent change
provides the grafyaml source in the environment; otherwise we go to
git directly.
Depends-On: https://review.openstack.org/572251
Change-Id: I3d5cd062414b77aaa70d2669730fda870ca0168e
The 'linters' tox environment was not running on roles, it was only
running on playbooks. This change adds a command to the linters
environment to ensure all roles are linted. This also cleans up a
comment in the tox file that was referring to a nonexistent zuul job.
Change-Id: I04d9d79fca16a7f78f9c646f96e5fba78a5f5329
Now that nodepool only support python3, best to test against it.
Change-Id: Iaa85c6f7f9d64b03c62e80437822735b6a0fa085
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Add back in tox.ini changes to test all versions of nodepool.yaml files.
This reverts partially commit 7c4da0c8b8.
Also, test nodepool master branch instead of deleted feature/zuulv3
branch.
Change-Id: I0b04d251f62efc362a80a7c6fb595e76c4bf3d76