34 Commits

Author SHA1 Message Date
Ian Wienand
8c6512107c
Standarise block/when ordering
Newer ansbile-lint finds "when" or "become" statements that are at the
end of blocks.  Ordering these before the block seems like a very
logical thing to do, as we read from top-to-bottom so it's good to see
if the block will execute or not.

This is a no-op, and just moves the places the newer linter found.

Change-Id: If4d1dc4343ea2575c64510e1829c3fe02d6c273f
2022-11-07 10:37:53 +11:00
James E. Blair
82d3910b17 More narrowly tailor the ensure-pip Debian workaround
The docker image that we build the zuul executor from is a Debian
image, but it does not follow the same python3 policies as Debian
itself.  While we would not necessarily expect all roles to work
on the executor, it is reasonable to want to use the ensure-pip
role (which logically should be a no-op on the executor) for the
side effect of finding and returning the appropriate pip command.

Currently, the role fails on the executor because it mistakenly
concludes that it must install python3-venv to get a working
venv module.  By increasing the precision of the check for what
is missing (the actual error is a missing "ensurepip" python module
(oh irony!), we can avoid attempting an installation of
python3-venv on python docker images (including the Zuul executor
images).

This also adds the ensure-pip-localhost job

This tests that the ensure-pip role works on the Zuul executor.

The executor is a debian host with a working python environment,
so it should be a no-op (and no packages should need to be installed).

Change-Id: Id7f13f2f73d45e680f79c00a83751b185212a63d
2022-10-04 11:35:24 -07:00
Ian Wienand
a016a1a565 linters: standardise on newline at end of file
I noticed this by accident when I ran ansible-lint over this repo from
an outside context; it didn't use the .yamllint in here and started
compalining about eof whitespace.

After scratching my head for a bit as to why this didn't fail here, I
realised we've allowed various newlines since the initial commit
I936fe2c997597972d884c5fc62655d28e8aaf8c5.

Remove this and just use the default eof rules, and fixup the
whitespace as required.  This is fairly unimportant, but is nice for
consistency.

Change-Id: Idb46a1f39ba798b0bf70eaa27b4c6b4758ce3d26
2022-07-28 16:19:06 +10:00
Ian Wienand
6d23d20f2f linters: add names to blocks
This is preparation for a later version of ansbile-lint, which finds
missing names on blocks.  This seems a reasonable rule, and the
Ansible manual says [1]

  Names for blocks have been available since Ansible 2.3. We recommend
  using names in all tasks, within blocks or elsewhere, for better
  visibility into the tasks being executed when you run the playbook.

This simply adds a name tag for blocks that are missing it.  This
should have no operational change, but allows us to update the linter
in a follow-on change.

[1] https://docs.ansible.com/ansible/latest/user_guide/playbooks_blocks.html

Change-Id: I92ed4616775650aced352bc9088a07e919f1a25f
2022-07-27 17:13:39 +10:00
Ian Wienand
372f3af706 Revert "Install venv for all platforms in ensure-pip"
This reverts commit 0aca37281458b94b30fd124f885cadc18ad3c61e.

This uses the variable ensure_pip_from_upstream_interpreters to
install python-venv packages.

ensure_pip_from_upstream_interpreters is documented as being a list of
python interpreters to run "get-pip.py" with to install pip globally.
This only happens if ensure_pip_from_upstream is set to True.

It's unclear from the original discussion in the change at least, what
this was trying to fix.  It's also unclear why this would skip
packages, and what circumstances that is required in.  It's now
causing more confusion as we try to add fixes ontop of that.

Revert this -- whatever this is doing, we should find the right way to
do it, which isn't this.

Change-Id: Ieada7ec2d2c7320d7aacb9a5dc6de52394c11ed1
2022-06-17 14:52:20 +10:00
Ian Wienand
4d4e386bad ensure-pip: fix typo in ensure_pip_virtualenv_command documentation
Fix a typo in the documentation for this variable.

Change-Id: I8da3d5e512e8fddccf6bd6a39078579c7433fba9
2022-02-28 14:52:30 +11:00
Dariusz Smigiel
1f76342ccf Pull correct upstream pip version
Recent release of pip removed support for Python 3.6 [1]

Configure default url for pip, depending on supported
Python version.

[1]: https://pip.pypa.io/en/stable/news/#v22-0

Change-Id: I01faeba1dff864de31b3bbf9ddbe0a4d454fdbe9
2022-02-01 08:33:58 -08:00
Zuul
4d3b20b351 Merge "ensure-pip: Clarify situation with ensure_pip_from_packages_with_python2" 2020-08-26 23:14:22 +00:00
Tristan Cacqueray
594aa7e6f5 ensure-pip: add instructions for RedHat system
This change adds instructions to help user figure out what to do when
unittest job fail since https://review.opendev.org/736070 requires
epel to be installed.

Change-Id: Ie69ba81405bc3fbc7666c7ee12ed4996c5323709
2020-08-07 12:27:36 +00:00
Ian Wienand
50375ac31c ensure-pip: Clarify situation with ensure_pip_from_packages_with_python2
With I450171aad5e31d2925239ab3d0641cd23f6815a2 we defaulted
ensure_pip_from_packages_with_python2 to True when Ansible was running
under Python 2.  Clarify in the documentation that it will be set in
this condition.  Also clarify that EPEL is required to install Python
2 pip packages on CentOS 7.

The RedHat.yaml started installing python2 packages unconditionally
when running under Python 2 with
I2ab11bb45b6b2a49d54db39195228ab40141185c.  This should have just
relied on ensure_pip_from_packages_with_python2 variable, update that.
Thus this will default to True when under Python 2, but will allow
people to turn off including the Python 2 packages if they know they
will not need them (e.g. they may run Ansible under Python 2, but only
ever use python3 in testing, and thus can avoid the epel dependency).

All of the distribution roles actually install the Python 3
pip/setuptools/wheel packages unconditionally when
ensure_pip_from_packages is set; they do *not* install it for the
``ansible_python_interpreter`` as implied by the documentation.

Change-Id: Ib948c8a156a0a96d610a310b6291781c5d3c7f96
2020-07-23 10:55:18 +10:00
Sean McGinnis
09b4be329e
Strip path from default ensure_pip_from_upstream_interpreters
The ensure-pip role has an option to provide multiple interpreters to
use. If not specified, we default to ansible_python.executable. This
default includes the full path to the executable. Those manually
providing which interpreters to use will not know what the full path is,
only the versions they would like to use. To make things consistent,
this strips off the path so we just have the version (python, python3,
python3.8, etc).

Change-Id: I339afc08393e9c6b1d26a05cf13b6fdc151f46d5
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-07-10 11:24:27 -05:00
Sean McGinnis
0aca372814
Install venv for all platforms in ensure-pip
The ensure-pip tasks will install python3-venv, but multiple
interpreters may be specified. This adds handling to make sure the venv
package is installed for all requested interpreters.

Change-Id: I670d5815bfc902f9c50a98df715ef60b61bab594
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-07-09 16:02:45 -05:00
Albin Vass
ed97235bf8 ensure-pip debian: update package lists
The 'package' module doesn't update any package lists before installing
packages which can cause problems if the package lists haven't been
updated in a while. So run 'apt-get update' before
installing pip.

Change-Id: I836f7faa25dd8c06d0158828e21226fbe93fbb40
2020-06-23 15:58:12 +02:00
Andreas Jaeger
83982f9fb7 nit: Fix missing word in roles/ensure-pip/README.rst
Change-Id: Ibb898d0d50a8dfb79ae1def2d56e39daf1e4ffdb
2020-06-18 13:57:17 +00:00
Ian Wienand
67f223b53a Partial revert "Ensure wheel exists for build-release-python"; move to ensure-pip
This partially reverts commit
3f961ce202d7d24e2944de09636b35cec9c13bf6.

This alternative installs wheel with the ensure-pip role instead of in
a separate role.  wheel is very closely linked with pip install
operations so this isn't a large overreach of the role.

I suggest this for several reasons; firstly the python-wheel role
doesn't try to install packages, so we end up with mixed system pip
and upstream versions of wheel most of the time.  This is the type of
thing that has proven problematic in the past.  It also installs via
pip --user; something we've already had problems with tox when for
various reasons roles want to run this as non-zuul user.  Using
ensure-pip we keep the packaged versions together.

[1] did try to install wheel with root, but during runtime which
didn't work due to sudo being revoked.  This should work for the
existing build-python-release job, because it already includes
ensure-pip in pre-run via playbooks/python/pre.yaml

I believe our conclusion on the ensure-* roles was that requiring
root/become: for installation is OK, but we should have a no-op path
if the tools are found.  This is consistent with that approach
(i.e. if you want wheel and can't do sudo, you should pre-install it
on your image using whatever you build that with).

This adds a check to the existing "is pip installed" check to also
check if wheel packages are available.  If not we trigger the install
path.

This revealed some issues with RedHat.yaml -- we can always install
Python 3 (packages available for CentOS 7) so remove that check, and
if Ansible is running under Python 2; ensure we install the
dependencies too (not only if it is forced).

Update the documentation to describe that it will enable support for
bdist_wheel, and add a basic sanity test that wheels are produced by
pip.  The existing build-python-release job is kept; although it is
modified to use the playbooks/python/pre.yaml playbook as the build
job does.

Change-Id: I2ab11bb45b6b2a49d54db39195228ab40141185c
[1] https://review.opendev.org/#/c/736001/5/roles/build-python-release/tasks/main.yaml
2020-06-18 12:51:56 +00:00
Ian Wienand
0089a2f905 ensure-pip: remove Xenial venv workaround
This was introduced with Id8347b6b09735659a7ed9bbe7f9d2798fbec9620,
but is no longer necessary as the OpenDev images don't ship with
pip-and-virtualenv.  Remove to avoid any ongoing confusion.

Change-Id: I87c4d949c9d9602dfa39023b337fa593f8fafde0
2020-06-17 10:53:06 +10:00
Ian Wienand
9926c5ac80 ensure-pip: install from EPEL for Python 2
On CentOS 7, the python-pip package comes from EPEL.  CentOS 8 and
Fedora don't have python-pip so it's safe to switch this to "yum" and
then we can use enablerepo.

Change-Id: I267f082b2db4e501299226f0033b5d940752d931
2020-06-17 10:53:06 +10:00
Albin Vass
58381f58f1 ensure-pip: update include to include_tasks
include is being deprecated and import_tasks is supported
since 2.4 so update bindep to use that instead.

Change-Id: I21670d7d687b81ac745a74cf43cbe747eefde08c
2020-05-26 10:20:33 +02:00
Ian Wienand
b6b6e36368 ensure-pip: use full python3 path
As seen by Iaa3ecd05b64af6dd9b2ee17a39bcbe6cde8686ba "python3 -m venv
--help" doesn't actually tell you if the command will work, so it's
really not helpful.  Modify this to run the venv from the python in
the path.  This also expands the comments around why a full path is
necessary.

Change-Id: I03bded0345ff734fd6c5d02fdd44ed1cc14360d2
2020-05-11 13:40:18 +10:00
Ian Wienand
bbb149ebf9 ensure-pip: always check for python3-venv on Debuntu
As explained inline, we can have systems where pip is installed, but
python3-venv isn't.

This role is assuming that "python3 -m venv --help" indicates that you
will be able to create a working venv with this command, but this is
unfortunately incorrect.  On Debuntu this requires the python3-venv
package as well.  Put in an unconditional probe for this in the
workarounds, and install it if required.

Change-Id: Iaa3ecd05b64af6dd9b2ee17a39bcbe6cde8686ba
2020-05-11 11:10:04 +10:00
Ian Wienand
f578a38693 ensure-pip : fix Xenial exported virtualenv command
This is a fix for Id8347b6b09735659a7ed9bbe7f9d2798fbec9620 which did
not specify the full path in ensure_pip_virtualenv_command for Xenial.

This slipped by testing because there we check
ensure_pip_virtualenv_command runs under a shell:, but not when called
as the argument to the pip: module (which exec's it differently and
requires the full path).  Update testing to do that too.

Change-Id: I65ff5ce913917079ab2fc1d88c56d1c0a24ea83e
2020-05-06 11:15:00 +10:00
Zuul
4f008f15c7 Merge "ensure-pip: Install backported pip for Xenial" 2020-05-05 23:18:46 +00:00
Ian Wienand
d9bd10de1a ensure-pip: Install backported pip for Xenial
It turns out the extant comment, removed here, is correct in
identifying the problem, but incorrect about the solution.

As noted the v8 pip included with Xenial doesn't fall back to PyPi
correctly when nodes are configured with mirrors.  However, the note
about virtualenv upgrading pip is incorrect.  This was not tested on
our "plain" nodes (this will be added by a follow-on
https://review.opendev.org/724776 when it can pass) so virtualenv was
picking up the pip installed by the pip-and-virtualenv element.

Installing pip from source doesn't really help; in fact it makes
things even more confusing because "python3 -m venv" still uses the
inbuilt pip from the python-pip-whl package [1].  e.g.

 root@ubuntu-xenial-plain:~# pip --version
 pip 20.1 from /usr/local/lib/python3.5/dist-packages/pip (python 3.5)
   ^ this is installed by get-pip.py
 root@ubuntu-xenial-plain:~# python3 -m venv test
 root@ubuntu-xenial-plain:~# ./test/bin/pip --version
 pip 8.1.1 from /root/test/lib/python3.5/site-packages (python 3.5)
   ^ it still deploys pip in the venv from the whl

and thus will *not* pick up the source pip install.  This is a problem
on our extant Xenial hosts, so clearly nobody is using it.  However,
as part of this work we want to standardise other tools we are
installing in zuul-jobs to use "python3 -m venv".  Thus we want all
our platforms need to support a working venv out of the box.

The solution proposed here is to install a backport of Bionic's pip 9
into Xenial when using this element.  This way, we are still shipping
packaged pip on the host and keeping our images as close to plain
vanilla upstream as possible, but with almost as small change as we
can manage to actually work in our environment.  Given the sunsetting
lifespan of Xenial, this should require not further maintenance until
we are no longer interested in the distro.

Because we skip the install phase on nodes with pre-installed pip, we
put in a work-around to set "ensure_pip_virtualenv_command" to
virtualenv on extant nodes that have been configured with
pip-and-virtualenv.  We can remove this when we have only "plain"
nodes (i.e. no pip-and-virtualenv element) and then we will
consistently be using venv's.

[1] https://packages.ubuntu.com/xenial/python-pip-whl

Change-Id: Id8347b6b09735659a7ed9bbe7f9d2798fbec9620
2020-05-06 06:41:50 +10:00
Zuul
25206338a4 Merge "ansible-lint: use matchplay instead of matchtask" 2020-05-05 20:18:14 +00:00
Zuul
79bc6abaa5 Merge "ensure-pip: use python2-pip when running under Python 2" 2020-05-05 20:12:38 +00:00
Albin Vass
bee0c6ae2f ansible-lint: use matchplay instead of matchtask
For some reason matchtask doesn't match includes, matchplay does so use
that instead.

Change-Id: I040f7f3394503e92d06c05e8ff671a43b14baebc
2020-05-05 20:42:38 +02:00
Ian Wienand
b4c195b419 ensure-pip: use python2-pip when running under Python 2
When Ansible runs under Python 2, such as on SuSE 15, we need
python2-pip for pip: to work.  Update the default to install Python 2
deps when Ansible is running under v2.

Additionally for SuSE, as described inline we can have pip but not
setuptools.  Put in a work-around to always pull in the setuptools
package.  Pull this out into a separate file; history has shown that
we may need more :/

Change-Id: I450171aad5e31d2925239ab3d0641cd23f6815a2
2020-05-05 09:27:24 +10:00
Albin Vass
d0e2016592 Add loop var policy to ansible-lint
This adds a custom ansible-lint rule at .rules/ZuulJobsNamespaceLoopVar.py
that enforces the loop var policy described at:
https://zuul-ci.org/docs/zuul-jobs/policy.html#ansible-loops-in-roles

It also updates existing roles to follow the policy.

Change-Id: I92b2ff56a1c2702542fc07b316f1809087a4c92f
2020-04-29 17:20:59 +02:00
Ian Wienand
2f7ff89384 Python roles: misc doc updates
Some minor formatting things I noticed when reading through the page
of role documentation.

Change-Id: I2f94102badd83d82073df2dfb1e36a524c7b0686
2020-04-17 10:45:26 +02:00
Ian Wienand
9e92459592 Document output variables
Several roles are setting facts that are expected to be consumed by
roles following them; generally things like the path to installed
tools, etc.

Add a section to policy about this behaviour in general, noting it
should use the cachable flags to persist the values.

Add specific documentation notes in an "Output Variable" section for
roles currently implementing this behaviour.

Change-Id: I05657fec198176c7d7345e84293f4902c81fa30c
2020-04-17 10:45:15 +02:00
Ian Wienand
88124dffb9 Update Fedora to 31
Update the Fedora test platform to 31.  Since we have dropped the
pip-and-virtualenv element with this image, we do not need the
work-around for excluded packages any more.

Story: #2007386
Task: #39310

Change-Id: I713a3cc790039dee3d28d83198f5afff5bebc491
2020-04-17 14:50:51 +10:00
Ian Wienand
6dce7ba605 ensure-pip: export ensure_pip_virtualenv_command
This makes ensure-pip export something that should be sensible for
users of pip: to put in their 'virtualenv_command' for their host.

The theory of operation is fairly simple; see if "python3 -m venv"
looks like it works, and set that as the command if it does.  If not,
set "virtualenv".  For sanity, we check if it works.

We pull in virtualenv in the Python 2 case, but for Python 3 we are
deliberately do not bring it as it is an unnecessary dependency.  If
jobs do require the actual `virtualenv` package, they should provision
it themselves ... except for Xenial, which, as described inline, has
issues.

Follow-on changes will convert existing zuul-jobs roles that install
tools into virtualenvs to use this argument.

Change-Id: Idad14c0e77eed5bf8df2c8f84f52fbdea2236a9f
2020-04-15 15:34:19 -07:00
Clark Boylan
fbf8242401 Check if pip is preinstalled before installing it
In ensure-pip we need to check if pip is available before installing it.
If it isn't available proceed and install it otherwise noop.

Change-Id: Id5c5b95995766bde8bb5a9f1862a5f2f95db6f21
2020-04-15 11:00:09 -07:00
Ian Wienand
5068744eff ensure-pip: Add role
This role is intended to ensure that the `pip:` module, and jobs that
may wish to use `pip` from the shell, have the necessary requirements.

It is intended as a partial replacement for the pip-and-virtualenv
element in diskimage-builder, which currently pre-installs pip on our
infra CI images during image build.

We wish to remain broady compatible with this element, but not
replicate some of the more problematic areas of its implementation.

By default, this installs the system packages for pip and setuptools
(the latter being a requirement of the Ansible pip module, which
imports it directly, despite pip itself not requiring it).

In this role, we ensure the libraries for the currently running
ansible_python_interpreter version are installed.

There is provision to provide a flag to install the packages directly
from upstream via get-pip.py, although this is not recommended.

Story: #2007386
Task: #39309

Change-Id: Iac2d518a66caf1b801273225f75a0a748412903c
2020-04-14 10:20:33 +10:00