WIP Install pip and virtualenv only from get-pip
We have two diametrically opposed use cases amongst our user base. The first are the folks who just need tox/virtualenv to pre-exist, so that they can do things like run tests. Neither tox nor virtualenv themselves are fundamental to the thing under test, but are tools that need to exist. The second are folks for whom tools like pip or virtualenv are essential to the software (such as OSA). For these users, it is important that the version of those match what they expect their users get from the distro. Up until now, we've been using the pip-and-virtualenv element from DIB which installs distro versions of pip and virtualenv, then overwrites them with pip installed versions of the same, doing what it can to prevent subsequent distro installs of the packages from breaking things. This produces a really strange environment for our friends in teh second camp, but honestly is a weird scenarios to wrap the head around. Instead, remove ALL distro packages of pip and virtualenv. Then install both via get-pip. But - also leave a cleanup script. Since pip installs are pretty well self-contained, it's easy to remove them with a simple "rm". This cleanup script can be run by folks like OSA in their first pre-playbook in zuul, and then their jobs can subsequently apt-get or dnf install python3-virtualenv or whatever they want to do that is appropriate. Change-Id: Ifd9062e5a87923093e84b3e4fc933dc08375df82
This commit is contained in:
parent
9ce6de3dfc
commit
4a139969e1
@ -1,3 +1,3 @@
|
|||||||
package-installs
|
package-installs
|
||||||
pip-and-virtualenv
|
simple-pip
|
||||||
zuul-worker
|
zuul-worker
|
||||||
|
54
nodepool/elements/simple-pip/README.rst
Normal file
54
nodepool/elements/simple-pip/README.rst
Normal file
@ -0,0 +1,54 @@
|
|||||||
|
==========
|
||||||
|
simple-pip
|
||||||
|
==========
|
||||||
|
|
||||||
|
This element installs pip and virtualenv in the image.
|
||||||
|
|
||||||
|
get-pip
|
||||||
|
=======
|
||||||
|
|
||||||
|
The simple-pip element removes all distro packages for pip and
|
||||||
|
virtualenv and then installs pip from get-pip.py and virtualenv
|
||||||
|
from pip.
|
||||||
|
|
||||||
|
The system will be left in the following state:
|
||||||
|
|
||||||
|
* ``/usr/bin/pip`` : python2 pip
|
||||||
|
* ``/usr/bin/pip2`` : python2 pip (same as prior)
|
||||||
|
* ``/usr/bin/pip3`` : python3 pip
|
||||||
|
* ``/usr/bin/virtualenv`` : python2 virtualenv
|
||||||
|
|
||||||
|
(note python3 ``virtualenv`` script is *not* installed, see below)
|
||||||
|
|
||||||
|
Explicit use of the tools
|
||||||
|
=========================
|
||||||
|
|
||||||
|
Due to the essentially unsolvable problem of "who owns the script", it
|
||||||
|
is recommended to *not* call ``pip`` or ``virtualenv`` directly. You
|
||||||
|
can directly call them with the ``-m`` argument to the python
|
||||||
|
interpreter you wish to install with.
|
||||||
|
|
||||||
|
For example, to create a python3 environment do::
|
||||||
|
|
||||||
|
# python3 -m virtualenv myenv
|
||||||
|
# myenv/bin/pip install mytool
|
||||||
|
|
||||||
|
To install a python2 tool from pip::
|
||||||
|
|
||||||
|
# python2 -m pip install mytool
|
||||||
|
|
||||||
|
In this way, you can always know which interpreter is being used (and
|
||||||
|
affected by) the call.
|
||||||
|
|
||||||
|
cleanup-pip.sh
|
||||||
|
==============
|
||||||
|
|
||||||
|
A script is provided, ``/usr/local/bin/cleanup-pip.sh`` which will
|
||||||
|
uninstall pip installed pip and virtualenv. This script is suitable
|
||||||
|
to be used in Zuul jobs where the job content wants to install its
|
||||||
|
own pip or virtualenv infrastructure.
|
||||||
|
|
||||||
|
Ordering
|
||||||
|
========
|
||||||
|
Any element that uses these commands must be designated as
|
||||||
|
05-* or higher to ensure that they are first installed.
|
4
nodepool/elements/simple-pip/bin/cleanup-pip.sh
Normal file
4
nodepool/elements/simple-pip/bin/cleanup-pip.sh
Normal file
@ -0,0 +1,4 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Remove pip installed
|
||||||
|
rm -rf /usr/local/bin/{pip,virtualenv,wheel,easy_install}* /usr/local/lib/python*/{dist,site}-packages/{setuptools,pkg_resources,wheel,pip}*
|
5
nodepool/elements/simple-pip/element-deps
Normal file
5
nodepool/elements/simple-pip/element-deps
Normal file
@ -0,0 +1,5 @@
|
|||||||
|
dib-python
|
||||||
|
epel
|
||||||
|
install-bin
|
||||||
|
package-installs
|
||||||
|
source-repositories
|
@ -0,0 +1 @@
|
|||||||
|
export DIB_PYTHON_VIRTUALENV="/usr/bin/python3 -m virtualenv"
|
80
nodepool/elements/simple-pip/install.d/04-install-pip
Executable file
80
nodepool/elements/simple-pip/install.d/04-install-pip
Executable file
@ -0,0 +1,80 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
|
||||||
|
set -x
|
||||||
|
fi
|
||||||
|
set -eu
|
||||||
|
set -o pipefail
|
||||||
|
|
||||||
|
if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then
|
||||||
|
|
||||||
|
if [[ $DISTRO_NAME =~ (centos7) ]]; then
|
||||||
|
|
||||||
|
# note python3 in epel
|
||||||
|
_extra_repo="--enablerepo=epel"
|
||||||
|
packages="python3 python3-devel"
|
||||||
|
|
||||||
|
${YUM:-yum} ${_extra_repo} install -y $packages
|
||||||
|
|
||||||
|
fi
|
||||||
|
|
||||||
|
# install the latest python2 pip; this overwrites packaged pip
|
||||||
|
python /tmp/get-pip.py
|
||||||
|
|
||||||
|
# Install latest setuptools; there is a slight chicken-egg issue in
|
||||||
|
# that pip requires setuptools for some operations like building a
|
||||||
|
# wheel. But this simple install should be fine.
|
||||||
|
pip install setuptools
|
||||||
|
|
||||||
|
# Repeat above for python3
|
||||||
|
|
||||||
|
# python2 on fedora always installs into /usr/bin. Move pip2
|
||||||
|
# binary out, as we want "pip" in the final image to be
|
||||||
|
# python2 for historical reasons.
|
||||||
|
mv /usr/bin/pip /usr/bin/pip2
|
||||||
|
|
||||||
|
# You would think that installing python3 bits first, then
|
||||||
|
# python2 would work -- alas get-pip.py doesn't seem to leave
|
||||||
|
# python3 alone:
|
||||||
|
# https://github.com/pypa/pip/issues/4435
|
||||||
|
python3 /tmp/get-pip.py
|
||||||
|
pip3 install setuptools
|
||||||
|
|
||||||
|
# on < 27, this installed pip3 to /usr/bin/pip. On >=27 it's
|
||||||
|
# /usr/local/bin/pip. reclaim /usr/bin/pip back to pip2 and
|
||||||
|
# remove the /usr/local/bin/pip (i.e. python3 version) if it
|
||||||
|
# exists, so that "pip" calls pip2 always. if we want pip3 we
|
||||||
|
# call it explicitly.
|
||||||
|
ln -sf /usr/bin/pip2 /usr/bin/pip
|
||||||
|
rm -f /usr/local/bin/pip
|
||||||
|
|
||||||
|
# now install latest virtualenv. it vendors stuff it needs so
|
||||||
|
# doesn't have issues with other system packages.
|
||||||
|
pip install virtualenv
|
||||||
|
mv /usr/bin/virtualenv /usr/bin/virtualenv2
|
||||||
|
pip3 install virtualenv
|
||||||
|
|
||||||
|
# Reclaim virtualenv to virtualenv2; similar to above, on fedora
|
||||||
|
# >27 the pip3 version has gone into /usr/local/bin; remove it so
|
||||||
|
# only /usr/bin/virtualenv exists
|
||||||
|
ln -sf /usr/bin/virtualenv2 /usr/bin/virtualenv
|
||||||
|
rm -f /usr/local/bin/virtualenv
|
||||||
|
|
||||||
|
# at this point, we should have the latest
|
||||||
|
# pip/setuptools/virtualenv packages for python2 & 3, and
|
||||||
|
# "/usr/bin/pip" and "/usr/bin/virtualenv" should be python2
|
||||||
|
# versions.
|
||||||
|
|
||||||
|
else
|
||||||
|
|
||||||
|
# Debuntu
|
||||||
|
|
||||||
|
# These install into /usr/local/bin so override any packages, even
|
||||||
|
# if installed later.
|
||||||
|
|
||||||
|
python3 /tmp/get-pip.py
|
||||||
|
python2 /tmp/get-pip.py
|
||||||
|
|
||||||
|
pip3 install setuptools virtualenv
|
||||||
|
pip install setuptools virtualenv
|
||||||
|
fi
|
21
nodepool/elements/simple-pip/package-installs.yaml
Normal file
21
nodepool/elements/simple-pip/package-installs.yaml
Normal file
@ -0,0 +1,21 @@
|
|||||||
|
python:
|
||||||
|
phase: pre-install.d
|
||||||
|
dib_python_version: 2
|
||||||
|
python3:
|
||||||
|
phase: pre-install.d
|
||||||
|
dib_python_version: 3
|
||||||
|
python-pip:
|
||||||
|
uninstall: True
|
||||||
|
python3-pip:
|
||||||
|
uninstall: True
|
||||||
|
python-setuptools:
|
||||||
|
uninstall: True
|
||||||
|
python3-setuptools:
|
||||||
|
uninstall: True
|
||||||
|
python-virtualenv:
|
||||||
|
uninstall: True
|
||||||
|
python3-virtualenv:
|
||||||
|
uninstall: True
|
||||||
|
# Separate package on debubuntu
|
||||||
|
python3-venv:
|
||||||
|
dib_python_version: 3
|
47
nodepool/elements/simple-pip/pkg-map
Normal file
47
nodepool/elements/simple-pip/pkg-map
Normal file
@ -0,0 +1,47 @@
|
|||||||
|
{
|
||||||
|
"release": {
|
||||||
|
"rhel": {
|
||||||
|
"8": {
|
||||||
|
"python3-dev": "platform-python-devel"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"centos": {
|
||||||
|
"7": {
|
||||||
|
"python3": "",
|
||||||
|
"python3-dev": ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"family": {
|
||||||
|
"debian": {
|
||||||
|
"python3-venv": "python3-venv"
|
||||||
|
},
|
||||||
|
"gentoo": {
|
||||||
|
"python-pip": "dev-python/pip",
|
||||||
|
"python3-pip": "dev-python/pip",
|
||||||
|
"python-virtualenv": "dev-python/virtualenv",
|
||||||
|
"python3-virtualenv": "dev-python/virtualenv",
|
||||||
|
"python-dev": "dev-lang/python",
|
||||||
|
"python3-dev": "dev-lang/python"
|
||||||
|
},
|
||||||
|
"suse": {
|
||||||
|
"python-dev": "python-devel",
|
||||||
|
"python3-dev": "python3-devel",
|
||||||
|
"python3": "python3"
|
||||||
|
"python-pip": "python2-pip",
|
||||||
|
"python-setuptools": "python2-setuptools",
|
||||||
|
"python-virtualenv": "python2-virtualenv"
|
||||||
|
},
|
||||||
|
"redhat": {
|
||||||
|
"python-dev": "python2-devel",
|
||||||
|
"python3-dev": "python3-devel",
|
||||||
|
"python-pip": "python2-pip",
|
||||||
|
"python-setuptools": "python2-setuptools",
|
||||||
|
"python-virtualenv": "python2-virtualenv"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"default": {
|
||||||
|
"python3": "",
|
||||||
|
"python3-venv": ""
|
||||||
|
}
|
||||||
|
}
|
13
nodepool/elements/simple-pip/post-install.d/89-clean-up-pip-cache
Executable file
13
nodepool/elements/simple-pip/post-install.d/89-clean-up-pip-cache
Executable file
@ -0,0 +1,13 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
|
||||||
|
set -x
|
||||||
|
fi
|
||||||
|
set -eu
|
||||||
|
set -o pipefail
|
||||||
|
|
||||||
|
# TODO(johnsom) Fix this when https://github.com/pypa/pip/issues/4685 is
|
||||||
|
# available.
|
||||||
|
if [ "${DIB_DISABLE_PIP_CLEANUP:-0}" != "1" ]; then
|
||||||
|
rm -rf ~/.cache/pip
|
||||||
|
fi
|
@ -0,0 +1 @@
|
|||||||
|
pip-and-virtualenv file /tmp/get-pip.py https://bootstrap.pypa.io/get-pip.py
|
Loading…
Reference in New Issue
Block a user