WIP Install pip and virtualenv only from get-pip

We have two diametrically opposed use cases amongst our user base.

The first are the folks who just need tox/virtualenv to pre-exist,
so that they can do things like run tests. Neither tox nor virtualenv
themselves are fundamental to the thing under test, but are tools
that need to exist.

The second are folks for whom tools like pip or virtualenv are
essential to the software (such as OSA). For these users, it is
important that the version of those match what they expect their
users get from the distro.

Up until now, we've been using the pip-and-virtualenv element from
DIB which installs distro versions of pip and virtualenv, then
overwrites them with pip installed versions of the same, doing what
it can to prevent subsequent distro installs of the packages from
breaking things. This produces a really strange environment for
our friends in teh second camp, but honestly is a weird scenarios
to wrap the head around.

Instead, remove ALL distro packages of pip and virtualenv. Then
install both via get-pip. But - also leave a cleanup script. Since
pip installs are pretty well self-contained, it's easy to remove
them with a simple "rm".  This cleanup script can be run by folks
like OSA in their first pre-playbook in zuul, and then their jobs
can subsequently apt-get or dnf install python3-virtualenv or
whatever they want to do that is appropriate.

Change-Id: Ifd9062e5a87923093e84b3e4fc933dc08375df82

nodepool/elements/nodepool-base/element-deps

@@ -1,3 +1,3 @@
 package-installs
-pip-and-virtualenv
+simple-pip
 zuul-worker

nodepool/elements/simple-pip/README.rst

@@ -0,0 +1,54 @@
+==========
+simple-pip
+==========
+
+This element installs pip and virtualenv in the image.
+
+get-pip
+=======
+
+The simple-pip element removes all distro packages for pip and
+virtualenv and then installs pip from get-pip.py and virtualenv
+from pip.
+
+The system will be left in the following state:
+
+* ``/usr/bin/pip`` : python2 pip
+* ``/usr/bin/pip2`` : python2 pip (same as prior)
+* ``/usr/bin/pip3`` : python3 pip
+* ``/usr/bin/virtualenv`` : python2 virtualenv
+
+(note python3 ``virtualenv`` script is *not* installed, see below)
+
+Explicit use of the tools
+=========================
+
+Due to the essentially unsolvable problem of "who owns the script", it
+is recommended to *not* call ``pip`` or ``virtualenv`` directly.  You
+can directly call them with the ``-m`` argument to the python
+interpreter you wish to install with.
+
+For example, to create a python3 environment do::
+
+  # python3 -m virtualenv myenv
+  # myenv/bin/pip install mytool
+
+To install a python2 tool from pip::
+
+  # python2 -m pip install mytool
+
+In this way, you can always know which interpreter is being used (and
+affected by) the call.
+
+cleanup-pip.sh
+==============
+
+A script is provided, ``/usr/local/bin/cleanup-pip.sh`` which will
+uninstall pip installed pip and virtualenv. This script is suitable
+to be used in Zuul jobs where the job content wants to install its
+own pip or virtualenv infrastructure.
+
+Ordering
+========
+Any element that uses these commands must be designated as
+05-* or higher to ensure that they are first installed.

nodepool/elements/simple-pip/bin/cleanup-pip.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+# Remove pip installed 
+rm -rf /usr/local/bin/{pip,virtualenv,wheel,easy_install}* /usr/local/lib/python*/{dist,site}-packages/{setuptools,pkg_resources,wheel,pip}* 

nodepool/elements/simple-pip/element-deps

@@ -0,0 +1,5 @@
+dib-python
+epel
+install-bin
+package-installs
+source-repositories

nodepool/elements/simple-pip/environment.d/51-pip-and-virutalenv-default.bash

@@ -0,0 +1 @@
+export DIB_PYTHON_VIRTUALENV="/usr/bin/python3 -m virtualenv"

nodepool/elements/simple-pip/install.d/04-install-pip

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
+    set -x
+fi
+set -eu
+set -o pipefail
+
+# These install into /usr/local/bin so override any packages, even
+# if installed later.
+
+python3 /tmp/get-pip.py
+python2 /tmp/get-pip.py
+
+pip3 install -U --force-reinstall virtualenv
+pip install -U --force-reinstall virtualenv

nodepool/elements/simple-pip/package-installs.yaml

@@ -0,0 +1,21 @@
+python:
+  phase: pre-install.d
+  dib_python_version: 2
+python3:
+  phase: pre-install.d
+  dib_python_version: 3
+python-pip:
+  uninstall: True
+python3-pip:
+  uninstall: True
+python-setuptools:
+  uninstall: True
+python3-setuptools:
+  uninstall: True
+python-virtualenv:
+  uninstall: True
+python3-virtualenv:
+  uninstall: True
+# Separate package on debubuntu
+python3-venv:
+  dib_python_version: 3

nodepool/elements/simple-pip/pkg-map

@@ -0,0 +1,47 @@
+{
+  "release": {
+    "rhel": {
+      "8": {
+        "python3-dev": "platform-python-devel"
+      }
+    },
+    "centos": {
+      "7": {
+        "python3": "",
+        "python3-dev": ""
+      }
+    }
+  },
+  "family": {
+    "debian": {
+      "python3-venv": "python3-venv"
+    },
+    "gentoo": {
+      "python-pip": "dev-python/pip",
+      "python3-pip": "dev-python/pip",
+      "python-virtualenv": "dev-python/virtualenv",
+      "python3-virtualenv": "dev-python/virtualenv",
+      "python-dev": "dev-lang/python",
+      "python3-dev": "dev-lang/python"
+    },
+    "suse": {
+      "python-dev": "python-devel",
+      "python3-dev": "python3-devel",
+      "python3": "python3"
+      "python-pip": "python2-pip",
+      "python-setuptools": "python2-setuptools",
+      "python-virtualenv": "python2-virtualenv"
+    },
+    "redhat": {
+      "python-dev": "python2-devel",
+      "python3-dev": "python3-devel",
+      "python-pip": "python2-pip",
+      "python-setuptools": "python2-setuptools",
+      "python-virtualenv": "python2-virtualenv"
+    }
+  },
+  "default": {
+    "python3": "",
+    "python3-venv": ""
+  }
+}

nodepool/elements/simple-pip/post-install.d/89-clean-up-pip-cache

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
+    set -x
+fi
+set -eu
+set -o pipefail
+
+# TODO(johnsom) Fix this when https://github.com/pypa/pip/issues/4685 is
+#               available.
+if [ "${DIB_DISABLE_PIP_CLEANUP:-0}" != "1" ]; then
+    rm -rf ~/.cache/pip
+fi

nodepool/elements/simple-pip/source-repository-pip-and-virtualenv

@@ -0,0 +1 @@
+pip-and-virtualenv file /tmp/get-pip.py https://bootstrap.pypa.io/get-pip.py