Pre-install pip/virtualenv packages

If we're installing pip/virtualenv from source, we need to make sure
we pre-install the packaged versions before the upstream
versions. Otherwise, CI jobs later on that depend on packaged versions
of pip/virtualenv can bring them in and overwrite the upstream
versions we have installed, which leads to a heck of a mess and
usually very confusing failures.

I have also moved in a small hack from system-config:install_puppet.sh
that we found was necessary when using pip versions from upstream.

Note this is not as much of an issue on Debian/Ubuntu, as they keep
their pip packages in a separate place to the system packages, so you
don't have these overwite conflicts as much.

Change-Id: Ib40708c07b939b84661c44df88a5a308fd0c7216

elements/centos-minimal/environment.d/10-centos-distro-name.bash

@@ -4,3 +4,6 @@ export DIB_RELEASE=${DIB_RELEASE:-7}
 # by default, enable DHCP configuration of eth0 & eth1 in network
 # scripts.  See yum-minimal for full details
 export DIB_YUM_MINIMAL_CREATE_INTERFACES=${DIB_YUM_MINIMAL_CREATE_INTERFACES:-1}
+
+# Useful for elements that work with fedora (dnf) & centos
+export YUM=${YUM:-yum}

elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/01-install-pip

@@ -6,5 +6,50 @@ fi
 set -eu
 set -o pipefail
 
-python /tmp/get-pip.py
-pip install virtualenv
+if [[ $DISTRO_NAME =~ (centos|fedora) ]]; then
+    # GENERAL WARNING : mixing packaged python libraries with
+    # pip-installed versions always creates issues.  Upstream
+    # openstack-infra uses this a lot (especially devstack) but be
+    # warned: here be dragons :)
+
+    # Firstly we want to install the system packages.  Otherwise later
+    # on somebody does a "yum install python-virtualenv" and goes and
+    # overwrites the pip installed version with the packaged version,
+    # leading to all sorts of weird version issues.
+    ${YUM} install -y python-virtualenv python-pip python-setuptools
+
+    # install pip; this overwrites packaged pip
+    python /tmp/get-pip.py
+
+    # pip and setuptools are closely related; we want to ensure the
+    # latest for sanity.  Because distro packages don't include enough
+    # info in the egg for pip to be certain it has fully uninstalled
+    # the old package, for safety we clear it out by hand (this seems
+    # to have been a problem with very old to new updates,
+    # e.g. centos6 to current-era, but less so for smaller jumps).
+    # There is a bit of chicken-and-egg problem with pip in that it
+    # requires setuptools for some operations, such as wheel creation.
+    # But just installing setuptools shouldn't require setuptools
+    # itself, so we are safe for this small section.
+    rm -rf /usr/lib/python2.7/site-packages/setuptools*
+    pip install -U setuptools
+
+    # now install latest virtualenv.  it vendors stuff it needs so
+    # doesn't have issues with other system packages.
+    pip install -U virtualenv
+
+    # Add this to exclude so that we don't install a later package
+    # over it if it updates.  Note that fedora-minimal, bootstrapped
+    # via yum, can have an old yum.conf around, so look for dnf first.
+    if [[ -f /etc/dnf/dnf.conf ]]; then
+        conf=/etc/dnf/dnf.conf
+    elif [[ -f /etc/yum.conf ]]; then
+        conf=/etc/yum.conf
+    else
+        die "No conf to modify?"
+    fi
+    echo "exclude=python-virtualenv,python-pip,python-setuptools" >> ${conf}
+else
+    python /tmp/get-pip.py
+    pip install virtualenv
+fi