Support virtualenv installation in baremetal role

Installing python packages directly to the system site-packages can
cause various problems, in particular when pip overwrites a system
package. Python virtualenvs are one solution to this issue, as they
allow python packages to be installed in an isolated environment.

This change adds support to the baremetal role for installing python
dependencies in a virtualenv. Typically we will need to enable use of
system site-packages from within this virtualenv, to support the use of
modules such as yum, apt, and selinux, which are not available on PyPI.

The path to the virtualenv is configured via the 'virtualenv' variable,
and access to site-packages is controlled via
'virtualenv_site_packages'.

When executing other kolla-ansible commands, the variable
'ansible_python_interpreter' should be set to the python interpreter
installed in 'virtualenv'. Note that this variable cannot be templated.

Change-Id: I0741923065246f9c5b168059fcd66504f2753c41
Related-Bug: #1731026

ansible/roles/baremetal/defaults/main.yml

@@ -39,3 +39,12 @@ redhat_pkg_removals:
  - libvirt
  - libvirt-daemon
  - iscsi-initiator-utils
+
+# Path to a virtualenv in which to install python packages. If None, a
+# virtualenv will not be used.
+virtualenv:
+
+# Whether the virtualenv will inherit packages from the global site-packages
+# directory. This is typically required for modules such as yum and apt which
+# are not available on PyPI.
+virtualenv_site_packages: True

ansible/roles/baremetal/tasks/install.yml

@@ -56,17 +56,47 @@
   with_items: "{{ redhat_pkg_install }}"
   when: ansible_os_family == 'RedHat'
 
+- name: Install virtualenv packages
+  package:
+    name: python-virtualenv
+    state: present
+  become: True
+  when: virtualenv is not none
+
 - name: Install pip
   easy_install:
     name: pip
+    virtualenv: "{{ virtualenv is none | ternary(omit, virtualenv) }}"
+    virtualenv_site_packages: "{{ virtualenv is none | ternary(omit, virtualenv_site_packages) }}"
   become: True
 
+- name: Install latest pip in the virtualenv
+  pip:
+    name: pip
+    state: latest
+    virtualenv: "{{ virtualenv }}"
+    virtualenv_site_packages: "{{ virtualenv_site_packages }}"
+  become: True
+  when: virtualenv is not none
+
 - name: Install docker SDK for python
   pip:
     name: docker
     state: latest
+    virtualenv: "{{ virtualenv is none | ternary(omit, virtualenv) }}"
+    virtualenv_site_packages: "{{ virtualenv is none | ternary(omit, virtualenv_site_packages) }}"
   become: True
 
+- name: Ensure virtualenv has correct ownership
+  file:
+    path: "{{ virtualenv }}"
+    recurse: True
+    state: directory
+    owner: kolla
+    group: kolla
+  become: True
+  when: virtualenv is not none
+
 - name: Remove packages
   package:
     name: "{{ item }}"

doc/source/user/index.rst

@@ -5,6 +5,7 @@ User Guides
    :maxdepth: 1
 
    quickstart
+   virtual-environments
    multinode
    multi-regions
    operating-kolla

doc/source/user/virtual-environments.rst

@@ -0,0 +1,76 @@
+.. _virtual-environments:
+
+====================
+Virtual Environments
+====================
+
+Python `virtual environments <https://virtualenv.pypa.io/en/stable/>`_ provide
+a mechanism for isolating python packages from the system site packages, and
+other virtual environments. Kolla-ansible largely avoids this problem by
+deploying services in Docker containers, however some python dependencies must
+be installed both on the Ansible control host and the target hosts.
+
+Ansible Control Host
+====================
+
+The kolla-ansible python package and its dependencies may be installed in a
+python virtual environment on the Ansible control host. For example:
+
+.. code-block:: console
+
+   virtualenv /path/to/venv
+   source /path/to/venv/bin/activate
+   pip install -U pip
+   pip install kolla-ansible
+   deactivate
+
+It may be advantageous to also install Ansible in the virtual environment.
+
+.. code-block:: console
+
+   source /path/to/venv/bin/activate
+   (venv) pip install ansible
+   (venv) deactivate
+
+To use the virtual environment, it should first be activated:
+
+.. code-block:: console
+
+   source /path/to/venv/bin/activate
+   (venv) kolla-ansible --help
+
+The virtual environment can be deactivated when necessary:
+
+.. code-block:: console
+
+   (venv) deactivate
+
+Note that the use of a virtual environment on the Ansible control host does not
+imply that a virtual environment will be used for execution of Ansible modules
+on the target hosts.
+
+Target Hosts
+============
+
+Ansible supports remote execution of modules in a python virtual environment
+via the ``ansible_python_interpreter`` variable. This may be configured to be
+the path to a python interpreter installed in a virtual environment.  For
+example:
+
+.. code-block:: yaml
+
+   ansible_python_interpreter: /path/to/venv/bin/python
+
+Note that ``ansible_python_interpreter`` cannot be templated.
+
+Kolla-ansible provides support for creating a python virtual environment on the
+target hosts as part of the ``bootstrap-servers`` command. The path to the
+virtualenv is configured via the ``virtualenv`` variable, and access to
+site-packages is controlled via ``virtualenv_site_packages``. Typically we
+will need to enable use of system site-packages from within this virtualenv, to
+support the use of modules such as yum, apt, and selinux, which are not
+available on PyPI.
+
+When executing kolla-ansible commands other than ``bootstrap-servers``, the
+variable ``ansible_python_interpreter`` should be set to the python interpreter
+installed in ``virtualenv``.

releasenotes/notes/bootstrap-servers-virtualenv-723a0e80942604bd.yaml

@@ -0,0 +1,23 @@
+---
+features:
+  - |
+    Adds support for installing python dependencies into a virtualenv on remote
+    hosts.
+
+    Installing python packages directly to the system site-packages can cause
+    various problems, in particular when pip overwrites a system package.
+    Python virtualenvs are one solution to this issue, as they allow python
+    packages to be installed in an isolated environment.  Typically we will
+    need to enable use of system site-packages from within this virtualenv, to
+    support the use of modules such as yum, apt, and selinux, which are not
+    available on PyPI.
+
+    The path to the virtualenv is configured via the ``virtualenv`` variable,
+    and access to site-packages is controlled via ``virtualenv_site_packages``.
+    The default value for ``virtualenv`` is None, in which case the old
+    behaviour of installing packages directly to the system site-packages is
+    maintained.
+
+    When executing other kolla-ansible commands, the variable
+    ``ansible_python_interpreter`` should be set to the python interpreter
+    installed in ``virtualenv``. Note that this variable cannot be templated.