Use in-tree env.d files, provide override support

In past versions, upgrading OpenStack-Ansible lead to contention between
the code base and deployer customizations when env.d files were changed.
Deployers were encouraged to make their customizations, while the
project needed to sometimes adjust architecture. Detecting these
conflicts in an automated way was difficult, since the files were simple
dictionaries and lists, leaving no metadata to describe intent for the
changes.

This change modifies the dynamic inventory system to first use the
in-tree env.d directory as the base environment, then reads in files
from the /etc/openstack_deploy/env.d directory and updates existing keys
with the new values. In this way, the OSA project can modify the
environment and deployers can customize the environment without directly
manipulating the same files.

As part of this change, the env.d directory was moved in to the
playbooks/inventory directory, in order to reduce the path manipulation
done inside of the dynamic_inventory.py script. The example files were
left in the etc/openstack_deploy directory for reference.

Note that this change supports deleting elements by specifying a
empty value, such as an empty list or an empty dictionary.

When overriding, only the path to the values that changed is necessary.
For example, changing the 'is_metal' property for cinder only needs the
following in /etc/openstack_deploy/env.d/cinder.yml:

    container_skel:
      cinder_volumes_container:
        properties:
          is_metal: false

This is instead of the entirity of the container_skel dict or even the
other top-level dicts.

For AIO/gate scenarioes, the env.d copy logic has been removed, as it is
now redundant.

Change-Id: Ic637fa385fd3fec7365fb9bc5e0ff54a7f4c8bee

doc/source/developer-docs/extending.rst

@@ -77,6 +77,9 @@ env.d
 The ``/etc/openstack_deploy/env.d`` directory sources all YAML files into the
 deployed environment, allowing a deployer to define additional group mappings.
 
+This directory is used to extend the environment skeleton, or modify the
+defaults defined in the ``playbooks/inventory/env.d`` directory.
+
 See also `Understanding Container Groups`_ in Appendix H.
 
 .. _Understanding Container Groups: ../install-guide/app-custom-layouts.html#understanding-container-groups

doc/source/developer-docs/inventory.rst

@@ -38,6 +38,9 @@ Inputs
 The ``dynamic_inventory.py`` script takes a single argument, ``--config``. If
 not specified, the default is ``/etc/openstack_deploy/``.
 
+In addition to this argument, the base environment skeleton is provided in the
+``playbooks/inventory/env.d`` directory of the OpenStack-Ansible codebase.
+
 .. note:: In all versions prior to Mitaka, this argument was ``--file``.
 
 The following file must be present in the configuration directory:
@@ -45,10 +48,10 @@ The following file must be present in the configuration directory:
     * ``openstack_user_config.yml``
 
 Additionally, the configuration or environment could be spread between two
-additional directories:
+additional sub-directories:
 
     * ``conf.d``
-    * ``env.d``
+    * ``env.d`` (for environment customization)
 
 The dynamic inventory script does the following:
 
@@ -71,13 +74,13 @@ As an example, consider the following excerpt from
 
 The ``identity_hosts`` dictionary defines an Ansible inventory group named
 ``identity_hosts`` containing the three infra hosts. The configuration file
-``etc/openstack_deploy/env.d/keystone.yml`` defines additional Ansible
+``playbooks/inventory/env.d/keystone.yml`` defines additional Ansible
 inventory groups for the containers that are deployed onto the three hosts
 named with the prefix *infra*.
 
 Note that any services marked with ``is_metal: true`` will run on the allocated
 physical host and not in a container. For an example of ``is_metal: true``
-being used refer to ``etc/openstack_deploy/env.d/cinder.yml`` in the
+being used refer to ``playbooks/inventory/env.d/cinder.yml`` in the
 ``container_skel`` section.
 
 Outputs

doc/source/install-guide/app-custom-layouts.rst

@@ -8,10 +8,10 @@ Understanding the default layout
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 The default layout of containers and services in OpenStack-Ansible is driven
 by the ``/etc/openstack_deploy/openstack_user_config.yml`` file and the
-contents of both the ``/etc/openstack_deploy/conf.d/`` and
-``/etc/openstack_deploy/env.d/`` directories. Use these sources to define the
-group mappings used by the playbooks to target hosts and containers for roles
-used in the deploy.
+contents of the ``/etc/openstack_deploy/conf.d/``,
+``playbooks/inventory/env.d/`` and ``/etc/openstack_deploy/env.d/``
+directories. Use these sources to define the group mappings used by the
+playbooks to target hosts and containers for roles used in the deploy.
 
 Conceptually, these can be thought of as mapping from two directions. You
 define host groups, which gather the target hosts into inventory groups,
@@ -54,7 +54,7 @@ variables to any component containers on the specific host.
 Understanding container groups
 ------------------------------
 Additional group mappings can be found within files in the
-``/etc/openstack_deploy/env.d/`` directory. These groupings are treated as
+``playbooks/inventory/env.d/`` directory. These groupings are treated as
 virtual mappings from the host groups (described above) onto the container
 groups which define where each service deploys. By reviewing files within the
 ``env.d/`` directory, you can begin to see the nesting of groups represented
@@ -99,12 +99,27 @@ Customizing existing components
 Numerous customization scenarios are possible, but three popular ones are
 presented here as starting points and also as common recipes.
 
+Modifying the default environment
+---------------------------------
+
+In order to avoid conflicts between deployer and project changes, the base
+configuration for the OpenStack-Ansible components resides in the
+``playbooks/inventory/env.d/`` directory.
+
+The ``/etc/openstack_deploy/env.d`` directory is used to override and extend
+the environment to the deployer's needs. To modify an existing configuration,
+copy the relevant service file to the ``/etc/openstack_deploy/env.d``
+directory. Then, modify the values under the relevant keys. Only the keys
+and the modified value are required to be present; other information can
+safely be omitted.
+
+
 Deploying directly on hosts
 ---------------------------
 
-To deploy a component directly on the host instead of within a container, set
-the ``is_metal`` property to ``true`` for the container group under the
-``container_skel`` in the appropriate file.
+To deploy a component directly on the host instead of within a container, copy
+the relevant file to ``/etc/openstack_deploy/env.d/`` and set the ``is_metal``
+property to ``true`` for the container group under the ``container_skel``.
 
 The use of ``container_vars`` and mapping from container groups to host groups
 is the same for a service deployed directly onto the host.
@@ -112,7 +127,7 @@ is the same for a service deployed directly onto the host.
 .. note::
 
    The ``cinder_volume`` component is also deployed directly on the host by
-   default. See the ``env.d/cinder.yml`` file for this example.
+   default. See the ``playbooks/inventory/env.d/cinder.yml`` file for this example.
 
 Omit a service or component from the deployment
 -----------------------------------------------
@@ -120,8 +135,14 @@ Omit a service or component from the deployment
 To omit a component from a deployment, several options exist.
 
 - You could remove the ``physical_skel`` link between the container group and
-  the host group. The simplest way to do this is to simply delete the related
-  file located in the ``env.d/`` directory.
+  the host group. The simplest way to do this is to simply copy the relevant file
+  to the ``/etc/openstack_deploy/env.d/`` directory, and set the following
+  information:
+
+.. code-block:: yaml
+
+  physical_skel: {}
+
 - You could choose to not run the playbook which installs the component.
   Unless you specify the component to run directly on a host using is_metal, a
   container creates for this component.

playbooks/inventory/dynamic_inventory.py

@@ -28,7 +28,6 @@ import tarfile
 import uuid
 import yaml
 
-
 USED_IPS = set()
 INVENTORY_SKEL = {
     '_meta': {
@@ -892,12 +891,15 @@ def append_if(array, item):
 def _merge_dict(base_items, new_items):
     """Recursively merge new_items into some base_items.
 
+    If an empty dictionary is provided as a new value, it will
+    completely replace the existing dictionary.
+
     :param base_items: ``dict``
     :param new_items: ``dict``
     :return dictionary:
     """
     for key, value in new_items.iteritems():
-        if isinstance(value, dict):
+        if isinstance(value, dict) and value:
             base_merge = _merge_dict(base_items.get(key, {}), value)
             base_items[key] = base_merge
         else:
@@ -1010,14 +1012,13 @@ def _check_config_settings(cidr_networks, config, container_skel):
     _check_multiple_ips_to_host(config)
 
 
-def load_environment(config_path):
+def load_environment(config_path, environment):
     """Create an environment dictionary from config files
 
     :param config_path: ``str``path where the environment files are kept
+    :param environment: ``dict`` dictionary to populate with environment data
     """
 
-    environment = dict()
-
     # Load all YAML files found in the env.d directory
     env_plugins = os.path.join(config_path, 'env.d')
 
@@ -1095,7 +1096,8 @@ def main(all_args):
 
     user_defined_config = load_user_configuration(config_path)
 
-    environment = load_environment(config_path)
+    base_env = load_environment(os.path.dirname(__file__), {})
+    environment = load_environment(config_path, base_env)
 
     # Load existing inventory file if found
     dynamic_inventory_file = os.path.join(

releasenotes/notes/intree-and-override-envd-371cf9a809b51fe0.yaml

@@ -0,0 +1,14 @@
+---
+features:
+  - The env.d directory included with OpenStack-Ansible is now used as the
+    first source for the environment skeleton, and
+    ``/etc/openstack_deploy/env.d`` will be used only to override values.
+    Deployers without customizations will no longer need to copy the env.d
+    directory to /etc/openstack_deploy.
+
+    As a result, the env.d copy operation has been removed from the node
+    bootstrap role.
+upgrade:
+  - Upgrades will not replace entries in the /etc/openstack_deploy/env.d
+    directory, though new versions of OpenStack-Ansible will now use the
+    shipped env.d as a base, which may alter existing deployments.

tests/inventory/env.d

@@ -1 +0,0 @@
-../../etc/openstack_deploy/env.d

tests/roles/bootstrap-host/tasks/prepare_aio_config.yml

@@ -21,64 +21,9 @@
   with_items:
     - "/etc/openstack_deploy/"
     - "/etc/openstack_deploy/conf.d"
-    - "/etc/openstack_deploy/env.d"
   tags:
     - create-directories
 
-- name: Deploy environment (env.d) configuration
-  config_template:
-    src: "../etc/openstack_deploy/env.d/{{ item.name }}"
-    dest: "/etc/openstack_deploy/env.d/{{ item.name }}"
-    config_overrides: "{{ item.override }}"
-    config_type: "yaml"
-  with_items:
-    - name: aodh.yml
-      override: "{{ aodh_env_overrides | default({}) }}"
-    - name: ceilometer.yml
-      override: "{{ ceilometer_env_overrides | default({}) }}"
-    - name: cinder.yml
-      override: "{{ cinder_env_overrides | default({}) }}"
-    - name: galera.yml
-      override: "{{ galera_env_overrides | default({}) }}"
-    - name: glance.yml
-      override: "{{ glance_env_overrides | default({}) }}"
-    - name: haproxy.yml
-      override: "{{ haproxy_env_overrides | default({}) }}"
-    - name: heat.yml
-      override: "{{ heat_env_overrides | default({}) }}"
-    - name: horizon.yml
-      override: "{{ horizon_env_overrides | default({}) }}"
-    - name: infra.yml
-      override: "{{ infra_env_overrides | default({}) }}"
-    - name: ironic.yml
-      override: "{{ ironic_env_overrides | default({}) }}"
-    - name: keystone.yml
-      override: "{{ keystone_env_overrides | default({}) }}"
-    - name: memcache.yml
-      override: "{{ memcache_env_overrides | default({}) }}"
-    - name: neutron.yml
-      override: "{{ neutron_env_overrides | default({}) }}"
-    - name: nova.yml
-      override: "{{ nova_env_overrides | default({}) }}"
-    - name: os-infra.yml
-      override: "{{ os_infra_env_overrides | default({}) }}"
-    - name: pkg_repo.yml
-      override: "{{ pkg_repo_env_overrides | default({}) }}"
-    - name: rabbitmq.yml
-      override: "{{ rabbitmq_env_overrides | default({}) }}"
-    - name: rsyslog.yml
-      override: "{{ rsyslog_env_overrides | default({}) }}"
-    - name: shared-infra.yml
-      override: "{{ shared_infra_env_overrides | default({}) }}"
-    - name: swift-remote.yml
-      override: "{{ swift_remote_env_overrides | default({}) }}"
-    - name: swift.yml
-      override: "{{ swift_env_overrides | default({}) }}"
-    - name: utility.yml
-      override: "{{ utility_env_overrides | default({}) }}"
-  tags:
-    - deploy-envd
-
 - name: Deploy user conf.d configuration
   config_template:
     src: "../etc/openstack_deploy/conf.d/{{ item.name }}"

tests/test_inventory.py

@@ -20,6 +20,7 @@ sys.path.append(path.join(os.getcwd(), INV_DIR))
 import dynamic_inventory as di
 
 TARGET_DIR = path.join(os.getcwd(), 'tests', 'inventory')
+BASE_ENV_DIR = INV_DIR
 USER_CONFIG_FILE = path.join(TARGET_DIR, "openstack_user_config.yml")
 
 # These files will be placed in TARGET_DIR by INV_SCRIPT.
@@ -292,7 +293,7 @@ class TestUserConfiguration(unittest.TestCase):
 class TestEnvironments(unittest.TestCase):
     def setUp(self):
         self.longMessage = True
-        self.loaded_environment = di.load_environment(TARGET_DIR)
+        self.loaded_environment = di.load_environment(BASE_ENV_DIR, {})
 
     def test_loading_environment(self):
         """Test that the environment can be loaded"""
@@ -747,7 +748,7 @@ class TestMultipleRuns(unittest.TestCase):
 
 class TestEnsureInventoryUptoDate(unittest.TestCase):
     def setUp(self):
-        self.env = di.load_environment(TARGET_DIR)
+        self.env = di.load_environment(BASE_ENV_DIR, {})
         # Copy because we manipulate the structure in each test;
         # not copying would modify the global var in the target code
         self.inv = copy.deepcopy(di.INVENTORY_SKEL)
@@ -802,5 +803,88 @@ class TestEnsureInventoryUptoDate(unittest.TestCase):
         self.inv = None
 
 
+class TestOverridingEnvVars(unittest.TestCase):
+    def setUp(self):
+        self.base_env = di.load_environment(BASE_ENV_DIR, {})
+
+        # Use the cinder configuration as our sample for override testing
+        with open(path.join(BASE_ENV_DIR, 'env.d', 'cinder.yml'), 'r') as f:
+            self.cinder_config = yaml.safe_load(f.read())
+
+        self.override_path = path.join(TARGET_DIR, 'env.d')
+        os.mkdir(self.override_path)
+
+    def write_override_env(self):
+        with open(path.join(self.override_path, 'cinder.yml'), 'w') as f:
+            f.write(yaml.safe_dump(self.cinder_config))
+
+    def test_cinder_metal_override(self):
+        vol = self.cinder_config['container_skel']['cinder_volumes_container']
+        vol['properties']['is_metal'] = False
+
+        self.write_override_env()
+
+        di.load_environment(TARGET_DIR, self.base_env)
+
+        test_vol = self.base_env['container_skel']['cinder_volumes_container']
+        self.assertFalse(test_vol['properties']['is_metal'])
+
+    def test_deleting_elements(self):
+        # Leave only the 'properties' dictionary attached to simulate writing
+        # a partial override file
+
+        vol = self.cinder_config['container_skel']['cinder_volumes_container']
+        for key in vol.keys():
+            if not key == 'properties':
+                del vol[key]
+
+        self.write_override_env()
+
+        di.load_environment(TARGET_DIR, self.base_env)
+
+        test_vol = self.base_env['container_skel']['cinder_volumes_container']
+
+        self.assertIn('belongs_to', test_vol)
+
+    def test_adding_new_keys(self):
+        vol = self.cinder_config['container_skel']['cinder_volumes_container']
+        vol['a_new_key'] = 'Added'
+
+        self.write_override_env()
+
+        di.load_environment(TARGET_DIR, self.base_env)
+
+        test_vol = self.base_env['container_skel']['cinder_volumes_container']
+
+        self.assertIn('a_new_key', test_vol)
+        self.assertEqual(test_vol['a_new_key'], 'Added')
+
+    def test_emptying_dictionaries(self):
+        self.cinder_config['container_skel']['cinder_volumes_container'] = {}
+
+        self.write_override_env()
+
+        di.load_environment(TARGET_DIR, self.base_env)
+
+        test_vol = self.base_env['container_skel']['cinder_volumes_container']
+
+        self.assertNotIn('belongs_to', test_vol)
+
+    def test_emptying_lists(self):
+        vol = self.cinder_config['container_skel']['cinder_volumes_container']
+        vol['belongs_to'] = []
+
+        self.write_override_env()
+
+        di.load_environment(TARGET_DIR, self.base_env)
+
+        test_vol = self.base_env['container_skel']['cinder_volumes_container']
+
+        self.assertEqual(test_vol['belongs_to'], [])
+
+    def tearDown(self):
+        os.remove(path.join(self.override_path, 'cinder.yml'))
+        os.rmdir(self.override_path)
+
 if __name__ == '__main__':
     unittest.main()