Split inventory into multiple dirs and move hostvars

Make inventory/service for service-specific things, including the
groups.yaml group definitions, and inventory/base for hostvars
related to the base system, including the list of hosts.

Move the exisitng host_vars into inventory/service, since most of
them are likely service-specific. Move group_vars/all.yaml into
base/group_vars as almost all of it is related to base things,
with the execption of the gerrit public key.

A followup patch will move host-specific values into equivilent
files in inventory/base.

This should let us override hostvars in gate jobs. It should also
allow us to do better file matchers - and to be able to organize
our playbooks move if we want to.

Depends-On: https://review.opendev.org/731583
Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf

doc/source/dns.rst

@@ -17,7 +17,7 @@ At a Glance
   * ns1.opendev.org
   * ns2.opendev.org
 :Ansible:
-  * :git_file:`playbooks/group_vars/dns.yaml`
+  * :git_file:`inventory/service/group_vars/dns.yaml`
 :Projects:
   * https://www.nlnetlabs.nl/projects/nsd/
   * https://www.isc.org/downloads/bind/doc/
@@ -27,7 +27,7 @@ Adding a Zone
 
 To add a new zone, identify an existing git repository or create a new
 one to hold the contents of the zone, then update
-:git_file:`playbooks/group_vars/dns.yaml`.
+:git_file:`inventory/service/group_vars/dns.yaml`.
 
 Run::
 

doc/source/etherpad.rst

@@ -19,7 +19,7 @@ At a Glance
   * https://opendev.org/opendev/system-config
   * :git_file:`playbooks/roles/etherpad`
   * :git_file:`playbooks/service-etherpad.yaml`
-  * :git_file:`playbooks/host_vars/etherpad01.opendev.org.yaml`
+  * :git_file:`inventory/service/host_vars/etherpad01.opendev.org.yaml`
 :Projects:
   * http://etherpad.org/
   * https://github.com/ether/etherpad-lite

doc/source/gitea.rst

@@ -17,7 +17,7 @@ At a Glance
   * :git_file:`playbooks/roles/gitea`
   * :git_file:`playbooks/roles/haproxy`
 :Configuration:
-  * :git_file:`playbooks/group_vars/gitea-lb.yaml`
+  * :git_file:`inventory/service/group_vars/gitea-lb.yaml`
 :Projects:
   * https://gitea.io/
 :Bugs:

doc/source/letsencrypt.rst

@@ -79,7 +79,7 @@ A basic configuration consists of the following steps:
    The roles look for certificate configuration in a
    ``letsencrypt_certs`` variable defined for each host.  This is
    usually done via specific host variables in
-   ``playbooks/host_vars/<hostname>.opendev.org.yaml``.  For a simple
+   ``inventory/service/host_vars/<hostname>.opendev.org.yaml``.  For a simple
    host that wants a single certificate to cover its numeric hostname
    and regular ``CNAME`` this would look like ::
 

doc/source/sysadmin.rst

@@ -134,7 +134,7 @@ To create a new server, do the following:
 
 * You should be able to install and configure most software only with
   ansible or puppet.  Nonetheless, if you need SSH access to the host,
-  add your public key to :git_file:`playbooks/group_vars/all.yaml` and
+  add your public key to :git_file:`inventory/service/group_vars/all.yaml` and
   include a stanza like this in your server class::
 
     extra_users:

inventory/base/group_vars/all.yaml

@@ -184,5 +184,3 @@ disabled_users:
   - shrews
   - dmsimard
   - pabelanger
-
-gerrit_ssh_rsa_pubkey_contents: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+pCQlTAQYmCrOY6aPbvbyKQDcOCXibPNGIjnPPMuEItCS0vtRnqEBz7znWZS5Drq9yKpROh6uFF01ao2VnNjw6f+NdRNV19RWVe6mYN+qa2VrH2caLwBrKPiH0Xc/eK41D55dZU7IWwKYAw/NpiBaBfHavFwipI+rmEb68MH2hcimDdr/bji+0hkh3X+42dkNvmMdtkuCW6nKdAEhnXaHZc5SJR/EvzgRCfB8vbML13p46O9xhoJgn7ZWvMb3vaR5jxIkQwstUR36raEVhttBDEuWasWnHYbrM1zd3ooudbTEQf5vXISZKFygHyJFFqb4iQ76i+hDlb0VQKZCdaol gerrit-code-review@829f141b0fa5

inventory/service/group_vars/all.yaml

@@ -0,0 +1 @@
+gerrit_ssh_rsa_pubkey_contents: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+pCQlTAQYmCrOY6aPbvbyKQDcOCXibPNGIjnPPMuEItCS0vtRnqEBz7znWZS5Drq9yKpROh6uFF01ao2VnNjw6f+NdRNV19RWVe6mYN+qa2VrH2caLwBrKPiH0Xc/eK41D55dZU7IWwKYAw/NpiBaBfHavFwipI+rmEb68MH2hcimDdr/bji+0hkh3X+42dkNvmMdtkuCW6nKdAEhnXaHZc5SJR/EvzgRCfB8vbML13p46O9xhoJgn7ZWvMb3vaR5jxIkQwstUR36raEVhttBDEuWasWnHYbrM1zd3ooudbTEQf5vXISZKFygHyJFFqb4iQ76i+hDlb0VQKZCdaol gerrit-code-review@829f141b0fa5

playbooks/roles/install-ansible/files/inventory_plugins/openstack.py

@@ -1,313 +0,0 @@
-# Copyright (c) 2012, Marco Vito Moscaritolo <marco@agavee.com>
-# Copyright (c) 2013, Jesse Keating <jesse.keating@rackspace.com>
-# Copyright (c) 2015, Hewlett-Packard Development Company, L.P.
-# Copyright (c) 2016, Rackspace Australia
-# Copyright (c) 2017 Ansible Project
-# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
-
-from __future__ import (absolute_import, division, print_function)
-__metaclass__ = type
-
-DOCUMENTATION = '''
-    name: openstack
-    plugin_type: inventory
-    authors:
-      - Marco Vito Moscaritolo <marco@agavee.com>
-      - Jesse Keating <jesse.keating@rackspace.com>
-    short_description: OpenStack inventory source
-    extends_documentation_fragment:
-        - inventory_cache
-        - constructed
-    description:
-        - Get inventory hosts from OpenStack clouds
-        - Uses openstack.(yml|yaml) YAML configuration file to configure the inventory plugin
-        - Uses standard clouds.yaml YAML configuration file to configure cloud credentials
-    options:
-        show_all:
-            description: toggles showing all vms vs only those with a working IP
-            type: bool
-            default: 'no'
-        inventory_hostname:
-            description: |
-                What to register as the inventory hostname.
-                If set to 'uuid' the uuid of the server will be used and a
-                group will be created for the server name.
-                If set to 'name' the name of the server will be used unless
-                there are more than one server with the same name in which
-                case the 'uuid' logic will be used.
-                Default is to do 'name', which is the opposite of the old
-                openstack.py inventory script's option use_hostnames)
-            type: string
-            choices:
-                - name
-                - uuid
-            default: "name"
-        expand_hostvars:
-            description: |
-                Run extra commands on each host to fill in additional
-                information about the host. May interrogate cinder and
-                neutron and can be expensive for people with many hosts.
-                (Note, the default value of this is opposite from the default
-                old openstack.py inventory script's option expand_hostvars)
-            type: bool
-            default: 'no'
-        private:
-            description: |
-                Use the private interface of each server, if it has one, as
-                the host's IP in the inventory. This can be useful if you are
-                running ansible inside a server in the cloud and would rather
-                communicate to your servers over the private network.
-            type: bool
-            default: 'no'
-        only_clouds:
-            description: |
-                List of clouds from clouds.yaml to use, instead of using
-                the whole list.
-            type: list
-            default: []
-        fail_on_errors:
-            description: |
-                Causes the inventory to fail and return no hosts if one cloud
-                has failed (for example, bad credentials or being offline).
-                When set to False, the inventory will return as many hosts as
-                it can from as many clouds as it can contact. (Note, the
-                default value of this is opposite from the old openstack.py
-                inventory script's option fail_on_errors)
-            type: bool
-            default: 'no'
-        clouds_yaml_path:
-            description: |
-                Override path to clouds.yaml file. If this value is given it
-                will be searched first. The default path for the
-                ansible inventory adds /etc/ansible/openstack.yaml and
-                /etc/ansible/openstack.yml to the regular locations documented
-                at https://docs.openstack.org/os-client-config/latest/user/configuration.html#config-files
-            type: string
-        compose:
-            description: Create vars from jinja2 expressions.
-            type: dictionary
-            default: {}
-        groups:
-            description: Add hosts to group based on Jinja2 conditionals.
-            type: dictionary
-            default: {}
-'''
-
-EXAMPLES = '''
-# file must be named openstack.yaml or openstack.yml
-# Make the plugin behave like the default behavior of the old script
-plugin: openstack
-expand_hostvars: yes
-fail_on_errors: yes
-'''
-
-import collections
-
-from ansible.errors import AnsibleParserError
-from ansible.plugins.inventory import BaseInventoryPlugin, Constructable, Cacheable
-
-try:
-    # Due to the name shadowing we should import other way
-    import importlib
-    sdk = importlib.import_module('openstack')
-    sdk_inventory = importlib.import_module('openstack.cloud.inventory')
-    client_config = importlib.import_module('openstack.config.loader')
-    HAS_SDK = True
-except ImportError:
-    HAS_SDK = False
-
-
-class InventoryModule(BaseInventoryPlugin, Constructable, Cacheable):
-    ''' Host inventory provider for ansible using OpenStack clouds. '''
-
-    NAME = 'openstack'
-
-    def parse(self, inventory, loader, path, cache=True):
-
-        super(InventoryModule, self).parse(inventory, loader, path)
-
-        cache_key = self._get_cache_prefix(path)
-
-        # file is config file
-        self._config_data = self._read_config_data(path)
-
-        msg = ''
-        if not self._config_data:
-            msg = 'File empty. this is not my config file'
-        elif 'plugin' in self._config_data and self._config_data['plugin'] != self.NAME:
-            msg = 'plugin config file, but not for us: %s' % self._config_data['plugin']
-        elif 'plugin' not in self._config_data and 'clouds' not in self._config_data:
-            msg = "it's not a plugin configuration nor a clouds.yaml file"
-        elif not HAS_SDK:
-            msg = "openstacksdk is required for the OpenStack inventory plugin. OpenStack inventory sources will be skipped."
-
-        if msg:
-            raise AnsibleParserError(msg)
-
-        # The user has pointed us at a clouds.yaml file. Use defaults for
-        # everything.
-        if 'clouds' in self._config_data:
-            self._config_data = {}
-
-        if cache:
-            cache = self.get_option('cache')
-        source_data = None
-        if cache:
-            try:
-                source_data = self.cache.get(cache_key)
-            except KeyError:
-                pass
-
-        if not source_data:
-            clouds_yaml_path = self._config_data.get('clouds_yaml_path')
-            if clouds_yaml_path:
-                config_files = (clouds_yaml_path +
-                                client_config.CONFIG_FILES)
-            else:
-                config_files = None
-
-            # TODO(mordred) Integrate shade's logging with ansible's logging
-            sdk.enable_logging()
-
-            cloud_inventory = sdk_inventory.OpenStackInventory(
-                config_files=config_files,
-                private=self._config_data.get('private', False))
-            only_clouds = self._config_data.get('only_clouds', [])
-            if only_clouds and not isinstance(only_clouds, list):
-                raise ValueError(
-                    'OpenStack Inventory Config Error: only_clouds must be'
-                    ' a list')
-            if only_clouds:
-                new_clouds = []
-                for cloud in cloud_inventory.clouds:
-                    if cloud.name in only_clouds:
-                        new_clouds.append(cloud)
-                cloud_inventory.clouds = new_clouds
-
-            expand_hostvars = self._config_data.get('expand_hostvars', False)
-            fail_on_errors = self._config_data.get('fail_on_errors', False)
-
-            source_data = cloud_inventory.list_hosts(
-                expand=expand_hostvars, fail_on_cloud_config=fail_on_errors)
-
-            self.cache.set(cache_key, source_data)
-
-        self._populate_from_source(source_data)
-
-    def _populate_from_source(self, source_data):
-        groups = collections.defaultdict(list)
-        firstpass = collections.defaultdict(list)
-        hostvars = {}
-
-        use_server_id = (
-            self._config_data.get('inventory_hostname', 'name') != 'name')
-        show_all = self._config_data.get('show_all', False)
-
-        for server in source_data:
-            if 'interface_ip' not in server and not show_all:
-                continue
-            firstpass[server['name']].append(server)
-
-        for name, servers in firstpass.items():
-            if len(servers) == 1 and not use_server_id:
-                self._append_hostvars(hostvars, groups, name, servers[0])
-            else:
-                server_ids = set()
-                # Trap for duplicate results
-                for server in servers:
-                    server_ids.add(server['id'])
-                if len(server_ids) == 1 and not use_server_id:
-                    self._append_hostvars(hostvars, groups, name, servers[0])
-                else:
-                    for server in servers:
-                        self._append_hostvars(
-                            hostvars, groups, server['id'], server,
-                            namegroup=True)
-
-        self._set_variables(hostvars, groups)
-
-    def _set_variables(self, hostvars, groups):
-
-        # set vars in inventory from hostvars
-        for host in hostvars:
-
-            # create composite vars
-            self._set_composite_vars(
-                self._config_data.get('compose'), hostvars, host)
-
-            # actually update inventory
-            for key in hostvars[host]:
-                self.inventory.set_variable(host, key, hostvars[host][key])
-
-            # constructed groups based on conditionals
-            self._add_host_to_composed_groups(
-                self._config_data.get('groups'), hostvars, host)
-
-        for group_name, group_hosts in groups.items():
-            self.inventory.add_group(group_name)
-            for host in group_hosts:
-                self.inventory.add_child(group_name, host)
-
-    def _get_groups_from_server(self, server_vars, namegroup=True):
-        groups = []
-
-        region = server_vars['region']
-        cloud = server_vars['cloud']
-        metadata = server_vars.get('metadata', {})
-
-        # Create a group for the cloud
-        groups.append(cloud)
-
-        # Create a group on region
-        groups.append(region)
-
-        # And one by cloud_region
-        groups.append("%s_%s" % (cloud, region))
-
-        # Check if group metadata key in servers' metadata
-        if 'group' in metadata:
-            groups.append(metadata['group'])
-
-        for extra_group in metadata.get('groups', '').split(','):
-            if extra_group:
-                groups.append(extra_group.strip())
-
-        groups.append('instance-%s' % server_vars['id'])
-        if namegroup:
-            groups.append(server_vars['name'])
-
-        for key in ('flavor', 'image'):
-            if 'name' in server_vars[key]:
-                groups.append('%s-%s' % (key, server_vars[key]['name']))
-
-        for key, value in iter(metadata.items()):
-            groups.append('meta-%s_%s' % (key, value))
-
-        az = server_vars.get('az', None)
-        if az:
-            # Make groups for az, region_az and cloud_region_az
-            groups.append(az)
-            groups.append('%s_%s' % (region, az))
-            groups.append('%s_%s_%s' % (cloud, region, az))
-        return groups
-
-    def _append_hostvars(self, hostvars, groups, current_host,
-                         server, namegroup=False):
-        hostvars[current_host] = dict(
-            ansible_ssh_host=server['interface_ip'],
-            ansible_host=server['interface_ip'],
-            openstack=server)
-        self.inventory.add_host(current_host)
-
-        for group in self._get_groups_from_server(server, namegroup=namegroup):
-            groups[group].append(current_host)
-
-    def verify_file(self, path):
-
-        if super(InventoryModule, self).verify_file(path):
-            for fn in ('openstack', 'clouds'):
-                for suffix in ('yaml', 'yml'):
-                    maybe = '{fn}.{suffix}'.format(fn=fn, suffix=suffix)
-                    if path.endswith(maybe):
-                        return True
-        return False

playbooks/roles/install-ansible/files/inventory_plugins/test-fixtures/groups.yaml

@@ -1 +1 @@
-../../../../../../inventory/groups.yaml
+../../../../../../inventory/service/groups.yaml

playbooks/roles/install-ansible/tasks/main.yaml

@@ -98,10 +98,10 @@
     src: ansible.cfg.j2
     dest: /etc/ansible/ansible.cfg
 
-- name: Copy static inventory in place
-  copy:
-    src: 'inventory/{{ item }}'
-    dest: '/etc/ansible/hosts/{{ item }}'
+- name: Remove old inventory files
+  file:
+    path: '/etc/ansible/hosts/{{ item }}'
+    state: absent
   loop:
     - openstack.yaml
     - groups.yaml
@@ -139,14 +139,6 @@
     cmd: bash install_modules.sh
     chdir: /etc/puppet
 
-# NOTE(mordred) The copy of the openstack inventory plugin from 2.6 is busted.
-# It doesn't proerly deal with caching. A fix has been submitted upstream, but
-# for now this is a fixed copy.
-- name: Copy fixed openstack inventory in place
-  copy:
-    src: inventory_plugins/openstack.py
-    dest: /etc/ansible/inventory_plugins/openstack.py
-
 - name: Copy yamlgroup inventory in place
   copy:
     src: inventory_plugins/yamlgroup.py

playbooks/roles/install-ansible/templates/ansible.cfg.j2

@@ -1,5 +1,5 @@
 [defaults]
-inventory=/etc/ansible/hosts/openstack.yaml,/etc/ansible/hosts/groups.yaml,/etc/ansible/hosts/emergency.yaml
+inventory=/home/zuul/src/opendev.org/src/opendev/system-config/inventory/base/hosts.yaml,/home/zuul/src/opendev.org/src/opendev/system-config/inventory/service/groups.yaml,/etc/ansible/hosts/emergency.yaml
 library=/usr/share/ansible
 log_path=/var/log/ansible/ansible.log
 inventory_plugins=/etc/ansible/inventory_plugins

playbooks/zuul/run-base.yaml

@@ -11,7 +11,7 @@
       include_role:
         name: write-inventory
       vars:
-        write_inventory_dest: /etc/ansible/hosts/inventory.yaml
+        write_inventory_dest: /home/zuul/src/opendev.org/opendev/system-config/inventory/base/gate-hosts.yaml
         write_inventory_exclude_hostvars:
           - ansible_user
           - ansible_python_interpreter
@@ -27,7 +27,7 @@
         path: /etc/ansible/ansible.cfg
         section: defaults
         option: inventory
-        value: /etc/ansible/hosts/inventory.yaml,/etc/ansible/hosts/groups.yaml,/etc/ansible/hosts/gate-groups.yaml
+        value: /home/zuul/src/opendev.org/opendev/system-config/inventory/base/gate-hosts.yaml,/home/zuul/src/opendev.org/opendev/system-config/inventory/service/groups.yaml,/etc/ansible/hosts/gate-groups.yaml
     - name: Make host_vars directory
       file:
         path: "/etc/ansible/hosts/host_vars"
@@ -84,9 +84,10 @@
 
     # In prod, bridge installs a zuul user, but in zuul we already have a zuul user, so we really need 
     # to not modify it.
+    # TODO(mordred) We should be able to replace this with overriding directly.
     - name: Load bridge hostvars
       slurp:
-        path: /home/zuul/src/opendev.org/opendev/system-config/playbooks/host_vars/bridge.openstack.org.yaml
+        path: /home/zuul/src/opendev.org/opendev/system-config/inventory/service/host_vars/bridge.openstack.org.yaml
       register: bridge_hostvar_content
     - name: Parse bridge_hostvars
       set_fact:
@@ -100,7 +101,7 @@
     - name: Save bridge hostvars
       copy:
         content: "{{ bridge_hostvars | to_nice_yaml }}"
-        dest: /home/zuul/src/opendev.org/opendev/system-config/playbooks/host_vars/bridge.openstack.org.yaml
+        dest: /home/zuul/src/opendev.org/opendev/system-config/inventory/service/host_vars/bridge.openstack.org.yaml
       become: true
 
     - name: Run base.yaml

testinfra/conftest.py

@@ -7,7 +7,7 @@ def zuul_data():
 
     data = {}
 
-    with open('/etc/ansible/hosts/inventory.yaml') as f:
+    with open('/home/zuul/src/opendev.org/opendev/system-config/inventory/base/gate-hosts.yaml') as f:
         inventory = yaml.load(f)
         data['inventory'] = inventory