Add ceph-ansible v3 support

ceph-ansible has undergone significant refactoring in their v3
development, adding new required roles such as ceph-defaults and
ceph-config, which are used to provide vars and configuration to
the ceph service roles. These roles must be executed before the
service roles to avoid missing vars[1][2].

During the v3 refactoring, ceph-common was removed as a galaxy-style
role dependency in the service roles meta files[3]. This means we
will need to explicitly execute ceph-common from now on also.

This change adds the defaults and config roles and executes them.
Also some minor cleanup such as alphabetizing the OpenStack roles
list is done.

Also added is an upgrade playbook, reno, and docs to assist in
cleaning up the older galaxy-named ceph common roles which are
no longer galaxy namespaced in our cloning configuration.

[1] http://jenkins-logs.objects-us-dfw-1.cloud.lstn.net/osa-ci/490192/1/21/logs/console.log
[2] https://github.com/ceph/ceph-ansible/issues/1737
[3] https://github.com/ceph/ceph-ansible/pull/1727

Change-Id: Ia8c0cb0a23f331fce7914afbfc05ef54ee3ffb0e

ansible-role-requirements.yml

@@ -70,6 +70,10 @@
   scm: git
   src: https://git.openstack.org/openstack/openstack-ansible-os_cinder
   version: master
+- name: os_designate
+  scm: git
+  src: https://git.openstack.org/openstack/openstack-ansible-os_designate
+  version: master
 - name: os_glance
   scm: git
   src: https://git.openstack.org/openstack/openstack-ansible-os_glance
@@ -94,9 +98,9 @@
   scm: git
   src: https://git.openstack.org/openstack/openstack-ansible-os_magnum
   version: master
-- name: os_trove
+- name: os_molteniron
   scm: git
-  src: https://git.openstack.org/openstack/openstack-ansible-os_trove
+  src: https://git.openstack.org/openstack/openstack-ansible-os_molteniron
   version: master
 - name: os_neutron
   scm: git
@@ -106,6 +110,10 @@
   scm: git
   src: https://git.openstack.org/openstack/openstack-ansible-os_nova
   version: master
+- name: os_octavia
+  scm: git
+  src: https://git.openstack.org/openstack/openstack-ansible-os_octavia
+  version: master
 - name: os_rally
   scm: git
   src: https://git.openstack.org/openstack/openstack-ansible-os_rally
@@ -118,10 +126,18 @@
   scm: git
   src: https://git.openstack.org/openstack/openstack-ansible-os_swift
   version: master
+- name: os_tacker
+  scm: git
+  src: https://git.openstack.org/openstack/openstack-ansible-os_tacker
+  version: master
 - name: os_tempest
   scm: git
   src: https://git.openstack.org/openstack/openstack-ansible-os_tempest
   version: master
+- name: os_trove
+  scm: git
+  src: https://git.openstack.org/openstack/openstack-ansible-os_trove
+  version: master
 - name: plugins
   scm: git
   src: https://git.openstack.org/openstack/openstack-ansible-plugins
@@ -166,46 +182,34 @@
   scm: git
   src: https://github.com/logan2211/ansible-resolvconf
   version: master
-- name: os_designate
+- name: ceph-defaults
   scm: git
-  src: https://git.openstack.org/openstack/openstack-ansible-os_designate
+  src: https://github.com/ceph/ansible-ceph-defaults
   version: master
-- name: ceph.ceph-common
+- name: ceph-common
   scm: git
   src: https://github.com/ceph/ansible-ceph-common
-  version: 41b7be03dec81e16502386ef17b7af96db6f0d6f
+  version: master
-- name: ceph.ceph-docker-common
+- name: ceph-config
   scm: git
-  src: https://github.com/ceph/ansible-ceph-docker-common
+  src: https://github.com/ceph/ansible-ceph-config
-  version: ca86fd0ef6d24aa2c750a625acdcb8012c374aa0
+  version: master
 - name: ceph-mon
   scm: git
   src: https://github.com/ceph/ansible-ceph-mon
-  version: 235ea7526a2bb621317b5263f3841deef7ab49e6
+  version: master
 - name: ceph-mgr
   scm: git
   src: https://github.com/ceph/ansible-ceph-mgr
-  version: d9ddb83bae6b054549efa1c96b0a37c63073ebea
+  version: master
 - name: ceph-osd
   scm: git
   src: https://github.com/ceph/ansible-ceph-osd
-  version: 3d4fceb960fbaa0331b965d0a9a4f3511022b86b
-- name: os_octavia
-  scm: git
-  src: https://git.openstack.org/openstack/openstack-ansible-os_octavia
-  version: master
-- name: os_molteniron
-  scm: git
-  src: https://git.openstack.org/openstack/openstack-ansible-os_molteniron
   version: master
 - name: opendaylight
   scm: git
   src: https://git.opendaylight.org/gerrit/p/integration/packaging/ansible-opendaylight.git
   version: master
-- name: os_tacker
-  scm: git
-  src: https://git.openstack.org/openstack/openstack-ansible-os_tacker
-  version: master
 - name: haproxy_endpoints
   scm: git
   src: https://github.com/logan2211/ansible-haproxy-endpoints

doc/source/user/manual-upgrade.rst

@@ -112,6 +112,20 @@ For more information, see :ref:`pip-conf-removal`.
 
     # openstack-ansible "${UPGRADE_PLAYBOOKS}/pip-conf-removal.yml"
 
+Clean up the ceph-ansible galaxy namespaced roles
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ceph-ansible common roles are no longer namespaced with a galaxy-style
+'.' (ie. ``ceph.ceph-common`` is now cloned as ``ceph-common``), due to a
+change in the way upstream meta dependencies are handled in the ceph roles.
+The roles will be cloned according to the new naming, and an upgrade
+playbook ``ceph-galaxy-removal.yml`` has been added to clean up the stale
+galaxy-named roles.
+
+.. code-block:: console
+
+    # openstack-ansible "${UPGRADE_PLAYBOOKS}/ceph-galaxy-removal.yml"
+
 Upgrade hosts
 ~~~~~~~~~~~~~
 

doc/source/user/reference-upgrade-playbooks.rst

@@ -48,6 +48,18 @@ packages on the repo servers. If this file exists on a repo server or a
 physical node, it causes a circular dependency issue and the upgrade fails.
 This playbook removes the file on all the repo servers and physical nodes.
 
+.. _ceph-galaxy-removal:
+
+ceph-galaxy-removal.yml
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The ceph-ansible common roles are no longer namespaced with a galaxy-style
+'.' (ie. ``ceph.ceph-common`` is now cloned as ``ceph-common``), due to a
+change in the way upstream meta dependencies are handled in the ceph roles.
+The roles will be cloned according to the new naming, and an upgrade
+playbook ``ceph-galaxy-removal.yml`` has been added to clean up the stale
+galaxy-named roles.
+
 .. _setup-infra-playbook:
 
 setup-infrastructure.yml

group_vars/all/ceph.yml

@@ -19,7 +19,8 @@ ceph_client_package_state: "{{ package_state }}"
 mon_group_name: ceph-mon
 mgr_group_name: "{{ mon_group_name }}"
 osd_group_name: ceph-osd
-ceph_stable: true
+ceph_origin: repository
+ceph_repository: community
 # The _stable_release var is used by both the OSA ceph_client role and the
 # ceph-ansible roles. It is defaulted in ceph_client but set here to keep the
 # OSA/ceph-ansible integrations in sync.

playbooks/ceph-install.yml

@@ -33,7 +33,8 @@
         key: "{{ ceph_stable_key }}"
         state: present
       when:
-        - ceph_stable
+        - ceph_origin == 'repository'
+        - ceph_repository == 'community'
         - ansible_pkg_mgr in ['yum', 'dnf']
     - name: add ceph stable repository
       package:
@@ -41,7 +42,8 @@
         state: present
       changed_when: false
       when:
-        - ceph_stable
+        - ceph_origin == 'repository'
+        - ceph_repository == 'community'
         - ansible_pkg_mgr in ['yum', 'dnf']
     - name: Purge yum/dnf cache
       command: "{{ ansible_pkg_mgr }} clean all"
@@ -84,19 +86,28 @@
       when:
         - ansible_pkg_mgr in ['yum', 'dnf']
   roles:
-    - role: "ceph-mon"
+    - role: ceph-defaults
       tags:
         - skip_ansible_lint
-    - role: "ceph-mgr"
+    - role: ceph-common
       tags:
         - skip_ansible_lint
-    - role: "rsyslog_client"
+    - role: ceph-config
+      tags:
+        - skip_ansible_lint
+    - role: ceph-mon
+      tags:
+        - skip_ansible_lint
+    - role: ceph-mgr
+      tags:
+        - skip_ansible_lint
+    - role: rsyslog_client
       rsyslog_client_log_rotate_file: ceph_log_rotate
       rsyslog_client_log_dir: "/var/log/ceph"
       rsyslog_client_config_name: "99-ceph-rsyslog-client.conf"
       tags:
         - rsyslog
-    - role: "system_crontab_coordination"
+    - role: system_crontab_coordination
       tags:
         - crontab
   vars:
@@ -137,7 +148,8 @@
         key: "{{ ceph_stable_key }}"
         state: present
       when:
-        - ceph_stable
+        - ceph_origin == 'repository'
+        - ceph_repository == 'community'
         - ansible_pkg_mgr in ['yum', 'dnf']
     - name: add ceph stable repository
       package:
@@ -145,7 +157,8 @@
         state: present
       changed_when: false
       when:
-        - ceph_stable
+        - ceph_origin == 'repository'
+        - ceph_repository == 'community'
         - ansible_pkg_mgr in ['yum', 'dnf']
     - name: Purge yum/dnf cache
       command: "{{ ansible_pkg_mgr }} clean all"
@@ -163,16 +176,25 @@
       when:
         - ansible_pkg_mgr in ['yum', 'dnf']
   roles:
-    - role: "ceph-osd"
+    - role: ceph-defaults
       tags:
         - skip_ansible_lint
-    - role: "rsyslog_client"
+    - role: ceph-common
+      tags:
+        - skip_ansible_lint
+    - role: ceph-config
+      tags:
+        - skip_ansible_lint
+    - role: ceph-osd
+      tags:
+        - skip_ansible_lint
+    - role: rsyslog_client
       rsyslog_client_log_rotate_file: ceph_log_rotate
       rsyslog_client_log_dir: "/var/log/ceph"
       rsyslog_client_config_name: "99-ceph-rsyslog-client.conf"
       tags:
         - rsyslog
-    - role: "system_crontab_coordination"
+    - role: system_crontab_coordination
       tags:
         - crontab
   environment: "{{ deployment_environment_variables | default({}) }}"

releasenotes/notes/ceph-ansible-v3-a570cf4477c77860.yaml

@@ -0,0 +1,15 @@
+---
+upgrade:
+  - The ceph-ansible integration has been updated to support the ceph-ansible
+    v3.0 series tags. The new v3.0 series brings a significant refactoring of
+    the ceph-ansible roles and vars, so it is strongly recommended to consult
+    the upstream `ceph-ansible documentation
+    <http://docs.ceph.com/ceph-ansible/master/>`_ to perform any required
+    vars migrations before you upgrade.
+critical:
+  - The ceph-ansible integration has been updated to support the ceph-ansible
+    v3.0 series tags. The new v3.0 series brings a significant refactoring of
+    the ceph-ansible roles and vars, so it is strongly recommended to consult
+    the upstream `ceph-ansible documentation
+    <http://docs.ceph.com/ceph-ansible/master/>`_ to perform any required
+    vars migrations before you upgrade.

releasenotes/notes/ceph-galaxy-roles-1a0a13be026e45d8.yaml

@@ -0,0 +1,8 @@
+---
+upgrade:
+  - The ceph-ansible common roles are no longer namespaced with a galaxy-style
+    '.' (ie. ``ceph.ceph-common`` is now cloned as ``ceph-common``), due to a
+    change in the way upstream meta dependencies are handled in the ceph roles.
+    The roles will be cloned according to the new naming, and an upgrade
+    playbook ``ceph-galaxy-removal.yml`` has been added to clean up the stale
+    galaxy-named roles.

scripts/run-upgrade.sh

@@ -175,6 +175,7 @@ function main {
         RUN_TASKS+=("${UPGRADE_PLAYBOOKS}/deploy-config-changes.yml")
         RUN_TASKS+=("${UPGRADE_PLAYBOOKS}/user-secrets-adjustment.yml")
         RUN_TASKS+=("${UPGRADE_PLAYBOOKS}/pip-conf-removal.yml")
+        RUN_TASKS+=("${UPGRADE_PLAYBOOKS}/ceph-galaxy-removal.yml")
         # we don't want to trigger galera container restarts yet
         RUN_TASKS+=("setup-hosts.yml --limit '!galera_all'")
         # add new container config to galera containers but don't restart

scripts/upgrade-utilities/playbooks/ceph-galaxy-removal.yml

@@ -0,0 +1,31 @@
+---
+# Copyright 2017, Logan Vig <logan2211@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Remove the ceph galaxy named roles
+  hosts: localhost
+  gather_facts: true
+  user: root
+  tasks:
+    # These roles used to be named using galaxy format due to the naming of the
+    # meta dependencies in ceph-ansible. Now the meta dependencies no longer
+    # exist so we are free to name them more consistently with the rest of
+    # OSA's roles.
+    - name: Remove ceph galaxy named roles if found
+      file:
+        path: "/etc/ansible/roles/{{ item }}"
+        state: "absent"
+      with_items:
+        - ceph.ceph-docker-common
+        - ceph.ceph-common

tests/roles/bootstrap-host/templates/user_variables_ceph.yml.j2

@@ -18,7 +18,7 @@ common_single_host_mode: true
 monitor_interface: eth1 # Management network in the AIO
 public_network: "{{ (mgmt_range ~ '.0/' ~ netmask) | ipaddr('net') }}"
 journal_size: 100
-journal_collocation: true
+osd_scenario: collocated
 pool_default_pg_num: 32
 openstack_config: true # Ceph ansible automatically creates pools & keys
 cinder_ceph_client: cinder