Refactor ceph_client for multi-OS and ceph

This patch refactors the ceph_client role to add support for multiple
operating system distros and multiple sources for Ceph packages.

The support of multiple sources for the Ceph packages  is important
to organizations that must get packages from Canonical for service and
support. The current Ceph repo setup goes to upstream ceph.com
repositories and does not work with the UCA.

The use of UCA is also important when running OpenStack on the ppc64le
architecture because ceph.com does not have Debian packages available
for this architecture and the default trusty and trusty-updates repos
only have Ceph Giant, whereas the user can get later releases such as
Ceph Jewel from UCA.

The multiple operating system support for Trusty and Xenial also plays
into this since Xenial has Ceph Jewel by default.  For Xenial OSA
deployments users may want to use the modern ceph client already
available for the distro.

The choice of which Ceph source to use is simple for deployers. They
simply set it with the ceph_pkg_source variable but have additional
variables they can tweak to pick specific Ceph versions from the
sources:

The ceph_pkg_source variable controls the install source for the
Ceph packages.
Valid values include:
  * ceph This option installs Ceph from a ceph.com repo.  Additional
         variables to adjust items such as Ceph release and regional
         download mirror can be found in vars/*.yml

  * uca  This option installs Ceph from the Ubuntu Cloud Archive.
         Additional variables to adjust items such as the
         OpenStack/Ceph release can be found in vars/*.yml.

  * distro This options installs Ceph from the operating system's
           default repository and unlike the other options does not
           attempt to manage package keys or add additional package
           repositories.

Change-Id: Ib21b3f76ccf4556548180c8694786d43fa0a024f

playbooks/roles/ceph_client/defaults/main.yml

@@ -1,5 +1,6 @@
 ---
 # Copyright 2015, Serge van Ginderachter <serge@vanginderachter.be>
+# Copyright 2016 IBM Corp
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,10 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-## APT Cache Options
+# to use Ceph in OSA, you need to
-cache_timeout: 600
-
-# to user Ceph in OSA, you need to
 # - have the needed pools and a client user (for glance, cinder and/or nova)
 #   pre-provisioned in your ceph cluster; OSA assumes to have root access to
 #   the monitor hosts
@@ -27,25 +25,19 @@ cache_timeout: 600
 # - cinder gets configured with ceph if there are cinder backends defined with
 #   the rbd driver (see openstack_user_config.yml.example)
 
-# Ceph GPG Keys
+# The ceph_pkg_source variable controls the install source for the Ceph packages.
-ceph_gpg_keys:
+# Valid values include:
-  - key_name: 'ceph'
+#   * ceph This option installs Ceph from a ceph.com repo.  Additional variables to
-    keyserver: 'hkp://keyserver.ubuntu.com:80'
+#          adjust items such as Ceph release and regional download mirror can be found
-    fallback_keyserver: 'hkp://p80.pool.sks-keyservers.net:80'
+#          in vars/*.yml
-    hash_id: '0xe84ac2c0460f3994'
+#
-
+#   * uca  This option installs Ceph from the Ubuntu Cloud Archive. Additional variables
-# The apt-key command won't del a key when you give it the hash_id, so we have
+#          to adjust items such as the OpenStack/Ceph release can be found in vars/*.yml.
-# to use the short key ID here instead.
+#
-ceph_revoked_gpg_keys:
+#   * distro This options installs Ceph from the operating system's default repository and
-  - '17ED316D'
+#            unlike the other options does not attempt to manage package keys or add additional
-
+#            package repositories.
-# Ceph Repositories
+ceph_pkg_source: ceph
-ceph_apt_repo_url_region: "download"  # or "eu" for Netherlands based mirror
-ceph_stable_release: hammer
-ceph_apt_repo_url: "http://{{ ceph_apt_repo_url_region }}.ceph.com/debian-{{ ceph_stable_release }}/"
-ceph_apt_repo:
-  repo: "deb {{ ceph_apt_repo_url }} {{ ansible_lsb.codename }} main"
-  state: "present"
 
 ceph_apt_pinned_packages: [{ package: "*", release: Inktank,  priority: 1001 }]
 

playbooks/roles/ceph_client/tasks/ceph_all.yml

@@ -22,7 +22,18 @@
     - ceph-config
     - ceph-auth
 
+- name: Gather variables for each operating system
+  include_vars: "{{ item }}"
+  with_first_found:
+    - "{{ ansible_distribution | lower }}-{{ ansible_distribution_version | lower }}.yml"
+    - "{{ ansible_distribution | lower }}.yml"
+    - "{{ ansible_os_family | lower }}.yml"
+  tags:
+    - always
+
 - include:  ceph_preinstall.yml
+  when:
+    - ceph_pkg_source != 'distro'
   tags:     ceph-preinstall
 
 - include:  ceph_install.yml

playbooks/roles/ceph_client/tasks/ceph_install.yml

@@ -1,5 +1,5 @@
 ---
-# Copyright 2015, Serge van Ginderachter <serge@vanginderachter.be>
+# Copyright 2016 IBM Corp
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,35 +13,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-#TODO(evrardjp): Replace the next 2 tasks by a standard apt with cache
+- include: ceph_install_apt.yml
-#when https://github.com/ansible/ansible-modules-core/pull/1517 is merged
+  when:
-#in 1.9.x or we move to 2.0 (if tested working)
+    - ansible_pkg_mgr == 'apt'
-- name: Check apt last update file
-  stat:
-    path: /var/cache/apt
-  register: apt_cache_stat
   tags:
     - ceph-apt-packages
-
-- name: Update apt if needed
-  apt:
-    update_cache: yes
-  when: "ansible_date_time.epoch|float - apt_cache_stat.stat.mtime > {{cache_timeout}}"
-  tags:
-    - ceph-apt-packages
-
-- name: Install ceph packages
-  apt:
-    name: '{{ item.1 }}'
-    state: latest
-  register: install_packages
-  until: install_packages|success
-  retries: 5
-  delay: 2
-  with_subelements:
-  - ceph_components
-  - package
-  when: inventory_hostname in groups[item.0.component]
-  notify:
-  - Restart os services
-

playbooks/roles/ceph_client/tasks/ceph_install_apt.yml

@@ -0,0 +1,46 @@
+---
+# Copyright 2015, Serge van Ginderachter <serge@vanginderachter.be>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#TODO(evrardjp): Replace the next 2 tasks by a standard apt with cache
+#when https://github.com/ansible/ansible-modules-core/pull/1517 is merged
+#in 1.9.x or we move to 2.0 (if tested working)
+- name: Check apt last update file
+  stat:
+    path: /var/cache/apt
+  register: apt_cache_stat
+  tags:
+    - ceph-apt-packages
+
+- name: Update apt if needed
+  apt:
+    update_cache: yes
+  when: "ansible_date_time.epoch|float - apt_cache_stat.stat.mtime > {{cache_timeout}}"
+  tags:
+    - ceph-apt-packages
+
+- name: Install ceph packages
+  apt:
+    name: '{{ item.1 }}'
+    state: latest
+  register: install_packages
+  until: install_packages|success
+  retries: 5
+  delay: 2
+  with_subelements:
+  - ceph_components
+  - package
+  when: inventory_hostname in groups[item.0.component]
+  notify:
+  - Restart os services

playbooks/roles/ceph_client/tasks/ceph_preinstall.yml

@@ -1,5 +1,6 @@
 ---
 # Copyright 2015, Serge van Ginderachter <serge@vanginderachter.be>
+# Copyright 2016 IBM Corp
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,65 +14,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-- name: Remove revoked ceph apt-keys
+- include: ceph_preinstall_apt.yml
-  apt_key:
+  when:
-    id: "{{ item }}"
+    - ansible_pkg_mgr == 'apt'
-    state: "absent"
-  register: revoke_keys
-  with_items: ceph_revoked_gpg_keys
   tags:
     - ceph-apt-keys
-
-- name: Add ceph apt-keys
-  apt_key:
-    id: "{{ item.hash_id }}"
-    keyserver: "{{ item.keyserver | default(omit) }}"
-    data: "{{ item.data | default(omit) }}"
-    url: "{{ item.url | default(omit) }}"
-    state: "present"
-  register: add_keys
-  until: add_keys|success
-  ignore_errors: True
-  retries: 5
-  delay: 2
-  with_items: ceph_gpg_keys
-  tags:
-    - ceph-apt-keys
-
-- name: Add ceph apt-keys using fallback keyserver
-  apt_key:
-    id: "{{ item.hash_id }}"
-    keyserver: "{{ item.fallback_keyserver | default(omit) }}"
-    url: "{{ item.fallback_url | default(omit) }}"
-    state: "present"
-  register: add_keys_fallback
-  until: add_keys_fallback|success
-  retries: 5
-  delay: 2
-  with_items: ceph_gpg_keys
-  when: add_keys|failed and (item.fallback_keyserver is defined or item.fallback_url is defined)
-  tags:
-    - ceph-apt-keys
-
-- name: Add ceph repo(s)
-  apt_repository:
-    repo: "{{ ceph_apt_repo.repo }}"
-    state: "{{ ceph_apt_repo.state }}"
-  register: add_repos
-  until: add_repos|success
-  retries: 5
-  delay: 2
-  tags:
-    - ceph-repos
-
-# This is being added specifically for when a key is revoked, but should apply
-# to other tasks also.  The cache needs updating after changing keys but
-# ceph_install.yml (where packages get installed) only does so if cache > 600
-# seconds.
-- name: Update apt cache
-  apt:
-    update_cache: yes
-  when: revoke_keys|changed or add_keys|changed or add_keys_fallback|changed or add_repos|changed
-  tags:
-    - ceph-apt-keys
-    - ceph-repos

playbooks/roles/ceph_client/tasks/ceph_preinstall_apt.yml

@@ -0,0 +1,93 @@
+---
+# Copyright 2016 IBM Corp
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Remove revoked ceph apt-keys
+  apt_key:
+    id: "{{ item }}"
+    state: "absent"
+  register: revoke_keys
+  with_items: ceph_revoked_gpg_keys
+  when: ceph_pkg_source == 'ceph'
+  tags:
+    - ceph-apt-keys
+
+- name: Add ceph apt-keys
+  apt_key:
+    id: "{{ item.hash_id }}"
+    keyserver: "{{ item.keyserver | default(omit) }}"
+    data: "{{ item.data | default(omit) }}"
+    url: "{{ item.url | default(omit) }}"
+    state: "present"
+  register: add_keys
+  until: add_keys|success
+  ignore_errors: True
+  retries: 5
+  delay: 2
+  with_items: ceph_gpg_keys
+  when: ceph_pkg_source == 'ceph'
+  tags:
+    - ceph-apt-keys
+
+- name: Add ceph apt-keys using fallback keyserver
+  apt_key:
+    id: "{{ item.hash_id }}"
+    keyserver: "{{ item.fallback_keyserver | default(omit) }}"
+    url: "{{ item.fallback_url | default(omit) }}"
+    state: "present"
+  register: add_keys_fallback
+  until: add_keys_fallback|success
+  retries: 5
+  delay: 2
+  with_items: ceph_gpg_keys
+  when: ceph_pkg_source == 'ceph' and
+        add_keys|failed and
+        (item.fallback_keyserver is defined or
+         item.fallback_url is defined)
+  tags:
+    - ceph-apt-keys
+
+- name: add ubuntu cloud archive key package
+  apt:
+    pkg: ubuntu-cloud-keyring
+  register: add_keys
+  when: ceph_pkg_source == 'uca'
+  tags:
+    - ceph-apt-keys
+
+- name: Add ceph repo(s)
+  apt_repository:
+    repo: "{{ ceph_apt_repos[ceph_pkg_source].repo }}"
+    state: "{{ ceph_apt_repos[ceph_pkg_source].state }}"
+  register: add_repos
+  until: add_repos|success
+  retries: 5
+  delay: 2
+  tags:
+    - ceph-repos
+
+# This is being added specifically for when a key is revoked, but should apply
+# to other tasks also.  The cache needs updating after changing keys but
+# ceph_install.yml (where packages get installed) only does so if cache > 600
+# seconds.
+- name: Update apt cache
+  apt:
+    update_cache: yes
+  when: (revoke_keys|changed or
+        add_keys|changed or
+        add_keys_fallback|changed or
+        add_repos|changed)
+  tags:
+    - ceph-apt-keys
+    - ceph-repos

playbooks/roles/ceph_client/vars/ubuntu-14.04.yml

@@ -0,0 +1,49 @@
+---
+# Copyright 2016 IBM Corp
+# Copyright 2015, Serge van Ginderachter <serge@vanginderachter.be>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+## APT Cache Options
+cache_timeout: 600
+
+# Ceph GPG Keys
+ceph_gpg_keys:
+  - key_name: 'ceph'
+    keyserver: 'hkp://keyserver.ubuntu.com:80'
+    fallback_keyserver: 'hkp://p80.pool.sks-keyservers.net:80'
+    hash_id: '0xe84ac2c0460f3994'
+
+# The apt-key command won't del a key when you give it the hash_id, so we have
+# to use the short key ID here instead.
+ceph_revoked_gpg_keys:
+  - '17ED316D'
+
+# Ceph.com repository variables
+ceph_apt_repo_url_region: "download"  # or "eu" for Netherlands based mirror
+ceph_stable_release: hammer
+ceph_apt_repo_url: "http://{{ ceph_apt_repo_url_region }}.ceph.com/debian-{{ ceph_stable_release }}/"
+
+# Ubuntu Cloud Archive variables
+uca_openstack_release: mitaka
+uca_apt_repo_url: "http://ubuntu-cloud.archive.canonical.com/ubuntu"
+uca_repo_dist: "{{ ansible_lsb.codename }}-updates/{{ uca_openstack_release }}"
+
+# Apt repositories
+ceph_apt_repos:
+    ceph:
+        repo: "deb {{ ceph_apt_repo_url }} {{ ansible_lsb.codename }} main"
+        state: "present"
+    uca:
+        repo: "deb {{ uca_apt_repo_url }} {{ uca_repo_dist }} main"
+        state: "present"

playbooks/roles/ceph_client/vars/ubuntu-16.04.yml

@@ -0,0 +1,50 @@
+---
+# Copyright 2016 IBM Corp
+# Copyright 2015, Serge van Ginderachter <serge@vanginderachter.be>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+## APT Cache Options
+cache_timeout: 600
+
+# Ceph GPG Keys
+ceph_gpg_keys:
+  - key_name: 'ceph'
+    keyserver: 'hkp://keyserver.ubuntu.com:80'
+    fallback_keyserver: 'hkp://p80.pool.sks-keyservers.net:80'
+    hash_id: '0xe84ac2c0460f3994'
+
+# The apt-key command won't del a key when you give it the hash_id, so we have
+# to use the short key ID here instead.
+ceph_revoked_gpg_keys:
+  - '17ED316D'
+
+# Ceph.com repository variables
+ceph_apt_repo_url_region: "download"  # or "eu" for Netherlands based mirror
+ceph_stable_release: hammer
+ceph_apt_repo_url: "http://{{ ceph_apt_repo_url_region }}.ceph.com/debian-{{ ceph_stable_release }}/"
+
+# Ubuntu Cloud Archive variables
+# TODO(smatzek) Revisit the default uca release for 16.04 at newton-1
+uca_openstack_release: mitaka
+uca_apt_repo_url: "http://ubuntu-cloud.archive.canonical.com/ubuntu"
+uca_repo_dist: "{{ ansible_lsb.codename }}-updates/{{ uca_openstack_release }}"
+
+# Apt repositories
+ceph_apt_repos:
+    ceph:
+        repo: "deb {{ ceph_apt_repo_url }} {{ ansible_lsb.codename }} main"
+        state: "present"
+    uca:
+        repo: "deb {{ uca_apt_repo_url }} {{ uca_repo_dist }} main"
+        state: "present"

releasenotes/notes/ceph-from-uca-and-distro-2fa04e03c39a61bc.yaml

@@ -0,0 +1,21 @@
+---
+features:
+  - |
+    Support had been added to install the ceph_client packages and
+    dependencies from Ceph.com, Ubuntu Cloud Archive (UCA), or the
+    operating system's default repository.
+
+    The ``ceph_pkg_source`` variable controls the install source for the
+    Ceph packages. Valid values include:
+
+    * ``ceph``: This option installs Ceph from a ceph.com repo. Additional
+      variables to adjust items such as Ceph release and regional download
+      mirror can be found in the variables files.
+
+    * ``uca``: This option installs Ceph from the Ubuntu Cloud Archive. Additional
+      variables to adjust items such as the OpenStack/Ceph release can be
+      found in the variables files.
+
+    * ``distro``: This options installs Ceph from the operating system's default
+      repository and unlike the other options does not attempt to manage
+      package keys or add additional package repositories.