Support Docker CE in bootstrap-servers

Kolla Ansible's bootstrap-servers command provides support for
installing the Docker engine. This is currently done using the packages
at https://apt.dockerproject.org and https://yum.dockerproject.org.
These packages are outdated, with the most recent packages from May 2017
- docker-engine-17.05.

The source for up to date docker packages is
https://download.docker.com, which was introduced with the move to
Docker Community Edition (CE) and Docker Enterprise Edition (EE).

This change adds support to bootstrap-servers for Docker CE for CentOS
and Ubuntu.

It also adds a new variable, 'enable_docker_repo', which controls
whether a package repository for Docker will be enabled.

It also adds a new variable, 'docker_legacy_packages', which controls
whether the legacy packages at dockerproject.org will be used or the
newer packages at docker.com. The default value for this variable is
'false', meaning to use Docker CE.

Upgrading from docker-engine to docker-ce has been tested on CentOS 7.5
and Ubuntu 16.04, by running 'kolla-ansible bootstrap-servers' with
'docker_legacy_packages' set to 'false'. The upgrades were successful,
but result in all containers being stopped. For this reason, the
bootstrap-servers command checks running containers prior to upgrading
packages, and ensures they are running after the package upgrade is
complete.

As mentioned in the release note, care should be taken when upgrading
Docker with clustered services, which could lose quorum. To avoid this,
use --serial or --limit to apply the change in batches.

Change-Id: I6dfd375c868870f8646ef1a8f02c70812e8f6271
Implements: blueprint docker-ce

ansible/kolla-host.yml

@@ -33,7 +33,8 @@
 
 - name: Apply role baremetal
   hosts: baremetal
-  gather_facts: no
+  serial: '{{ kolla_serial|default("0") }}'
+  gather_facts: false
   roles:
     - { role: baremetal,
         tags: baremetal }

ansible/roles/baremetal/defaults/main.yml

@@ -1,10 +1,52 @@
 ---
-docker_apt_url: "{{ 'http://obs.linaro.org/ERP:/17.12/Debian_9' if ansible_architecture == 'aarch64' else 'https://apt.dockerproject.org' }}"
-docker_apt_key_file: "{{ 'Release.key' if ansible_architecture == 'aarch64' else 'gpg' }}"
-docker_apt_key_id: "{{ 'C32DA102AD89C2BE' if ansible_architecture == 'aarch64' else 'F76221572C52609D' }}"
+# Whether to enable a package repository for Docker.
+enable_docker_repo: true
 
-docker_yum_url: "https://yum.dockerproject.org"
-docker_gpg_fingerprint: "58118E89F3A912897C070ADBF76221572C52609D"
+# Whether to use the legacy Docker packages at dockerproject.org instead of the
+# newer packages at docker.com.
+docker_legacy_packages: false
+
+# Docker APT repository configuration.
+docker_apt_url: "{% if docker_legacy_packages | bool %}{{ docker_legacy_apt_url }}{% else %}{{ docker_new_apt_url }}{% endif %}"
+docker_apt_repo: "{% if docker_legacy_packages | bool %}{{ docker_legacy_apt_repo }}{% else %}{{ docker_new_apt_repo }}{% endif %}"
+docker_apt_key_file: "{% if docker_legacy_packages | bool %}{{ docker_legacy_apt_key_file }}{% else %}{{ docker_new_apt_key_file }}{% endif %}"
+docker_apt_key_id: "{% if docker_legacy_packages | bool %}{{ docker_legacy_apt_key_id }}{% else %}{{ docker_new_apt_key_id }}{% endif %}"
+docker_apt_package: "{% if docker_legacy_packages | bool %}{{ docker_legacy_apt_package }}{% else %}{{ docker_new_apt_package }}{% endif %}"
+
+# Docker APT repository configuration when docker_legacy_packages is false.
+docker_new_apt_url: "https://download.docker.com/linux/{{ ansible_distribution | lower }}"
+docker_new_apt_repo: "deb {{ docker_new_apt_url }} {{ ansible_lsb.codename }} stable"
+docker_new_apt_key_file: "gpg"
+docker_new_apt_key_id: "0EBFCD88"
+docker_new_apt_package: "docker-ce"
+
+# Docker APT repository configuration when docker_legacy_packages is true.
+docker_legacy_apt_url: "{{ 'http://obs.linaro.org/ERP:/17.12/Debian_9' if ansible_architecture == 'aarch64' else 'https://apt.dockerproject.org' }}"
+docker_legacy_apt_repo: "{{ docker_legacy_apt_repo_aarch64 if ansible_architecture == 'aarch64' else docker_legacy_apt_repo_x86_64 }}"
+docker_legacy_apt_repo_x86_64: "deb {{ docker_apt_url }}/repo {{ ansible_distribution | lower }}-{{ ansible_distribution_release | lower }} main"
+docker_legacy_apt_repo_aarch64: "deb {{ docker_apt_url }} ./"
+docker_legacy_apt_key_file: "{{ 'Release.key' if ansible_architecture == 'aarch64' else 'gpg' }}"
+docker_legacy_apt_key_id: "{{ 'C32DA102AD89C2BE' if ansible_architecture == 'aarch64' else 'F76221572C52609D' }}"
+docker_legacy_apt_package: "{{ 'docker-ce' if ansible_architecture == 'aarch64' else 'docker-engine=1.12.*' }}"
+
+# Docker Yum repository configuration.
+docker_yum_url: "{% if docker_legacy_packages | bool %}{{ docker_legacy_yum_url }}{% else %}{{ docker_new_yum_url }}{% endif %}"
+docker_yum_baseurl: "{% if docker_legacy_packages | bool %}{{ docker_legacy_yum_baseurl }}{% else %}{{ docker_new_yum_baseurl }}{% endif %}"
+docker_yum_gpgkey: "{% if docker_legacy_packages | bool %}{{ docker_legacy_yum_gpgkey }}{% else %}{{ docker_new_yum_gpgkey }}{% endif %}"
+docker_yum_gpgcheck: true
+docker_yum_package: "{% if docker_legacy_packages | bool %}{{ docker_legacy_yum_package }}{% else %}{{ docker_new_yum_package }}{% endif %}"
+
+# Docker Yum repository configuration when docker_legacy_packages is false.
+docker_new_yum_url: "https://download.docker.com/linux/{{ ansible_distribution | lower }}"
+docker_new_yum_baseurl: "{{ docker_yum_url }}/{{ ansible_distribution_major_version | lower }}/$basearch/stable"
+docker_new_yum_gpgkey: "{{ docker_yum_url }}/gpg"
+docker_new_yum_package: "docker-ce"
+
+# Docker Yum repository configuration when docker_legacy_packages is true.
+docker_legacy_yum_url: "https://yum.dockerproject.org"
+docker_legacy_yum_baseurl: "{{ docker_legacy_yum_url }}/repo/main/{{ ansible_distribution | lower }}/{{ ansible_distribution_major_version | lower }}"
+docker_legacy_yum_gpgkey: "{{ docker_legacy_yum_url }}/gpg"
+docker_legacy_yum_package: "docker-engine-1.12.0"
 
 customize_etc_hosts: True
 
@@ -27,14 +69,14 @@ docker_custom_option: ""
 docker_runtime_directory: ""
 
 debian_pkg_install:
- - "{{ 'docker-ce' if ansible_architecture == 'aarch64' else 'docker-engine=1.12.*' }}"
+ - "{{ docker_apt_package }}"
  - git
  - python-setuptools
  - ntp
 
 redhat_pkg_install:
  - epel-release
- - docker-engine-1.12.0
+ - "{{ docker_yum_package }}"
  - git
  - python-setuptools
  - ntp

ansible/roles/baremetal/tasks/install.yml

@@ -5,6 +5,12 @@
   become: True
   when: ansible_os_family == 'Debian'
 
+- name: Update yum cache
+  yum:
+    update_cache: yes
+  become: True
+  when: ansible_os_family == 'RedHat'
+
 # TODO(inc0): Gates don't seem to have ufw executable, check for it instead of ignore errors
 - name: Set firewall default policy
   become: True
@@ -32,6 +38,17 @@
     - ansible_os_family == 'RedHat'
     - firewalld_check.rc == 0
 
+# Upgrading docker engine may cause containers to stop. Take a snapshot of the
+# running containers prior to a potential upgrade of Docker.
+
+- name: Check which containers are running
+  command: docker ps -f 'status=running' -q
+  become: true
+  # If Docker is not installed this command may exit non-zero.
+  failed_when: false
+  changed_when: false
+  register: running_containers
+
 - name: Install apt packages
   package:
     name: "{{ item }}"
@@ -39,6 +56,7 @@
   become: True
   with_items: "{{ debian_pkg_install }}"
   when: ansible_os_family == 'Debian'
+  register: apt_install_result
 
 - name: Install deltarpm packages
   package:
@@ -56,6 +74,30 @@
   become: True
   with_items: "{{ redhat_pkg_install }}"
   when: ansible_os_family == 'RedHat'
+  register: yum_install_result
+
+# If any packages were updated, and any containers were running, wait for the
+# daemon to come up and start all previously running containers.
+
+- block:
+    - name: Wait for Docker to start
+      command: docker info
+      become: true
+      changed_when: false
+      register: result
+      until: result is success
+      retries: 6
+      delay: 10
+
+    - name: Ensure containers are running after Docker upgrade
+      command: "docker start {{ running_containers.stdout }}"
+      become: true
+  when:
+    - install_result is changed
+    - running_containers.rc == 0
+    - running_containers.stdout != ''
+  vars:
+    install_result: "{{ yum_install_result if ansible_os_family == 'RedHat' else apt_install_result }}"
 
 - name: Install virtualenv packages
   package:
@@ -94,7 +136,9 @@
     state: absent
   with_items: "{{ ubuntu_pkg_removals }}"
   become: True
-  when: ansible_distribution|lower == "ubuntu"
+  when:
+    - ansible_distribution|lower == "ubuntu"
+    - item != ""
 
 - name: Remove packages
   package:
@@ -102,4 +146,6 @@
     state: absent
   with_items: "{{ redhat_pkg_removals }}"
   become: True
-  when: ansible_os_family == 'RedHat'
+  when:
+    - ansible_os_family == 'RedHat'
+    - item != ""

ansible/roles/baremetal/tasks/pre-install.yml

@@ -47,11 +47,12 @@
   become: True
   when: create_kolla_user | bool
 
+- block:
+    - block:
         - name: Install apt packages
           apt:
             update_cache: yes
           become: True
-  when: ansible_os_family == 'Debian'
 
         - name: Install ca certs
           package:
@@ -61,8 +62,6 @@
           with_items:
             - ca-certificates
             - apt-transport-https
-  when:
-    - ansible_os_family == 'Debian'
 
         - name: Ensure apt sources list directory exists
           file:
@@ -70,14 +69,6 @@
             state: directory
             recurse: yes
           become: True
-  when: ansible_os_family == 'Debian'
-
-- name: Enable docker repo apt
-  template:
-    src: docker_apt_repo.j2
-    dest: /etc/apt/sources.list.d/docker.list
-  become: True
-  when: ansible_os_family == 'Debian'
 
         - name: Install docker apt gpg key
           apt_key:
@@ -85,26 +76,36 @@
             id: "{{ docker_apt_key_id }}"
             state: present
           become: True
+
+        - name: Enable docker apt repository
+          apt_repository:
+            repo: "{{ docker_apt_repo }}"
+            filename: docker
+          become: True
       when: ansible_os_family == 'Debian'
 
+    - block:
         - name: Ensure yum repos directory exists
           file:
             path: /etc/yum.repos.d/
             state: directory
             recurse: yes
           become: True
-  when: ansible_os_family == 'RedHat'
 
-- name: Enable docker repo yum
+        - name: Enable docker yum repository
+          yum_repository:
+            name: docker
+            description: Docker main Repository
+            baseurl: "{{ docker_yum_baseurl }}"
+            gpgcheck: "{{ docker_yum_gpgcheck | bool }}"
+            gpgkey: "{{ docker_yum_gpgkey }}"
           become: True
-  template:
-    src: docker_yum_repo.j2
-    dest: /etc/yum.repos.d/docker.repo
-  when: ansible_os_family == 'RedHat'
 
         - name: Install docker rpm gpg key
           rpm_key:
             state: present
             key: "{{ docker_yum_url }}/gpg"
           become: True
+          when: docker_yum_gpgcheck | bool
       when: ansible_os_family == 'RedHat'
+  when: enable_docker_repo | bool

ansible/roles/baremetal/templates/docker_apt_repo.j2

@@ -1,6 +0,0 @@
-{% if ansible_architecture == 'aarch64' %}
-deb {{ docker_apt_url }} ./
-{% else %}
-# main docker repo
-deb {{ docker_apt_url }}/repo {{ ansible_distribution | lower }}-{{ ansible_distribution_release | lower }} main
-{% endif %}

ansible/roles/baremetal/templates/docker_yum_repo.j2

@@ -1,6 +0,0 @@
-[docker-repo]
-name=Docker main Repository
-baseurl={{ docker_yum_url }}/repo/main/{{ ansible_distribution | lower }}/{{ ansible_distribution_major_version | lower }}
-enabled=1
-gpgcheck=1
-gpgkey={{ docker_yum_url }}/gpg

releasenotes/notes/docker-ce-722582da41cf6cd3.yaml

@@ -0,0 +1,23 @@
+---
+features:
+  - |
+    Adds support for installing Docker Community Edition (CE) using the
+    ``kolla-ansible bootstrap-servers`` command.  Existing support uses the
+    legacy packages from https://dockerproject.org.  New packages are
+    distributed via https://download.docker.com, and that location is now
+    supported and used by default.  Use of the legacy packages is enabled by
+    setting the variable ``docker_legacy_packages`` to ``true``.
+
+    It is also now possible to skip configuration of the Docker repository, by
+    setting the variable ``enable_docker_repo`` to ``false``.
+upgrade:
+  - |
+    The default value for ``docker_legacy_packages`` is ``false``, which means
+    that the Docker Community Edition (CE) should be installed. If the
+    ``kolla-ansible bootstrap-servers`` command is used on a previously
+    deployed host that is running a legacy Docker engine, it would result in
+    the Docker engine being upgraded to use the Docker Community Edition
+    packages, which will result in a restart of the Docker engine and the
+    containers running on that host.  Use the ``kolla-ansible`` ``--serial`` or
+    ``--limit`` arguments to avoid losing quorum in clustered services such as
+    MariaDB by restarting all containers at once.

tests/run.yml

@@ -98,7 +98,7 @@
 
     - name: create deamon.json for nodepool cache
       vars:
-        infra_dockerhub_mirror: "http://{{ zuul_site_mirror_fqdn }}:8081/registry-1.docker/"
+        infra_dockerhub_mirror: "http://{{ zuul_site_mirror_fqdn }}:8082/"
       template:
         src: "{{ kolla_ansible_full_src_dir }}/tests/templates/docker_daemon.json.j2"
         dest: "/etc/docker/daemon.json"