Add build-images role

Add new role to build overcloud and undercloud images.

This role can either operate directly against a host (direct)
or setup a build environment inside of a libvirt guest using
libguestfs-tools (isolated).

It starts by creating the overcloud images from the provided
yaml files. It then uses the convert-image role from
tripleo-quickstart to turn the overcloud-full.qcow2 image into
an undercloud image. Finally, it injects the previously created
overcloud-full and ironic-python-agent images into this new
undercloud image.

It is important to note that this process changes the type of
undercloud image produced. With the previous method we produced
a whole disk image for the undercloud, however by converting an
overcloud image we get a partition image for the undercloud. This
is because the overcloud image itself is a partition image.
The libvirt role in tripleo-quickstart handles this transparently,
and RDO is already producing the master image this way. However,
if we switch older releases to this method, external consumers of
the undercloud image not using tripleo-quickstart for libvirt
provisioning will be broken.

Change-Id: I7df1083291aa36cfef4aaacf9e345300ab5f4ed3

playbooks/build-images-v2.yml

@@ -0,0 +1,31 @@
+---
+
+- name: Add the virthost to the inventory
+  hosts: localhost
+  tasks:
+    - name: Add virthost
+      add_host:
+        name: "{{ virthost }}"
+        groups: "virthost"
+        ansible_fqdn: "{{ virthost }}"
+        ansible_user: "root"
+        ansible_host: "{{ virthost }}"
+  tags:
+    - provision
+
+- name:  Inventory the virthost
+  hosts: localhost
+  gather_facts: yes
+  roles:
+    - tripleo-inventory
+  tags:
+    - provision
+
+- include: teardown-provision.yml
+
+- include: provision.yml
+
+- name: Build images
+  hosts: virthost
+  roles:
+    - build-images

roles/build-images/README.md

@@ -0,0 +1,71 @@
+build-images
+============
+
+An Ansible role for building TripleO undercloud and overcloud images. The role
+can either operate directly against a host (direct) or setup a build
+environment inside of a libvirt guest using libguestfs-tools (isolated).
+
+It starts by creating the overcloud images from the provided yaml files. It
+then uses the convert-image role from tripleo-quickstart to turn the
+overcloud-full.qcow2 image into an undercloud image. Finally, it injects the
+previously created overcloud-full and ironic-python-agent images into this
+new undercloud image.
+
+Requirements
+------------
+
+* [convert-image](https://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/roles/convert-image) role from tripleo-quickstart (if building an undercloud image)
+* [fetch-images](https://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/roles/fetch-images) role from tripleo-quickstart (if using isolated build)
+* [modify-image](https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/modify-image) role from tripleo-quickstart-extras
+* [repo-setup](https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/repo-setup) role from tripleo-quickstart-extras
+
+Role Variables
+--------------
+
+* `images_working_dir` -- Directory on the host where images and logs will be
+   placed
+* `images_destroy_working_dir` -- Whether to destroy the previous image
+   directory before starting. (Default true)
+* `overcloud_image_build_script` -- Template used for the overcloud image build
+* `overcloud_image_build_log` -- Log file for output from the image build
+   script.
+* `build_image_isolated` -- Whether to use libguestfs to create an isolated
+   build environment. (Default true)
+* `build_isolation_image_url` -- URL for image to use as the isolated build
+   environment. (Currently requires an .md5 file in the same location because
+   the fetch-images role from quickstart is used to get the image)
+* `build_undercloud` -- Whether to build an undercloud image. (Default true)
+* `package_images` -- Whether to create tarballs and md5s for all of the
+   produced images. (Default true)
+* `overcloud_repo_paths` -- List of repo paths that will be passed to DIB for
+   package installs in the overcloud images. These repos will also be copied on
+   to the undercloud image.
+* `image_build_yaml_paths` -- List of yaml files to be passed to the overcloud
+   image build. (Defaults to yamls packaged in tripleo-common)
+* `image_build_extract_list` -- List of artifacts to extract from the isolated
+   build environment after building.
+* `inject_images` -- List of artifacts to inject into the undercloud image
+
+Example Usage
+-------------
+
+```yaml
+---
+- name: Build images using an isolated build environment
+  hosts: virthost
+  roles:
+    - build-images
+
+- name: Build images with repos directly installed on the host
+  hosts: virthost
+  vars:
+    build_image_isolated: false
+  roles:
+    - build-images
+
+```
+
+License
+-------
+
+Apache

roles/build-images/defaults/main.yml

@@ -0,0 +1,31 @@
+---
+images_working_dir: /var/lib/oooq-images/{{ release }}
+images_destroy_working_dir: true
+overcloud_image_build_script: overcloud-image-build.sh.j2
+overcloud_image_build_log: overcloud_image_build.log
+build_image_isolated: true
+build_isolation_image_url: https://images.rdoproject.org/CentOS-7-x86_64-GenericCloud.qcow2
+build_isolation_image:
+  name: isolation-image
+  url: "{{ build_isolation_image_url }}"
+  type: qcow2
+build_undercloud: true
+package_images: true
+overcloud_repo_paths:
+  - "$(ls /etc/yum.repos.d/delorean*)"
+  - "$(ls /etc/yum.repos.d/CentOS-Ceph-*)"
+image_build_yaml_paths:
+  - "/usr/share/tripleo-common/image-yaml/overcloud-images-centos7.yaml"
+  - "/usr/share/tripleo-common/image-yaml/overcloud-images.yaml"
+image_build_extract_list:
+  - "/ironic-python-agent.initramfs"
+  - "/ironic-python-agent.kernel"
+  - "/overcloud-full.qcow2"
+  - "/overcloud-full.initrd"
+  - "/overcloud-full.vmlinuz"
+inject_images:
+  - "ironic-python-agent.initramfs"
+  - "ironic-python-agent.kernel"
+  - "overcloud-full.qcow2"
+  - "overcloud-full.initrd"
+  - "overcloud-full.vmlinuz"

roles/build-images/meta/main.yml

@@ -0,0 +1,2 @@
+dependencies:
+  - extras-common

roles/build-images/tasks/convert-image.yml

@@ -0,0 +1,3 @@
+- name: indirect role include (workaround to https://github.com/ansible/ansible/issues/19472)
+  include_role:
+    name: convert-image

roles/build-images/tasks/fetch-images.yml

@@ -0,0 +1,3 @@
+- name: indirect role include (workaround to https://github.com/ansible/ansible/issues/19472)
+  include_role:
+    name: fetch-images

roles/build-images/tasks/main.yml

@@ -0,0 +1,151 @@
+---
+- name: Delete prior working directory if present
+  file:
+    path: "{{ images_working_dir }}"
+    state: absent
+  become: true
+  when: "{{ images_destroy_working_dir }}|bool"
+
+- name: Create the working directory
+  file:
+    path: "{{ images_working_dir }}"
+    state: directory
+    mode: 0755
+    owner: "{{ non_root_user }}"
+  become: true
+
+- name: create the repo setup script for the build host
+  include_role:
+    name: repo-setup
+  vars:
+    repo_run_live: false
+    repo_setup_dir: "{{ images_working_dir }}"
+
+- name: create overcloud/ipa image build script
+  template:
+    src: "{{ overcloud_image_build_script }}"
+    dest: "{{ images_working_dir }}/overcloud_image_build_script.sh"
+    mode: 0755
+  become: true
+
+- when: not build_image_isolated|bool
+  block:
+    - name: run the repo script (direct)
+      shell: "{{ images_working_dir }}/repo_setup.sh"
+
+    - name: run the image build script (direct)
+      shell: >
+        {{ images_working_dir }}/overcloud_image_build_script.sh >
+        {{ overcloud_image_build_log }} 2>&1
+
+- when: build_image_isolated|bool
+  block:
+
+    - name: get image to use as isolated build env
+      include: fetch-images.yml
+      vars:
+        image_fetch_dir: "{{ images_working_dir }}"
+        images:
+          - "{{ build_isolation_image }}"
+
+    - name: run the repo setup script (isolated)
+      include: modify-image.yml
+      vars:
+        modify_image_working_dir: "{{ images_working_dir }}"
+        image_to_modify: "{{ images_working_dir }}/{{ build_isolation_image.name }}.qcow2"
+        modify_script: "{{ images_working_dir }}/repo_setup.sh"
+
+    - name: run the image build script (isolated)
+      include: modify-image.yml
+      vars:
+        modify_image_working_dir: "{{ images_working_dir }}"
+        image_to_modify: "{{ images_working_dir }}/{{ build_isolation_image.name }}.qcow2"
+        modify_script: "{{ images_working_dir }}/overcloud_image_build_script.sh"
+        modify_image_vc_ram: 16384
+        modify_image_vc_cpu: 8
+        modify_image_extract_list: "{{ image_build_extract_list }}"
+
+  rescue:
+    - set_fact: image_build_failed=true
+
+  always:
+    - shell: |
+        virt-cat -a {{ build_isolation_image.name }}.qcow2 \
+            /tmp/builder.log > builder.log 2>&1 || true
+        virt-cat -a {{ build_isolation_image.name }}.qcow2 \
+            /ironic-python-agent.log > ironic-python-agent.log 2>&1 || true
+        virt-cat -a {{ build_isolation_image.name }}.qcow2 \
+            /overcloud-full.log > overcloud-full.log 2>&1 || true
+      environment:
+        LIBGUESTFS_BACKEND: direct
+        LIBVIRT_DEFAULT_URI: "{{ libvirt_uri }}"
+      args:
+        chdir: "{{ images_working_dir }}"
+
+
+# Note(trown) We got a false positive in the RDO job which is already using this
+# because even though the fail was executed it did not actually cause ansible to
+# exit with nonzero. I suspect it is some interaction with block, so moving the
+# actual fail out of the block.
+- fail: msg='*** Image Build Error ***'
+  when: image_build_failed|default(false)|bool
+
+- when: build_undercloud|bool
+  block:
+    - name: run the repo setup script on overcloud image
+      include: modify-image.yml
+      vars:
+        modify_images_working_dir: "{{ images_working_dir }}"
+        image_to_modify: "{{ images_working_dir }}/overcloud-full.qcow2"
+        modify_script: "{{ images_working_dir }}/repo_setup.sh"
+
+    - name: convert the overcloud image to an undercloud image
+      include: convert-image.yml
+      vars:
+        convert_image_working_dir: "{{ images_working_dir }}"
+
+    - name: Inject overcloud/ipa images
+      command: >
+        virt-customize -a {{ images_working_dir }}/undercloud.qcow2
+        --upload {{ images_working_dir }}/{{ item }}:/home/{{ undercloud_user }}/{{ item }}
+        --run-command
+        'chown {{ undercloud_user }}:{{ undercloud_user}} /home/{{ undercloud_user }}/{{ item }}'
+      changed_when: true
+      environment:
+        LIBGUESTFS_BACKEND: direct
+        LIBVIRT_DEFAULT_URI: "{{ libvirt_uri }}"
+      with_items: "{{ inject_images }}"
+
+    - name: Compress the undercloud image
+      shell: |
+        qemu-img convert -c -O qcow2 undercloud.qcow2 undercloud-compressed.qcow2
+        mv undercloud-compressed.qcow2 undercloud.qcow2
+      args:
+        chdir: "{{ images_working_dir }}"
+
+    - name: Create undercloud md5sum
+      shell: md5sum undercloud.qcow2 > undercloud.qcow2.md5
+      args:
+        chdir: "{{ images_working_dir }}"
+
+- when: package_images|bool
+  block:
+    - name: Create overcloud and ironic-python-agent image tars
+      shell: |
+        tar -cf ironic-python-agent.tar \
+                ironic-python-agent.initramfs \
+                ironic-python-agent.kernel
+        tar -cf overcloud-full.tar \
+                overcloud-full.qcow2 \
+                overcloud-full.initrd \
+                overcloud-full.vmlinuz
+      args:
+        chdir: "{{ images_working_dir }}"
+
+    - name: Create md5sums
+      shell: |
+        md5sum ironic-python-agent.tar > ironic-python-agent.tar.md5
+        md5sum overcloud-full.tar > overcloud-full.tar.md5
+      args:
+        chdir: "{{ images_working_dir }}"
+

roles/build-images/tasks/modify-image.yml

@@ -0,0 +1,3 @@
+- name: indirect role include (workaround to https://github.com/ansible/ansible/issues/19472)
+  include_role:
+    name: modify-image

roles/build-images/templates/overcloud-image-build.sh.j2

@@ -0,0 +1,34 @@
+#!/bin/bash
+# script to build overcloud images
+
+set -eux
+
+{% if build_image_isolated %}
+# NOTE(trown): DIB expects /dev/pts to exist and libguestfs is not mounting the guest with it
+# so we just manually mount it in the guest
+# This will be fixed when https://www.redhat.com/archives/libguestfs/2016-December/msg00024.html
+# is available in CentOS version of libguestfs
+mkdir /dev/pts
+mount devpts /dev/pts -t devpts
+{% endif %}
+
+
+sudo yum -y install python-tripleoclient
+
+export DIB_YUM_REPO_CONF=""
+{% for repo_path in overcloud_repo_paths %}
+export DIB_YUM_REPO_CONF="$DIB_YUM_REPO_CONF {{ repo_path }}"
+{% endfor %}
+
+{# note(trown): there was not support in tripleoclient for YAML config images in Newton #}
+{% if release == 'newton'-%}
+tripleo-build-images \
+{% for yaml_path in image_build_yaml_paths -%}
+    --image-config-file {{ yaml_path }} \
+{% endfor -%}
+{% else -%}
+openstack overcloud image build \
+{% for yaml_path in image_build_yaml_paths -%}
+    --config-file {{ yaml_path }} \
+{% endfor -%}
+{% endif -%}