From 828f9279c0d3b7a5fc9377c768c0711a3dac92a2 Mon Sep 17 00:00:00 2001 From: Andreas Florath Date: Sat, 16 Jul 2016 14:52:30 +0200 Subject: [PATCH 01/31] Spec for changing the block device handling: partitioning During the creation of a disk image (e.g. for a VM), there is the need to create, setup, configure and afterwards detach some kind of storage where the newly installed OS can be copied to or directly installed in. This document describes the details of partitioning. Change-Id: I72d8459e81d7da7a8feaebbf295d6c436a686a07 Signed-off-by: Andreas Florath --- .../block-device-lvl1-partitioning.rst | 225 ++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 doc/source/specs/v1/approved/block-device-lvl1-partitioning.rst diff --git a/doc/source/specs/v1/approved/block-device-lvl1-partitioning.rst b/doc/source/specs/v1/approved/block-device-lvl1-partitioning.rst new file mode 100644 index 000000000..ec17e9eb7 --- /dev/null +++ b/doc/source/specs/v1/approved/block-device-lvl1-partitioning.rst @@ -0,0 +1,225 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +======================================== +Block Device Setup Level 1: Partitioning +======================================== + +During the creation of a disk image (e.g. for a VM), there is the need +to create, setup, configure and afterwards detach some kind of storage +where the newly installed OS can be copied to or directly installed +in. + +Remark +------ + +The implementation for this proposed changed already exists, was +discussed and is currently waiting for reviews [1]. To have a +complete overview over the block device setup, this document is +provided. +The dependencies are not implemented as they should be, because +* the spec process is currently in the phase of discussion and not + finalized [2], +* the implementation was finished and reviewed before the spec process + was described. [1] + +Problem description +=================== + +When setting up a block device there is the need to partitioning the +block device. + +Use Cases +--------- + +User (Actor: End User) wants to create multiple partitions in multiple +block devices where the new system is installed in. + +The user wants to specify if the image should be optimized for speed +or for size. + +The user wants the same behavior independently of the current host or +target OS. + +Proposed change +=============== + +Move the partitioning functionality from +`elements/vm/block-device.d/10-partition` to a new block_device +python module: `level1/partitioning.py`. + +Instead of using a program or a library, the data is written directly +with the help of python `file.write()` into the disk image. + +Alternatives +------------ + +The existing implementation uses the `parted` program (old versions of +DIB were using `sfdisk`). The first implementations of this change +used the python-parted library. + +All these approaches have a major drawback: they automatically +*optimize* based on information collected on the host system - and not +of the target system. Therefore the resulting partitioning layout may +lead to a degradation of performance on the target system. A change +in these external programs and libraries also lead to errors during a +DIB run [4] or there are general issues [7]. + +Also everything build around GNU parted falls under the GPL2 (not +LGPL2) license - which is incompatible with the currently used Apache +license in diskimage-builder. + +API impact +---------- + +Extends the (optional) environment variable +``DIB_BLOCK_DEVICE_CONFIG``: a JSON structure to configure the +(complete) block device setup. For this proposal the second entry in +the original list will be used (the first part (as described in [5]) +is used by the level 0 modules). + +The name of this module is `partitioning` (element[0]). The value +(element[1]) is a dictionary. + +For each disk that should be partitioned there exists one entry in the +dictionary. The key is the name of the disk (see [5] how to specify +names for block device level 0). The value is a dictionary that +defines the partitioning of each disk. + +There are the following key / value pairs to define one disk: + +label + (mandatory) Possible values: 'mbr' + This uses the Master Boot Record (MBR) layout for the disk. + (Later on this can be extended, e.g. using GPT). + +align + (optional - default value '1MiB') + Set the alignment of the partition. This must be a multiple of the + block size (i.e. 512 bytes). The default of 1MiB (~ 2048 * 512 + bytes blocks) is the default for modern systems and known to + perform well on a wide range of targets [6]. For each partition + there might be some space that is not used - which is `align` - 512 + bytes. For the default of 1MiB exactly 1048064 bytes (= 1 MiB - + 512 byte) are not used in the partition itself. Please note that + if a boot loader should be written to the disk or partition, + there is a need for some space. E.g. grub needs 63 * 512 byte + blocks between the MBR and the start of the partition data; this + means when grub will be installed, the `align` must be set at least + to 64 * 512 byte = 32 KiB. + +partitions + (mandatory) A list of dictionaries. Each dictionary describes one + partition. + +The following key / value pairs can be given for each partition: + +name + (mandatory) The name of the partition. With the help of this name, + the partition can later be referenced, e.g. while creating a + file system. + +flags + (optional) List of flags for the partition. Default: empty. + Possible values: + + boot + Sets the boot flag for the partition + +size + (mandatory) The size of the partition. The size can either be an + absolute number using units like `10GiB` or `1.75TB` or relative + (percentage) numbers: in the later case the size is calculated + based on the remaining free space. + +Example: + +:: + ["partitioning", + {"rootdisk": { + "label": "mbr", + "partitions": + [{"name": "part-01", + "flags": ["boot"], + "size": "100%"}]}}] + +Security impact +--------------- + +None - functionality stays the same. + +Other end user impact +--------------------- + +None. + +Performance Impact +------------------ + +Measurements showed there is a performance degradation for the target +system of the partition table is not correctly aligned: writing takes +about three times longer on an incorrect aligned system vs. one that +is correctly aligned. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + ansreas (andreas@florath.net) + +Work Items +---------- + +None - this is already a small part of a bigger change [1]. + +Dependencies +============ + +None. + +Testing +======= + +The refactoring introduces no new test cases: the functionality is +tested during each existing test building VM images. + +Documentation Impact +==================== + +End user: the additional environment variable is described. + +References +========== + +[1] Refactor: block-device handling (partitioning) + https://review.openstack.org/322671 + +[2] Add specs dir + https://review.openstack.org/336109 + +[3] Old implementation using parted-lib + https://review.openstack.org/#/c/322671/1..7/elements/block-device/pylib/block-device/level1/Partitioning.py + +[4] ERROR: embedding is not possible, but this is required + for cross-disk install + http://lists.openstack.org/pipermail/openstack-dev/2016-June/097789.html + +[5] Refactor: block-device handling (local loop) + https://review.openstack.org/319591 + +[6] Proper alignment of partitions on an Advanced Format HDD using Parted + http://askubuntu.com/questions/201164/proper-alignment-of-partitions-on-an-advanced-format-hdd-using-parted + +[7] Red Hat Enterprise Linux 6 - Creating a 7TB Partition Using + parted Always Shows "The resulting partition is not properly + aligned for best performance" + http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c03479326&DocLang=en&docLocale=en_US&jumpid=reg_r11944_uken_c-001_title_r0001 + +[8] Spec for changing the block device handling + https://review.openstack.org/336946 From cb3e1076e31319aaea12108eb259bfa129be344f Mon Sep 17 00:00:00 2001 From: Aparna Date: Thu, 10 Nov 2016 11:21:44 +0000 Subject: [PATCH 02/31] Update hpssacli to ssacli in proliant-tools element This commit updates the CLI utility name hpssacli to ssacli as part of company name change from HP to HPE. ssacli will work seamlessly for all the cases where hpssacli is used. It also updates the custom URL location for ssacli utility with new location. Change-Id: I2de4d3430e7308d25e36dc6960783699aa17dfae --- elements/proliant-tools/README.rst | 28 +++++++++++-------- .../install.d/65-proliant-tools-install | 24 +++++++++------- 2 files changed, 30 insertions(+), 22 deletions(-) diff --git a/elements/proliant-tools/README.rst b/elements/proliant-tools/README.rst index 836560b83..f745afc5d 100644 --- a/elements/proliant-tools/README.rst +++ b/elements/proliant-tools/README.rst @@ -3,7 +3,7 @@ proliant-tools * This element can be used when building ironic-agent ramdisk. It enables ironic-agent ramdisk to do in-band cleaning operations specific - to HP ProLiant hardware. + to HPE ProLiant hardware. * Works with ubuntu and fedora distributions (on which ironic-agent element is supported). @@ -11,20 +11,24 @@ proliant-tools * Currently the following utilities are installed: + `proliantutils`_ - This module registers an ironic-python-agent hardware - manager for HP ProLiant hardware, which implements in-band cleaning + manager for HPE ProLiant hardware, which implements in-band cleaning steps. The latest version of ``proliantutils`` available is installed. This python module is released with Apache license. - + `HP Smart Storage Administrator (HP SSA) CLI for Linux 64-bit`_ - This + + `HPE Smart Storage Administrator (HPE SSA) CLI for Linux 64-bit`_ - This utility is used by ``proliantutils`` library above for doing in-band RAID - configuration on HP ProLiant hardware. Currently installed version is - 2.30. Newer version of ``hpssacli`` when available, may be installed to - the ramdisk by using the environment variable ``DIB_HPSSACLI_URL``. - ``DIB_HPSSACLI_URL`` should contain the HTTP(S) URL for downloading the - RPM package for ``hpssacli`` utility. Availability of newer versions can - be in the Revision History in the above link. This utility is closed source - and is released with `HP End User License Agreement – Enterprise Version`_. + configuration on HPE ProLiant hardware. Currently installed version is + 2.60. Newer version of ``ssacli`` when available, may be installed to + the ramdisk by using the environment variable ``DIB_SSACLI_URL``. + ``DIB_SSACLI_URL`` should contain the HTTP(S) URL for downloading the + RPM package for ``ssacli`` utility. The old environmental variable + ``DIB_HPSSACLI_URL``,a HTTP(S) URL for downloading the RPM package for + ``hpssacli`` utility, is deprecated. The ``hpssacli`` utility is not + supported anymore, use ``ssacli`` instead for the same functionality. + Availability of newer versions can be in the Revision History + in the above link. This utility is closed source and is released with + `HPE End User License Agreement – Enterprise Version`_. .. _`proliantutils`: https://pypi.python.org/pypi/proliantutils -.. _`HP Smart Storage Administrator (HP SSA) CLI for Linux 64-bit`: http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_b6a6acb9762443b182280db805 -.. _`HP End User License Agreement – Enterprise Version`: ftp://ftp.hp.com/pub/softlib2/software1/doc/p2057331991/v33194/hpeula-en.html +.. _`HPE Smart Storage Administrator (HPE SSA) CLI for Linux 64-bit`: http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_3d16386b418a443388c18da82f&swEnvOid=4181 +.. _`HPE End User License Agreement – Enterprise Version`: https://downloads.hpe.com/pub/softlib2/software1/doc/p1796552785/v113125/eula-en.html diff --git a/elements/proliant-tools/install.d/65-proliant-tools-install b/elements/proliant-tools/install.d/65-proliant-tools-install index 38a9b9c28..2e3de5d8a 100755 --- a/elements/proliant-tools/install.d/65-proliant-tools-install +++ b/elements/proliant-tools/install.d/65-proliant-tools-install @@ -21,18 +21,22 @@ fi set -eu set -o pipefail -# Set the below variable to allow hpssacli to be installed from custom URLs. -DIB_HPSSACLI_URL=${DIB_HPSSACLI_URL:-https://ftp.hp.com/pub/softlib2/software1/pubsw-linux/p1857046646/v109216/hpssacli-2.30-6.0.x86_64.rpm} -curl -k -o /tmp/hpssacli.rpm $DIB_HPSSACLI_URL - -if [[ $DISTRO_NAME = "ubuntu" || $DISTRO_NAME = "debian" ]]; then - # There is no deb package for hpssacli. Install with alien. - alien -i /tmp/hpssacli.rpm -else - rpm -iv /tmp/hpssacli.rpm +# Set the below variable to allow ssacli to be installed from custom URLs. +if [[ -n "${DIB_HPSSACLI_URL:=}" ]]; then + echo "The environment variable DIB_HPSSACLI_URL is deprecated; use DIB_SSACLI_URL instead." fi -rm -f /tmp/hpssacli.rpm +DIB_SSACLI_URL=${DIB_SSACLI_URL:-${DIB_HPSSACLI_URL:-https://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p1857046646/v123474/ssacli-2.60-19.0.x86_64.rpm}} +curl -k -o /tmp/ssacli.rpm $DIB_SSACLI_URL + +if [[ $DISTRO_NAME = "ubuntu" || $DISTRO_NAME = "debian" ]]; then + # There is no deb package for ssacli. Install with alien. + alien -i /tmp/ssacli.rpm +else + rpm -iv /tmp/ssacli.rpm +fi + +rm -f /tmp/ssacli.rpm # Install proliantutils python module in the # virtual environment of ironic-python-agent. From cd1965f7b7e5d6a7ce31fdb816af00ad93166d5d Mon Sep 17 00:00:00 2001 From: Gregory Haynes Date: Thu, 8 Dec 2016 15:30:56 -0800 Subject: [PATCH 03/31] Allow disto-specific mirror settings Most of our distros support DIB_DISTRIBUTION_MIRROR for setting the mirror to be used when building. This has a major downside where if someone would like to create a static configuration for building various distro images it cannot set DIB_DISTRIBUTION_MIRROR (think of the case where we want to run our tests). By adding support for DIB_$DISTRO_DISTRIBUTION_MIRROR this problem is resolved. Change-Id: I92964b17ec3e47cf97e3a3091f054b2a205ac768 --- elements/centos/environment.d/10-centos6-distro-name.bash | 4 ++++ elements/centos/pre-install.d/02-yum-repos | 4 +++- elements/centos7/environment.d/10-centos7-distro-name.bash | 4 ++++ elements/debian-minimal/environment.d/10-debian-minimal.bash | 5 +++++ elements/fedora/environment.d/10-fedora-distro-name.bash | 3 +++ .../ubuntu-minimal/environment.d/10-ubuntu-distro-name.bash | 4 ++++ 6 files changed, 23 insertions(+), 1 deletion(-) diff --git a/elements/centos/environment.d/10-centos6-distro-name.bash b/elements/centos/environment.d/10-centos6-distro-name.bash index c65178fde..4f38ab2e9 100644 --- a/elements/centos/environment.d/10-centos6-distro-name.bash +++ b/elements/centos/environment.d/10-centos6-distro-name.bash @@ -3,3 +3,7 @@ export DIB_RELEASE=${DIB_RELEASE:-GenericCloud} # Useful for elements that work with fedora (dnf) & centos export YUM=${YUM:-yum} + +if [ -n "${DIB_CENTOS_DISTRIBUTION_MIRROR:-}" ]; then + export DIB_DISTRIBUTION_MIRROR=$DIB_CENTOS_DISTRIBUTION_MIRROR +fi diff --git a/elements/centos/pre-install.d/02-yum-repos b/elements/centos/pre-install.d/02-yum-repos index fb5a0b9b0..099a3eb6e 100755 --- a/elements/centos/pre-install.d/02-yum-repos +++ b/elements/centos/pre-install.d/02-yum-repos @@ -6,10 +6,12 @@ fi set -eu set -o pipefail +DIB_DISTRIBUTION_MIRROR=${DIB_DISTRIBUTION_MIRROR:-http://mirror.centos.org/centos} + cat << EOF > /etc/yum.repos.d/centos6-latest.repo [rhel6] name=centos6 -baseurl=http://mirror.centos.org/centos/6/os/x86_64/ +baseurl=$DIB_DISTRIBUTION_MIRROR/6/os/x86_64/ enabled=1 metadata_expire=7d gpgcheck=0 diff --git a/elements/centos7/environment.d/10-centos7-distro-name.bash b/elements/centos7/environment.d/10-centos7-distro-name.bash index c9476d071..c0c42a51c 100644 --- a/elements/centos7/environment.d/10-centos7-distro-name.bash +++ b/elements/centos7/environment.d/10-centos7-distro-name.bash @@ -3,3 +3,7 @@ export DIB_RELEASE=GenericCloud # Useful for elements that work with fedora (dnf) & centos export YUM=${YUM:-yum} + +if [ -n "${DIB_CENTOS_DISTRIBUTION_MIRROR:-}" ]; then + export DIB_DISTRIBUTION_MIRROR=$DIB_CENTOS_DISTRIBUTION_MIRROR +fi diff --git a/elements/debian-minimal/environment.d/10-debian-minimal.bash b/elements/debian-minimal/environment.d/10-debian-minimal.bash index 13ac22fc0..26f515990 100644 --- a/elements/debian-minimal/environment.d/10-debian-minimal.bash +++ b/elements/debian-minimal/environment.d/10-debian-minimal.bash @@ -1,6 +1,11 @@ export DISTRO_NAME=debian export DIB_RELEASE=${DIB_RELEASE:-stable} + +if [ -n "${DIB_DEBIAN_DISTRIBUTION_MIRROR:-}" ]; then + DIB_DISTRIBUTION_MIRROR=$DIB_DEBIAN_DISTRIBUTION_MIRROR +fi export DIB_DISTRIBUTION_MIRROR=${DIB_DISTRIBUTION_MIRROR:-http://ftp.us.debian.org/debian} + export DIB_DEBIAN_COMPONENTS=${DIB_DEBIAN_COMPONENTS:-main} export DIB_DEBIAN_COMPONENTS_WS=${DIB_DEBIAN_COMPONENTS//,/ } diff --git a/elements/fedora/environment.d/10-fedora-distro-name.bash b/elements/fedora/environment.d/10-fedora-distro-name.bash index 3477f1eac..07b4e60e4 100644 --- a/elements/fedora/environment.d/10-fedora-distro-name.bash +++ b/elements/fedora/environment.d/10-fedora-distro-name.bash @@ -1,2 +1,5 @@ export DISTRO_NAME=fedora export DIB_RELEASE=${DIB_RELEASE:-24} +if [ -n "${DIB_FEDORA_DISTRIBUTION_MIRROR:-}" ]; then + export DIB_DISTRIBUTION_MIRROR=$DIB_FEDORA_DISTRIBUTION_MIRROR +fi diff --git a/elements/ubuntu-minimal/environment.d/10-ubuntu-distro-name.bash b/elements/ubuntu-minimal/environment.d/10-ubuntu-distro-name.bash index e9ae083e3..829952b31 100644 --- a/elements/ubuntu-minimal/environment.d/10-ubuntu-distro-name.bash +++ b/elements/ubuntu-minimal/environment.d/10-ubuntu-distro-name.bash @@ -1,4 +1,8 @@ export DISTRO_NAME=ubuntu export DIB_RELEASE=${DIB_RELEASE:-xenial} export DIB_DEBIAN_COMPONENTS=${DIB_DEBIAN_COMPONENTS:-main,restricted,universe} + +if [ -n "${DIB_UBUNTU_DISTRIBUTION_MIRROR:-}" ]; then + DIB_DISTRIBUTION_MIRROR=$DIB_UBUNTU_DISTRIBUTION_MIRROR +fi export DIB_DISTRIBUTION_MIRROR=${DIB_DISTRIBUTION_MIRROR:-http://archive.ubuntu.com/ubuntu} From 8e3e66dfbd139f0180b40d73e1f65ff237a280d1 Mon Sep 17 00:00:00 2001 From: Ian Wienand Date: Tue, 13 Dec 2016 12:47:46 +1100 Subject: [PATCH 04/31] Speed up chroot checking loop It was noticed on a very busy system this can take about 1s per loop. This starts to add up on thousands of processes. Firstly, prune out all the kernel threads. Then introduce a very small inline python script to find any pids that seem to be in the chroot without forking to examine each one. After that the existing loop just kills anything as before. Change-Id: Icc7bc7eda80ffcd636f97e6542d70c220e9c225e --- lib/common-functions | 36 ++++++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 8 deletions(-) diff --git a/lib/common-functions b/lib/common-functions index ff0c74679..653aee4fd 100644 --- a/lib/common-functions +++ b/lib/common-functions @@ -135,24 +135,44 @@ function eval_run_d () { trap - ERR } +# Get any process that appears to be running in $TMP_BUILD_DIR +function _get_chroot_processes () { + # Deselect kernel threads, and use a python script to avoid + # forking lots and lots of readlink / grep processes on a busy + # system. + ps --ppid 2 -p 2 --deselect -o pid= | xargs python -c ' +import os +import sys + +for pid in sys.argv[2:]: + try: + root = os.readlink("/proc/%s/root" % pid) + except: + continue + if sys.argv[1] in root: + print("%s" % pid) +' $TMP_BUILD_DIR +} + + function kill_chroot_processes () { local xtrace xtrace=$(set +o | grep xtrace) set +o xtrace + local pidname + if [ -z "${1}" ]; then echo "ERROR: no chroot directory specified" exit 1 fi - for piddir in /proc/[0-9]*; do - pid=${piddir##/proc/} - pidname=$(cat $piddir/comm 2>/dev/null || echo "unknown") + + for pid in $(_get_chroot_processes); do # If there are open files from the chroot, just kill the process using - # these files. - if sudo readlink -f $piddir/root | grep -q $TMP_BUILD_DIR; then - echo "Killing chroot process: '${pidname}($pid)'" - sudo kill $pid - fi + # these files. This is racy, but good enough + pidname=$(cat $piddir/comm 2>/dev/null || echo "unknown") + echo "Killing chroot process: '${pidname}($pid)'" + sudo kill $pid done $xtrace From 6a40b607176857dc599ce9ca69da12b9bc1c6560 Mon Sep 17 00:00:00 2001 From: Gregory Haynes Date: Tue, 6 Dec 2016 16:08:28 -0800 Subject: [PATCH 05/31] Create ubuntu/fedora test for pip-and-virtualenv This element does some funky stuff WRT python2 vs python3 so lets get some multi-distro testing in place. Change-Id: I1e3c3bfa0a109419d4cbee7fa32a18392b7e1a93 --- .../test-elements/source-install-fedora/element-deps | 1 + .../environment.d/10-source-installtype-pip-and-virtualenv | 1 + .../test-elements/source-install-ubuntu/element-deps | 1 + .../environment.d/10-source-installtype-pip-and-virtualenv | 1 + 4 files changed, 4 insertions(+) create mode 100644 elements/pip-and-virtualenv/test-elements/source-install-fedora/element-deps create mode 100644 elements/pip-and-virtualenv/test-elements/source-install-fedora/environment.d/10-source-installtype-pip-and-virtualenv create mode 100644 elements/pip-and-virtualenv/test-elements/source-install-ubuntu/element-deps create mode 100644 elements/pip-and-virtualenv/test-elements/source-install-ubuntu/environment.d/10-source-installtype-pip-and-virtualenv diff --git a/elements/pip-and-virtualenv/test-elements/source-install-fedora/element-deps b/elements/pip-and-virtualenv/test-elements/source-install-fedora/element-deps new file mode 100644 index 000000000..b767efbb8 --- /dev/null +++ b/elements/pip-and-virtualenv/test-elements/source-install-fedora/element-deps @@ -0,0 +1 @@ +fedora-minimal diff --git a/elements/pip-and-virtualenv/test-elements/source-install-fedora/environment.d/10-source-installtype-pip-and-virtualenv b/elements/pip-and-virtualenv/test-elements/source-install-fedora/environment.d/10-source-installtype-pip-and-virtualenv new file mode 100644 index 000000000..0f27f00cc --- /dev/null +++ b/elements/pip-and-virtualenv/test-elements/source-install-fedora/environment.d/10-source-installtype-pip-and-virtualenv @@ -0,0 +1 @@ +export DIB_INSTALLTYPE_pip_and_virtualenv=source diff --git a/elements/pip-and-virtualenv/test-elements/source-install-ubuntu/element-deps b/elements/pip-and-virtualenv/test-elements/source-install-ubuntu/element-deps new file mode 100644 index 000000000..fe735d046 --- /dev/null +++ b/elements/pip-and-virtualenv/test-elements/source-install-ubuntu/element-deps @@ -0,0 +1 @@ +ubuntu-minimal diff --git a/elements/pip-and-virtualenv/test-elements/source-install-ubuntu/environment.d/10-source-installtype-pip-and-virtualenv b/elements/pip-and-virtualenv/test-elements/source-install-ubuntu/environment.d/10-source-installtype-pip-and-virtualenv new file mode 100644 index 000000000..0f27f00cc --- /dev/null +++ b/elements/pip-and-virtualenv/test-elements/source-install-ubuntu/environment.d/10-source-installtype-pip-and-virtualenv @@ -0,0 +1 @@ +export DIB_INSTALLTYPE_pip_and_virtualenv=source From 4585955a8b82889c61deb9ecb34b8713270406a7 Mon Sep 17 00:00:00 2001 From: Ian Wienand Date: Fri, 9 Dec 2016 14:41:24 +1100 Subject: [PATCH 06/31] Remove yum chroot caching Every run we are doing a full tar.gz of the chroot environment that never gets used. It's not suitable for CI since we use fresh images each time there. The cache in general isn't really isn't a very safe thing to have around, because there's no invalidation procedure and no real way to make one -- we've no guarantee that a new chroot build even moments after a previous one wouldn't bring in or different packages, etc (of course this is *unlikely*, but the longer you go between builds the worse the problem becomes. Also, tons of packages get installed after this not from any cache, so potential speed-up is rather marginal. Debian turned this off with I58fc485aacacaa17243bf9ce760ed91256d1f182. However, given the reasons above and it's complete lack of testing, I don't see this as useful. If we really want this type of thing, I think we should come up with a way to use a persistent external yum/dnf cache that yum/dnf keeps in sync with it's usual invalidation rules. Change-Id: I66789c35db75c41bc45ea1ad2e26f87456de4e4d --- elements/centos-minimal/README.rst | 12 +- elements/fedora-minimal/README.rst | 10 +- elements/yum-minimal/README.rst | 4 - elements/yum-minimal/root.d/08-yum-chroot | 192 ++++++++---------- .../yum-cache-removal-148c33012515e56e.yaml | 9 + 5 files changed, 106 insertions(+), 121 deletions(-) create mode 100644 releasenotes/notes/yum-cache-removal-148c33012515e56e.yaml diff --git a/elements/centos-minimal/README.rst b/elements/centos-minimal/README.rst index 3c83eef30..f43b9d919 100644 --- a/elements/centos-minimal/README.rst +++ b/elements/centos-minimal/README.rst @@ -6,13 +6,9 @@ Create a minimal image based on CentOS 7. Use of this element will require 'yum' and 'yum-utils' to be installed on Ubuntu and Debian. Nothing additional is needed on Fedora or CentOS. -The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE` -variables can be set to prefer the use of a pre-cached root filesystem -tarball. - -By default, `DIB_YUM_MINIMAL_CREATE_INTERFACES` is set to enable the -creation of `/etc/sysconfig/network-scripts/ifcfg-eth[0|1]` scripts to -enable DHCP on the `eth0` & `eth1` interfaces. If you do not have +By default, ``DIB_YUM_MINIMAL_CREATE_INTERFACES`` is set to enable the +creation of ``/etc/sysconfig/network-scripts/ifcfg-eth[0|1]`` scripts to +enable DHCP on the ``eth0`` & ``eth1`` interfaces. If you do not have these interfaces, or if you are using something else to setup the network such as cloud-init, glean or network-manager, you would want -to set this to `0`. +to set this to ``0``. diff --git a/elements/fedora-minimal/README.rst b/elements/fedora-minimal/README.rst index 88c88ecd3..f1c77370a 100644 --- a/elements/fedora-minimal/README.rst +++ b/elements/fedora-minimal/README.rst @@ -11,10 +11,6 @@ Due to a bug in the released version of urlgrabber, on many systems an installation of urlgrabber from git is required. The git repository can be found here: http://yum.baseurl.org/gitweb?p=urlgrabber.git;a=summary -The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE` -variables can be set to prefer the use of a pre-cached root filesystem -tarball. - -This element sets the `DIB_RELEASE` var to 'fedora'. The release of fedora -to be installed can be controlled through the `DIB_RELEASE` variable, which -defaults to '21'. +This element sets the ``DIB_RELEASE`` var to 'fedora'. The release of +fedora to be installed can be controlled through the ``DIB_RELEASE`` +variable, which defaults the latest supported release. diff --git a/elements/yum-minimal/README.rst b/elements/yum-minimal/README.rst index ebe6be3cf..e68d704d0 100644 --- a/elements/yum-minimal/README.rst +++ b/elements/yum-minimal/README.rst @@ -9,10 +9,6 @@ or fedora-minimal elements to get an actual base image. Use of this element will require 'yum' and 'yum-utils' to be installed on Ubuntu and Debian. Nothing additional is needed on Fedora or CentOS. -The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE` -variables can be set to prefer the use of a pre-cached root filesystem -tarball. - If you wish to have DHCP networking setup for eth0 & eth1 via /etc/sysconfig/network-config scripts/ifcfg-eth[0|1], set the environment variable `DIB_YUM_MINIMAL_CREATE_INTERFACES` to `1`. diff --git a/elements/yum-minimal/root.d/08-yum-chroot b/elements/yum-minimal/root.d/08-yum-chroot index 7760897b6..ee96e552b 100755 --- a/elements/yum-minimal/root.d/08-yum-chroot +++ b/elements/yum-minimal/root.d/08-yum-chroot @@ -31,8 +31,6 @@ if [ $ARCH = amd64 ]; then ARCH=x86_64 fi # Calling elements will need to set DISTRO_NAME and DIB_RELEASE -DIB_YUMCHROOT_EXTRA_ARGS=${DIB_YUMCHROOT_EXTRA_ARGS:-} -YUMCHROOT_TARBALL=$DIB_IMAGE_CACHE/yumchroot-${DISTRO_NAME}-${DIB_RELEASE}-${ARCH}.tar.gz # TODO Maybe deal with DIB_DISTRIBUTION_MIRROR http_proxy=${http_proxy:-} YUM=${YUM:-yum} @@ -189,107 +187,97 @@ function _install_pkg_manager { $TARGET_ROOT/etc/yum.repos.d/*repo } -if [ -n "$DIB_OFFLINE" -o -n "${DIB_YUMCHROOT_USE_CACHE:-}" ] && [ -f $YUMCHROOT_TARBALL ] ; then - echo $YUMCHROOT_TARBALL found in cache. Using. - sudo tar -C $TARGET_ROOT --numeric-owner -xzf $YUMCHROOT_TARBALL +# Note this is not usually done for root.d elements (see +# lib/common-functions:mount_proc_dev_sys) but it's important that +# we have things like /dev/urandom around inside the chroot for +# the rpm [pre|post]inst scripts within the packages. +sudo mkdir -p $TARGET_ROOT/proc $TARGET_ROOT/dev $TARGET_ROOT/sys +sudo mount -t proc none $TARGET_ROOT/proc +sudo mount --bind /dev $TARGET_ROOT/dev +sudo mount --bind /dev/pts $TARGET_ROOT/dev/pts +sudo mount -t sysfs none $TARGET_ROOT/sys + +# initalize rpmdb +sudo mkdir -p $TARGET_ROOT/var/lib/rpm +sudo $_RPM --root $TARGET_ROOT --initdb + +# this makes sure that running yum/dnf in the chroot it can get +# out to download stuff +sudo mkdir $TARGET_ROOT/etc +sudo cp /etc/resolv.conf $TARGET_ROOT/etc/resolv.conf + +# Bind mount the external yum cache inside the chroot. Same logic +# as in the yum element to provide for yum caching copied here +# because the sequencing is wrong otherwise +sudo mkdir -p $TMP_MOUNT_PATH/tmp/yum +sudo mount --bind $YUM_CACHE $TMP_MOUNT_PATH/tmp/yum + +_install_repos + +# install dnf for >= f22 +if [ $DIB_RELEASE -ge 22 ]; then + _install_pkg_manager dnf dnf-plugins-core yum else - # Note this is not usually done for root.d elements (see - # lib/common-functions:mount_proc_dev_sys) but it's important that - # we have things like /dev/urandom around inside the chroot for - # the rpm [pre|post]inst scripts within the packages. - sudo mkdir -p $TARGET_ROOT/proc $TARGET_ROOT/dev $TARGET_ROOT/sys - sudo mount -t proc none $TARGET_ROOT/proc - sudo mount --bind /dev $TARGET_ROOT/dev - sudo mount --bind /dev/pts $TARGET_ROOT/dev/pts - sudo mount -t sysfs none $TARGET_ROOT/sys - - # initalize rpmdb - sudo mkdir -p $TARGET_ROOT/var/lib/rpm - sudo $_RPM --root $TARGET_ROOT --initdb - - # this makes sure that running yum/dnf in the chroot it can get - # out to download stuff - sudo mkdir $TARGET_ROOT/etc - sudo cp /etc/resolv.conf $TARGET_ROOT/etc/resolv.conf - - # Bind mount the external yum cache inside the chroot. Same logic - # as in the yum element to provide for yum caching copied here - # because the sequencing is wrong otherwise - sudo mkdir -p $TMP_MOUNT_PATH/tmp/yum - sudo mount --bind $YUM_CACHE $TMP_MOUNT_PATH/tmp/yum - - _install_repos - - # install dnf for >= f22 - if [ $DIB_RELEASE -ge 22 ]; then - _install_pkg_manager dnf dnf-plugins-core yum - else - _install_pkg_manager yum - fi - - # we just installed yum/dnf with "outside" tools (yum/rpm) which - # might have created /var/lib/[yum|rpm] (etc) that are slighlty - # incompatible. Refresh everything with the in-chroot tools - sudo -E chroot $TARGET_ROOT rpm --rebuilddb - sudo -E chroot $TARGET_ROOT ${YUM} clean all - - # populate the lang reduction macro in the chroot - echo "%_install_langs C:en_US:en_US.UTF-8" | \ - sudo tee -a $TARGET_ROOT/etc/rpm/macros.langs > /dev/null - - # bootstrap the environment within the chroot; bring in new - # metadata with an update and install some base packages we need. - sudo -E chroot $TARGET_ROOT ${YUM} -y update - sudo -E chroot $TARGET_ROOT ${YUM} -y \ - --setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \ - install systemd passwd findutils sudo util-linux-ng - - # This package is split out from systemd on >F24, dracut is - # missing the dependency and will fail to make an initrd without - # it; see - # https://bugzilla.redhat.com/show_bug.cgi?id=1398505 - if [ $DISTRO_NAME = "fedora" -a $DIB_RELEASE -ge 24 ]; then - sudo -E chroot $TARGET_ROOT ${YUM} -y \ - --setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \ - install systemd-udev - fi - - # Put in a dummy /etc/resolv.conf over the temporary one we used - # to bootstrap. systemd has a bug/feature [1] that it will assume - # you want systemd-networkd as the network manager and create a - # broken symlink to /run/... if the base image doesn't have one. - # This broken link confuses things like dhclient. - # [1] https://bugzilla.redhat.com/show_bug.cgi?id=1197204 - echo -e "# This file intentionally left blank\n" | \ - sudo tee $TARGET_ROOT/etc/resolv.conf - - # set the most reliable UTF-8 locale - echo -e 'LANG="en_US.UTF-8"' | \ - sudo tee $TARGET_ROOT/etc/locale.conf - # default to UTC - sudo -E chroot $TARGET_ROOT ln -sf /usr/share/zoneinfo/UTC \ - /etc/localtime - - # cleanup - # TODO : move this into a exit trap; and reconsider how - # this integrates with the global exit cleanup path. - sudo umount $TMP_MOUNT_PATH/tmp/yum - sudo umount $TARGET_ROOT/proc - sudo umount $TARGET_ROOT/dev/pts - sudo umount $TARGET_ROOT/dev - sudo umount $TARGET_ROOT/sys - - # RPM doesn't know whether files have been changed since install - # At this point though, we know for certain that we have changed no - # config files, so anything marked .rpmnew is just a bug. - for newfile in $(sudo find $TARGET_ROOT -type f -name '*rpmnew') ; do - sudo mv $newfile $(echo $newfile | sed 's/.rpmnew$//') - done - - echo Caching result in $YUMCHROOT_TARBALL - sudo tar --numeric-owner \ - -C $TARGET_ROOT \ - -zcf $YUMCHROOT_TARBALL --exclude='./tmp/*' . + _install_pkg_manager yum fi +# we just installed yum/dnf with "outside" tools (yum/rpm) which +# might have created /var/lib/[yum|rpm] (etc) that are slighlty +# incompatible. Refresh everything with the in-chroot tools +sudo -E chroot $TARGET_ROOT rpm --rebuilddb +sudo -E chroot $TARGET_ROOT ${YUM} clean all + +# populate the lang reduction macro in the chroot +echo "%_install_langs C:en_US:en_US.UTF-8" | \ + sudo tee -a $TARGET_ROOT/etc/rpm/macros.langs > /dev/null + +# bootstrap the environment within the chroot; bring in new +# metadata with an update and install some base packages we need. +sudo -E chroot $TARGET_ROOT ${YUM} -y update +sudo -E chroot $TARGET_ROOT ${YUM} -y \ + --setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \ + install systemd passwd findutils sudo util-linux-ng + +# This package is split out from systemd on >F24, dracut is +# missing the dependency and will fail to make an initrd without +# it; see +# https://bugzilla.redhat.com/show_bug.cgi?id=1398505 +if [ $DISTRO_NAME = "fedora" -a $DIB_RELEASE -ge 24 ]; then + sudo -E chroot $TARGET_ROOT ${YUM} -y \ + --setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \ + install systemd-udev +fi + +# Put in a dummy /etc/resolv.conf over the temporary one we used +# to bootstrap. systemd has a bug/feature [1] that it will assume +# you want systemd-networkd as the network manager and create a +# broken symlink to /run/... if the base image doesn't have one. +# This broken link confuses things like dhclient. +# [1] https://bugzilla.redhat.com/show_bug.cgi?id=1197204 +echo -e "# This file intentionally left blank\n" | \ + sudo tee $TARGET_ROOT/etc/resolv.conf + +# set the most reliable UTF-8 locale +echo -e 'LANG="en_US.UTF-8"' | \ + sudo tee $TARGET_ROOT/etc/locale.conf + # default to UTC +sudo -E chroot $TARGET_ROOT ln -sf /usr/share/zoneinfo/UTC \ + /etc/localtime + +# cleanup +# TODO : move this into a exit trap; and reconsider how +# this integrates with the global exit cleanup path. +sudo umount $TMP_MOUNT_PATH/tmp/yum +sudo umount $TARGET_ROOT/proc +sudo umount $TARGET_ROOT/dev/pts +sudo umount $TARGET_ROOT/dev +sudo umount $TARGET_ROOT/sys + +# RPM doesn't know whether files have been changed since install +# At this point though, we know for certain that we have changed no +# config files, so anything marked .rpmnew is just a bug. +for newfile in $(sudo find $TARGET_ROOT -type f -name '*rpmnew') ; do + sudo mv $newfile $(echo $newfile | sed 's/.rpmnew$//') +done + sudo rm -f ${TARGET_ROOT}/.extra_settings diff --git a/releasenotes/notes/yum-cache-removal-148c33012515e56e.yaml b/releasenotes/notes/yum-cache-removal-148c33012515e56e.yaml new file mode 100644 index 000000000..86727a081 --- /dev/null +++ b/releasenotes/notes/yum-cache-removal-148c33012515e56e.yaml @@ -0,0 +1,9 @@ +--- +deprecations: + - The ``DIB_YUMCHROOT_USE_CACHE`` variable has been removed and the + Fedora and CentOS ``-minimal`` initial chroot will always be + created by the package manager. The default creation of a chroot + tarball is stopped for these elements. This unused option was + unsafe; there is no guarantee that the base system will not change + even between runs. Getting the package manager to reuse the cache + for the initial chroot install is future work. From 2209d343720f994a71d9451c4b9297af969f7bf5 Mon Sep 17 00:00:00 2001 From: Antoine Musso Date: Tue, 6 Dec 2016 15:51:12 +0100 Subject: [PATCH 07/31] debootstrap: avoid duplicate network configuration On Debian network configuration can be done via /etc/network/interfaces. It can accept a statement to load additional files, which varied in history: Wheezy only supports 'source' (see b822581) Jessie supports 'source-directory' and comes with the statement by default. However since 754dd05 we inconditionally inject 'source', thus on Jessie the configuration ends up with: source-directory /etc/network/interfaces.d/* source /etc/network/interfaces.d/* When networking is started, 'ifup -a' parses the list of interfaces twice. When configured with dhcp, that causes two dhclient to spawn which might conflict with each other. Inject the source statement only if there is neither a source or source-directory with the same path. Change-Id: Iefa9c9584f676e50481c621b4111eded3125a50b --- elements/debootstrap/install.d/10-debian-networking | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/elements/debootstrap/install.d/10-debian-networking b/elements/debootstrap/install.d/10-debian-networking index 7e85cbe57..d5cdc1400 100755 --- a/elements/debootstrap/install.d/10-debian-networking +++ b/elements/debootstrap/install.d/10-debian-networking @@ -26,7 +26,10 @@ echo $DISTRO_NAME > /etc/hostname # cloud images expect eth0 and eth1 to use dhcp. mkdir -p /etc/network/interfaces.d -echo "source /etc/network/interfaces.d/*" >> /etc/network/interfaces +if ! grep -E -q '^source(|-directory) /etc/network/interfaces.d/\*' /etc/network/interfaces; then + echo "source /etc/network/interfaces.d/*" >> /etc/network/interfaces + echo 'Network configuration set to source /etc/network/interfaces.d/*' +fi for interface in eth0 eth1; do cat << EOF | tee /etc/network/interfaces.d/$interface auto $interface From 0576d20d49fbbf07d30adc05caca3d749550c06d Mon Sep 17 00:00:00 2001 From: Pierre Riteau Date: Wed, 21 Dec 2016 11:49:27 +0000 Subject: [PATCH 08/31] Update documented default Ubuntu version Since commit fd5fbdd4b5c15c613133c5a18c8e83b486ec1dd0 xenial is the version used by default. Change-Id: I18e01d806635539b2d6c8a4e6b2d25460647c910 --- elements/ubuntu-minimal/README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/elements/ubuntu-minimal/README.rst b/elements/ubuntu-minimal/README.rst index 23764a45b..49a5e9380 100644 --- a/elements/ubuntu-minimal/README.rst +++ b/elements/ubuntu-minimal/README.rst @@ -6,7 +6,7 @@ Note: The ubuntu element is likely what you want unless you really know you want this one for some reason. The ubuntu element gets a lot more testing coverage and use. -Create a minimal image based on Ubuntu. We default to trusty but DIB_RELEASE +Create a minimal image based on Ubuntu. We default to xenial but DIB_RELEASE is mapped to any series of Ubuntu. If necessary, a custom apt keyring and debootstrap script can be From f6ecfb3b24448c38d1dea1379077bc804a2987fc Mon Sep 17 00:00:00 2001 From: xhzhf Date: Wed, 21 Dec 2016 22:10:50 +0800 Subject: [PATCH 09/31] H803 hacking have been deprecated H803 hacking have been removed. https://github.com/openstack-dev/hacking/blob/master/setup.cfg Closes-Bug: #1650741 Change-Id: Ic88b431d9260c6e31717b001ff657d10d76b9213 --- tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index 16a7cf9ed..4bd24a3f2 100644 --- a/tox.ini +++ b/tox.ini @@ -35,5 +35,5 @@ commands = python setup.py build_sphinx commands = sphinx-build -a -W -E -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html [flake8] -ignore = E125,H202,H803 +ignore = E125,H202 exclude = .venv,.git,.tox,dist,doc,*lib/python*,*egg,build,conf.py From 7337c9e706985148c988626814830a4768d1f6ba Mon Sep 17 00:00:00 2001 From: Andreas Florath Date: Wed, 21 Dec 2016 20:18:12 +0000 Subject: [PATCH 10/31] dib-lint: python3 compatibility fixes When using up to date distributions for dib development, pep8 installs using python3. This patch fixes the problem, that not the complete dib-lint (which is called) is compatible with python3. Change-Id: I417d03746edb4d34011b997edf8b5b9662ea6f09 Signed-off-by: Andreas Florath --- bin/dib-lint | 14 +++++++++----- .../pypi/pre-install.d/00-configure-pypi-mirror | 4 ++-- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/bin/dib-lint b/bin/dib-lint index 0b33790e8..d7e602f3b 100755 --- a/bin/dib-lint +++ b/bin/dib-lint @@ -34,11 +34,15 @@ parse_exclusions() { # ignore = sete setu section="dib-lint" option="ignore" - global_exclusions=$(python -c \ - "import ConfigParser; \ - conf=ConfigParser.ConfigParser(); \ - conf.read('tox.ini'); \ - print conf.get('$section', '$option') if conf.has_option('$section', '$option') else ''" + global_exclusions=$(python - < Date: Fri, 23 Dec 2016 20:02:38 -0600 Subject: [PATCH 11/31] update pkg-map entries for python3 Map python3 packages properly, or else we get invalid package names. Change-Id: I7af8621c856c689840a11d3cb9d094f02a0b9593 --- elements/pip-and-virtualenv/pkg-map | 3 +++ 1 file changed, 3 insertions(+) diff --git a/elements/pip-and-virtualenv/pkg-map b/elements/pip-and-virtualenv/pkg-map index 09e1529f0..64619ca94 100644 --- a/elements/pip-and-virtualenv/pkg-map +++ b/elements/pip-and-virtualenv/pkg-map @@ -2,7 +2,10 @@ "family": { "gentoo": { "python-pip": "dev-python/pip", + "python3-pip": "dev-python/pip", "python-virtualenv": "dev-python/virtualenv", + "python3-virtualenv": "dev-python/virtualenv", + "python-dev": "dev-lang/python", "python3-dev": "dev-lang/python" }, "suse": { From 1d4bb048531c8fc77f91c35f5b1ca5756fd52e15 Mon Sep 17 00:00:00 2001 From: Cady_Chen Date: Thu, 22 Dec 2016 22:52:17 +0800 Subject: [PATCH 12/31] Change "Openstack" to "OpenStack" According to the word choice convention in http://docs.openstack.org/contributor-guide/writing-style/word-choice.html We should use OpenStack instead of Openstack. Change-Id: I66f0bf9eb81593220eb0000fe8192c478e8d075d --- .../ironic-agent-source-install/ironic-python-agent.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/elements/ironic-agent/install.d/ironic-agent-source-install/ironic-python-agent.conf b/elements/ironic-agent/install.d/ironic-agent-source-install/ironic-python-agent.conf index cfd134583..6fe38dc9d 100644 --- a/elements/ironic-agent/install.d/ironic-agent-source-install/ironic-python-agent.conf +++ b/elements/ironic-agent/install.d/ironic-agent-source-install/ironic-python-agent.conf @@ -1,4 +1,4 @@ -# ironic-python-agent - Openstack Ironic Python Agnet +# ironic-python-agent - OpenStack Ironic Python Agent # # The ironic-python-agent helps ironic in deploying instances. From 197ab47226e0ecfed3e9ffdb0ff0db1ea0d7cb00 Mon Sep 17 00:00:00 2001 From: Julia Kreger Date: Tue, 3 Jan 2017 23:34:13 +0000 Subject: [PATCH 13/31] Update component docs to refer to ironic-agent The components documentation was previously referring to the ramdisk image for deployment, which was previously deprecated. Corrected to point to the ironic-agent element. Change-Id: I770460041eb13523896aaadb7705bdc3db1a54ca --- doc/source/developer/components.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/developer/components.rst b/doc/source/developer/components.rst index 35c9c7225..8720623d4 100644 --- a/doc/source/developer/components.rst +++ b/doc/source/developer/components.rst @@ -22,7 +22,7 @@ Components To generate kernel+ramdisk pair for use with ironic, use:: - ramdisk-image-create -o deploy.ramdisk deploy-ironic + ramdisk-image-create -o deploy.ramdisk ironic-agent `element-info` From 5bed4a6d5e0e037a41931a840c4399e844631cfb Mon Sep 17 00:00:00 2001 From: Ben Nemec Date: Wed, 4 Jan 2017 10:49:59 -0600 Subject: [PATCH 14/31] Run dhcp-interface@.service after network.target When we configure dhcp interfaces before network.target has run, network.target will try to bring up those interfaces a second time after our service does so. This causes two issues - first, the network target will always fail because it can't bring up an interface that is already up, and second, when configuring interfaces that don't actually have an available DHCP server it will result in a five minute delay waiting for DHCP on those interfaces. This will also cause the network target to fail and is an unnecessary delay. By moving the dhcp-interface service to run after the network target we avoid both of these problems. network.target will still bring up the interfaces on subsequent boots. This could result in the five minute delay happening on reboots, but the expected use case for interfaces without DHCP is that they would be configured statically on initial deployment so this should be a minor issue. The dhcp-interface service is also configured to run before the network-online target so that services which depend on the network actually being available will not race the DHCP process. A snippet from /var/log/messages on a node with this patch applied is included in the bug to demonstrate the behavior described above. Change-Id: I5cfabf20f920beea52abf4c42362b6f6ac0b37c4 Closes-Bug: 1653812 --- .../dhcp-all-interfaces/install.d/dhcp-interface@.service | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service b/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service index 112765029..903026e94 100644 --- a/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service +++ b/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service @@ -1,7 +1,11 @@ [Unit] Description=DHCP interface %I -Before=network-pre.target -Wants=network-pre.target +# We want to run after network.target so it doesn't try to bring +# up the interfaces a second time, but network-online should not +# be reached until after we've brought up the interfaces. +After=network.target +Before=network-online.target +Wants=network-online.target ConditionPathExists=!/etc/sysconfig/network-scripts/ifcfg-%I From ccd00b10b2ef7e447f8bcbcad0fa391ba2706271 Mon Sep 17 00:00:00 2001 From: Ben Nemec Date: Wed, 4 Jan 2017 11:21:03 -0600 Subject: [PATCH 15/31] Make DHCP timeout configurable As noted in the bug, there may be circumstances where a longer timeout than the current default is needed. This patch allows users to tune this timeout for their environment if need be. Change-Id: I173f3dad684894fbc3c27dece5ae15b5f63bae5a Closes-Bug: 1654027 --- elements/dhcp-all-interfaces/README.rst | 10 ++++++++++ .../install.d/50-dhcp-all-interfaces | 1 + .../install.d/dhcp-interface@.service | 2 +- 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/elements/dhcp-all-interfaces/README.rst b/elements/dhcp-all-interfaces/README.rst index dd7a3f8ca..7ecd387c7 100644 --- a/elements/dhcp-all-interfaces/README.rst +++ b/elements/dhcp-all-interfaces/README.rst @@ -17,3 +17,13 @@ configured properly before networking services are started. On Gentoo based distributions we will install the dhcpcd package and ensure the service starts at boot. This service automatically sets up all interfaces found via dhcp and/or dhcpv6 (or SLAAC). + +Environment Variables +--------------------- + +DIB_DHCP_TIMEOUT + :Required: No + :Default: 30 + :Description: Amount of time in seconds that the systemd service will + wait to get an address. + :Example: DIB_DHCP_TIMEOUT=300 diff --git a/elements/dhcp-all-interfaces/install.d/50-dhcp-all-interfaces b/elements/dhcp-all-interfaces/install.d/50-dhcp-all-interfaces index 972645a70..fa7e2fd7f 100755 --- a/elements/dhcp-all-interfaces/install.d/50-dhcp-all-interfaces +++ b/elements/dhcp-all-interfaces/install.d/50-dhcp-all-interfaces @@ -24,6 +24,7 @@ if [ "$DIB_INIT_SYSTEM" == "upstart" ]; then elif [ "$DIB_INIT_SYSTEM" == "systemd" ]; then install -D -g root -o root -m 0644 ${SCRIPTDIR}/dhcp-interface@.service /usr/lib/systemd/system/dhcp-interface@.service install -D -g root -o root -m 0644 ${SCRIPTDIR}/dhcp-all-interfaces-udev.rules /etc/udev/rules.d/99-dhcp-all-interfaces.rules + sed -i "s/TimeoutStartSec=DIB_DHCP_TIMEOUT/TimeoutStartSec=${DIB_DHCP_TIMEOUT:-30}s/" /usr/lib/systemd/system/dhcp-interface@.service elif [ "$DIB_INIT_SYSTEM" == "sysv" ]; then install -D -g root -o root -m 0755 ${SCRIPTDIR}/dhcp-all-interfaces.init /etc/init.d/dhcp-all-interfaces update-rc.d dhcp-all-interfaces defaults diff --git a/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service b/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service index 903026e94..c22594d58 100644 --- a/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service +++ b/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service @@ -15,7 +15,7 @@ User=root ExecStartPre=/usr/local/sbin/dhcp-all-interfaces.sh %I ExecStart=/sbin/ifup %I RemainAfterExit=true -TimeoutStartSec=30s +TimeoutStartSec=DIB_DHCP_TIMEOUT [Install] WantedBy=multi-user.target From 6c5234e162319f007e9fce03a8b50bd2a3a12c46 Mon Sep 17 00:00:00 2001 From: Matthew Thode Date: Tue, 20 Dec 2016 21:49:25 -0600 Subject: [PATCH 16/31] Fix Gentoo builds on Ubuntu 16.04 Xenial hosts Xenial's bind of /dev into the chroot includes /dev/shm which is in use by the host. An alternitive fix for this would be to use rbind to recursivly bind mount /dev instead of just the base bind of /dev Change-Id: I2c0f70afd1e82dd52a522f0dd2b3ea618b30b6c6 --- elements/gentoo/bin/install-packages | 34 ++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/elements/gentoo/bin/install-packages b/elements/gentoo/bin/install-packages index f0a4bf49a..a059bc78f 100755 --- a/elements/gentoo/bin/install-packages +++ b/elements/gentoo/bin/install-packages @@ -33,15 +33,25 @@ function show_options { } function fix_shm { + # make /dev/shm dir if it doesn't exist + # mount tmpfs and chown it + # existing programs could be using /dev/shm + # This means it cannot be moved or backed + # up as a copy easily. The only remaining + # option is to move the link if it exists + # as a link. Existing programs will still + # hold the file handle of the original + # location open and new programs can use + # the fixed /dev/shm. if [[ "${RUN_ONCE_SHM}" == '1' ]]; then - if [[ -L /dev/shm.orig ]]; then - rm /dev/shm.orig + if [[ ! -d /dev/shm ]]; then + if [[ ! -e /dev/shm ]]; then + if [[ -L /dev/shm ]]; then + mv /dev/shm /dev/shm.orig + fi + mkdir /dev/shm + fi fi - if [[ -d /dev/shm.orig ]]; then - rm -Rf /dev/shm.orig - fi - mv /dev/shm /dev/shm.orig - mkdir /dev/shm mount -t tmpfs none /dev/shm chmod 1777 /dev/shm RUN_ONCE_SHM='0' @@ -49,10 +59,16 @@ function fix_shm { } function unfix_shm { + # unmount tmpfs + # care about anything still using it if [[ "${RUN_ONCE_SHM}" == '0' ]]; then umount /dev/shm - rmdir /dev/shm - mv /dev/shm.orig /dev/shm + if fuser /dev/shm; then + rmdir /dev/shm + fi + if [[ -e /dev/shm.orig ]]; then + mv /dev/shm.orig /dev/shm + fi fi } From f8eba14d999a236e350e303d900d95174d524258 Mon Sep 17 00:00:00 2001 From: Bob Fournier Date: Thu, 12 Jan 2017 10:22:43 -0500 Subject: [PATCH 17/31] Handle failure of carrier check in dhcp-all-interfaces.sh As described in the bug, there are conditions with certain switches in which the interface is 'admin down'ed during initialization. Doing a 'cat' on /sys/class/net//carrier when it is 'admin down'ed produces an 'Invalid Argument' error and the script terminates. What this fix does is ignore failures of the 'cat' operation (by '|| echo 0') and place the link up inside the retry loop. Change-Id: I4f098aa5078b8482681394a3e9a6b17ed4bd4451 Closes-Bug: 1654046 --- .../dhcp-all-interfaces/install.d/dhcp-all-interfaces.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/elements/dhcp-all-interfaces/install.d/dhcp-all-interfaces.sh b/elements/dhcp-all-interfaces/install.d/dhcp-all-interfaces.sh index 4884c435a..20a49ab6f 100755 --- a/elements/dhcp-all-interfaces/install.d/dhcp-all-interfaces.sh +++ b/elements/dhcp-all-interfaces/install.d/dhcp-all-interfaces.sh @@ -38,7 +38,7 @@ function serialize_me() { } function get_if_link() { - cat /sys/class/net/${1}/carrier + cat /sys/class/net/${1}/carrier || echo 0 } function enable_interface() { @@ -87,11 +87,11 @@ function inspect_interface() { elif [ "$mac_addr_type" != "0" ]; then echo "Device has generated MAC, skipping." else - ip link set dev $interface up &>/dev/null - local has_link local tries for ((tries = 0; tries < 20; tries++)); do + # Need to set the link up on each iteration + ip link set dev $interface up &>/dev/null has_link=$(get_if_link $interface) [ "$has_link" == "1" ] && break sleep 1 From 812ded362dcf042f7eef258ccdae0d1d8aa05334 Mon Sep 17 00:00:00 2001 From: Ian Wienand Date: Fri, 13 Jan 2017 15:45:56 +1100 Subject: [PATCH 18/31] Also check bin/ for tabs Add bin/ to dib-lint basic script checks Change-Id: I92849ab608a6c792d790f041dc7614278c274d30 --- bin/dib-lint | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/dib-lint b/bin/dib-lint index 0b33790e8..c07ab99bf 100755 --- a/bin/dib-lint +++ b/bin/dib-lint @@ -204,7 +204,7 @@ done echo "Checking indents..." -for i in $(find elements -type f -and -name '*.rst' -or -type f -executable); do +for i in $(find bin elements -type f -and -name '*.rst' -or -type f -executable); do # Check for tab indentation if ! excluded tabindent; then if grep -q $'^ *\t' ${i}; then From 022d93ee822e71245af52c4cf8f8a8e82f599af3 Mon Sep 17 00:00:00 2001 From: Andreas Florath Date: Sat, 14 Jan 2017 09:35:24 +0000 Subject: [PATCH 19/31] Unify tidy up logs in lib/img-functions Cleaning logs was split, some was done in the img-functions.finalise_base, some was done in the base element. The version unifies tidy up logs in the lib/img-functions. Especially when building docker container images the base element cannot be used. This patch removes about some hundreds KB of useless logs in cases when the base element is not used. Change-Id: I165bafb73daf9144c2f3a83930e85e8d8cf5fae3 Signed-off-by: Andreas Florath --- elements/base/cleanup.d/99-tidy-logs | 30 ------------------- lib/img-functions | 4 +++ ...ve_tidy_logs_to_main-a8c03427fe1a445c.yaml | 8 +++++ 3 files changed, 12 insertions(+), 30 deletions(-) delete mode 100755 elements/base/cleanup.d/99-tidy-logs create mode 100644 releasenotes/notes/move_tidy_logs_to_main-a8c03427fe1a445c.yaml diff --git a/elements/base/cleanup.d/99-tidy-logs b/elements/base/cleanup.d/99-tidy-logs deleted file mode 100755 index 9274a7b0b..000000000 --- a/elements/base/cleanup.d/99-tidy-logs +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/bash -# -# Copyright 2014 Hewlett-Packard Development Company, L.P. -# All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. -# - -if [ ${DIB_DEBUG_TRACE:-1} -gt 0 ]; then - set -x -fi -set -eu -set -o pipefail - -# Truncate /var/log files in preparation for first boot -sudo find $TARGET_ROOT/var/log -type f -exec cp /dev/null '{}' \; - -# also /root logs -sudo find $TARGET_ROOT/root -name \*.log -type f -delete - diff --git a/lib/img-functions b/lib/img-functions index d803cc81c..bb74321aa 100644 --- a/lib/img-functions +++ b/lib/img-functions @@ -110,6 +110,10 @@ function finalise_base () { unmount_dir $TMP_MOUNT_PATH/tmp fi find $TMP_MOUNT_PATH/tmp -maxdepth 1 -mindepth 1 | xargs sudo rm -rf --one-file-system + # Truncate /var/log files in preparation for first boot + sudo find ${TMP_MOUNT_PATH}/var/log -type f -exec cp /dev/null '{}' \; + # also /root logs + sudo find ${TMP_MOUNT_PATH}/root -name \*.log -type f -delete } function compress_and_save_image () { diff --git a/releasenotes/notes/move_tidy_logs_to_main-a8c03427fe1a445c.yaml b/releasenotes/notes/move_tidy_logs_to_main-a8c03427fe1a445c.yaml new file mode 100644 index 000000000..fbd729561 --- /dev/null +++ b/releasenotes/notes/move_tidy_logs_to_main-a8c03427fe1a445c.yaml @@ -0,0 +1,8 @@ +--- +features: + - Cleaning logs was split, some was done in the + img-functions.finalise_base, some was done in the base element. + The version unifies tidy up logs in the lib/img-functions. + Especially when building docker container images the base element + cannot be used. This patch removes about some hundreds KB of + useless logs in cases when the base element is not used. \ No newline at end of file From ff8ae432653baaf3ee343669d2617e9968acb19f Mon Sep 17 00:00:00 2001 From: Luong Anh Tuan Date: Mon, 16 Jan 2017 15:01:51 +0700 Subject: [PATCH 20/31] Replace yaml.load() with yaml.safe_load() Avoid dangerous file parsing and object serialization libraries. yaml.load is the obvious function to use but it is dangerous[1] Because yaml.load return Python object may be dangerous if you receive a YAML document from an untrusted source such as the Internet. The function yaml.safe_load limits this ability to simple Python objects like integers or lists. In addition, Bandit flags yaml.load() as security risk so replace all occurrences with yaml.safe_load(). Thus I replace yaml.load() with yaml.safe_load() [1]https://security.openstack.org/guidelines/dg_avoid-dangerous-input-parsing-libraries.html Change-Id: I84640973fd9f45a69d2b21f6d594cd5bf10660a6 Closes-Bug: #1634265 --- bin/dib-lint | 2 +- elements/package-installs/bin/package-installs-squash | 2 +- elements/svc-map/bin/svc-map | 2 +- elements/svc-map/extra-data.d/10-merge-svc-map-files | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/bin/dib-lint b/bin/dib-lint index c07ab99bf..8d46a6dec 100755 --- a/bin/dib-lint +++ b/bin/dib-lint @@ -233,7 +233,7 @@ for i in $(find elements -type f -name '*.yaml'); do import yaml import sys try: - objs = yaml.load(open('$i')) + objs = yaml.safe_load(open('$i')) except yaml.parser.ParserError: sys.exit(1) " diff --git a/elements/package-installs/bin/package-installs-squash b/elements/package-installs/bin/package-installs-squash index eafdd9f42..949a8ae85 100755 --- a/elements/package-installs/bin/package-installs-squash +++ b/elements/package-installs/bin/package-installs-squash @@ -59,7 +59,7 @@ def collect_data(data, filename, element_name): try: objs = json.load(open(filename)) except ValueError: - objs = yaml.load(open(filename)) + objs = yaml.safe_load(open(filename)) for pkg_name, params in objs.items(): if not params: params = {} diff --git a/elements/svc-map/bin/svc-map b/elements/svc-map/bin/svc-map index 034c1fca8..2df6ba912 100755 --- a/elements/svc-map/bin/svc-map +++ b/elements/svc-map/bin/svc-map @@ -24,7 +24,7 @@ def load_service_mapping(filepath="/usr/share/svc-map/services"): if not os.path.isfile(filepath): return {} with open(filepath, 'r') as data_file: - return yaml.load(data_file.read()) + return yaml.safe_load(data_file.read()) def main(): diff --git a/elements/svc-map/extra-data.d/10-merge-svc-map-files b/elements/svc-map/extra-data.d/10-merge-svc-map-files index c8b0e0345..356976852 100755 --- a/elements/svc-map/extra-data.d/10-merge-svc-map-files +++ b/elements/svc-map/extra-data.d/10-merge-svc-map-files @@ -70,7 +70,7 @@ def main(): data_path = os.path.join(element_path, element, "svc-map") if os.path.exists(data_path): with open(data_path, 'r') as dataFile: - data = yaml.load(dataFile.read()) + data = yaml.safe_load(dataFile.read()) try: service_names = merge_data( data, From 6c8d7432a6e9797ed8777770e8d4325b00bf5271 Mon Sep 17 00:00:00 2001 From: Ian Wienand Date: Fri, 25 Nov 2016 11:08:11 +1100 Subject: [PATCH 21/31] Set grub device in /etc/default/grub Currently we run grub-mkconfig then go and edit the generated output in /boot/grub/grub.cfg to override the "root=" arguments to our label. If another element like project-config's finalise.d/99-fix-grub-timeout then goes and re-runs grub2-install, it overwrites these changes. Grub has GRUB_DEVICE which should override the root device. Let's try using this. Maybe it wasn't around at the time, this code is more-or-less unchanged from I7d83bb2b359e7a8c3858eca04c96e35cf4e1fe9e Change-Id: Ibaaa81124098f3c6febe48e455d3e1cd0a5f1761 --- elements/bootloader/finalise.d/50-bootloader | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/elements/bootloader/finalise.d/50-bootloader b/elements/bootloader/finalise.d/50-bootloader index fbf9e56c8..61782ae7d 100755 --- a/elements/bootloader/finalise.d/50-bootloader +++ b/elements/bootloader/finalise.d/50-bootloader @@ -133,6 +133,10 @@ function install_grub2 { GRUB_CFG=/boot/grub/grub.cfg fi + # Override the root device to the default label, and disable uuid + # lookup. + echo "GRUB_DEVICE=LABEL=${DIB_ROOT_LABEL}" >> /etc/default/grub + echo 'GRUB_DISABLE_LINUX_UUID=true' >> /etc/default/grub echo "GRUB_TIMEOUT=${DIB_GRUB_TIMEOUT:-5}" >>/etc/default/grub echo 'GRUB_TERMINAL="serial console"' >>/etc/default/grub echo 'GRUB_GFXPAYLOAD_LINUX=text' >>/etc/default/grub @@ -181,17 +185,7 @@ function install_grub2 { sed -i "s%search --no.*%%" $GRUB_CFG sed -i "s%set root=.*%set root=(hd0,1)%" $GRUB_CFG fi - # force use of a LABEL: - # NOTE: Updating the grub config by hand once deployed should work, its just - # prepping it in a different environment that needs fiddling. - sed -i "s%$PART_DEV%LABEL=${DIB_ROOT_LABEL}%" $GRUB_CFG - sed -i "s%search --no-floppy --fs-uuid --set=root .*$%search --no-floppy --set=root --label ${DIB_ROOT_LABEL}%" $GRUB_CFG - sed -i "s%root=UUID=[A-Za-z0-9\-]*%root=LABEL=${DIB_ROOT_LABEL}%" $GRUB_CFG - if [ "$DISTRO_NAME" = 'fedora' ] ; then - if [ "$DIB_RELEASE" = '19' ]; then - sed -i "s%UUID=[A-Za-z0-9\-]*%LABEL=${DIB_ROOT_LABEL}%" /etc/fstab - fi - fi + # Fix efi specific instructions in grub config file if [ -d /sys/firmware/efi ]; then sed -i 's%\(initrd\|linux\)efi /boot%\1 /boot%g' $GRUB_CFG From 1f43432313950930704f820a723a65320e31f93c Mon Sep 17 00:00:00 2001 From: Paul Belanger Date: Fri, 23 Dec 2016 12:38:07 -0500 Subject: [PATCH 22/31] Remove hardcoded components We can use ${DIB_DEBIAN_COMPONENTS} to get this information. We also already set the mirror. Change-Id: Idd11c2b7df1d247c6d32a5f936b8601b4741b519 Signed-off-by: Paul Belanger --- .../ubuntu-minimal/pre-install.d/01-ubuntu-apt-update | 10 ++++------ .../root.d/75-ubuntu-minimal-baseinstall | 10 ++++------ 2 files changed, 8 insertions(+), 12 deletions(-) diff --git a/elements/ubuntu-minimal/pre-install.d/01-ubuntu-apt-update b/elements/ubuntu-minimal/pre-install.d/01-ubuntu-apt-update index db487e747..f5e8ca4ce 100755 --- a/elements/ubuntu-minimal/pre-install.d/01-ubuntu-apt-update +++ b/elements/ubuntu-minimal/pre-install.d/01-ubuntu-apt-update @@ -21,14 +21,12 @@ fi set -eu set -o pipefail -DIB_DISTRIBUTION_MIRROR=${DIB_DISTRIBUTION_MIRROR:-http://archive.ubuntu.com/ubuntu} - # We should manage this in a betterer way cat << EOF >/etc/apt/sources.list -deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE main restricted universe -deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE-updates main restricted universe -deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE-backports main restricted universe -deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE-security main restricted universe +deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE ${DIB_DEBIAN_COMPONENTS//,/ } +deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE-updates ${DIB_DEBIAN_COMPONENTS//,/ } +deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE-backports ${DIB_DEBIAN_COMPONENTS//,/ } +deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE-security ${DIB_DEBIAN_COMPONENTS//,/ } EOF # Need to update to retrieve the signed Release file diff --git a/elements/ubuntu-minimal/root.d/75-ubuntu-minimal-baseinstall b/elements/ubuntu-minimal/root.d/75-ubuntu-minimal-baseinstall index 359389957..24c80fa4e 100755 --- a/elements/ubuntu-minimal/root.d/75-ubuntu-minimal-baseinstall +++ b/elements/ubuntu-minimal/root.d/75-ubuntu-minimal-baseinstall @@ -21,14 +21,12 @@ fi set -eu set -o pipefail -DIB_DISTRIBUTION_MIRROR=${DIB_DISTRIBUTION_MIRROR:-http://archive.ubuntu.com/ubuntu} - # We should manage this in a betterer way sudo bash -c "cat << EOF >$TARGET_ROOT/etc/apt/sources.list -deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE main restricted universe -deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE-updates main restricted universe -deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE-backports main restricted universe -deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE-security main restricted universe +deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE ${DIB_DEBIAN_COMPONENTS//,/ } +deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE-updates ${DIB_DEBIAN_COMPONENTS//,/ } +deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE-backports ${DIB_DEBIAN_COMPONENTS//,/ } +deb $DIB_DISTRIBUTION_MIRROR $DIB_RELEASE-security ${DIB_DEBIAN_COMPONENTS//,/ } EOF" sudo mount -t proc none $TARGET_ROOT/proc From d20754f6ed0b76ebf695c4eadf877edce0536464 Mon Sep 17 00:00:00 2001 From: Ben Nemec Date: Mon, 12 Dec 2016 15:46:25 -0600 Subject: [PATCH 23/31] Use %i instead of %I in dhcp-interface@.service Per the bug report, %I results in -'s in the interface name being replaced with /'s, which means when we try to look up the interface by path in dhcp-all-interfaces.sh we end up at an invalid path. Using %i instead should fix the problem. See https://www.freedesktop.org/software/systemd/man/systemd.unit.html Essentially what is happening is that we start with a name like br-ex that wasn't escaped in the first place. However, because of the - it looks like it could have been escaped to systemd. When we use %I, which tries to unescape escaped values, it unescapes a value that was already not escaped. Change-Id: I434ed2e084d4477dc7a2b7827164586f8ea0c1e3 Closes-Bug: 1649409 --- .../dhcp-all-interfaces/install.d/dhcp-interface@.service | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service b/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service index c22594d58..b066aed63 100644 --- a/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service +++ b/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service @@ -1,5 +1,5 @@ [Unit] -Description=DHCP interface %I +Description=DHCP interface %i # We want to run after network.target so it doesn't try to bring # up the interfaces a second time, but network-online should not # be reached until after we've brought up the interfaces. @@ -7,13 +7,13 @@ After=network.target Before=network-online.target Wants=network-online.target -ConditionPathExists=!/etc/sysconfig/network-scripts/ifcfg-%I +ConditionPathExists=!/etc/sysconfig/network-scripts/ifcfg-%i [Service] Type=oneshot User=root -ExecStartPre=/usr/local/sbin/dhcp-all-interfaces.sh %I -ExecStart=/sbin/ifup %I +ExecStartPre=/usr/local/sbin/dhcp-all-interfaces.sh %i +ExecStart=/sbin/ifup %i RemainAfterExit=true TimeoutStartSec=DIB_DHCP_TIMEOUT From baf3ced0c32072a5fa525277380a8777fc0a471a Mon Sep 17 00:00:00 2001 From: Ian Wienand Date: Wed, 18 Jan 2017 15:07:46 +1100 Subject: [PATCH 24/31] Fix coverage report There's a few things going on here Firstly, we need to install coverage tool in test-requirements Secondly, .testr.conf has to use PYTHON because the coverage report works by resetting PYTHON='coverage run ...' Thirdly, because we call ourselves diskimage-builder but the python module is diskimage_builder that seems to confuse things. We need to use "setup.py test" (note, that is different to "setup.py testr"!) to use the PBR testr wrapper. That exposes a --coverage-package-name argument that calls the coverage tool with the right argument. With this I got a coverage report for our unit tests Change-Id: I9012e18eb7d01bee035140e70afa76c47c27eb01 --- .testr.conf | 2 +- test-requirements.txt | 2 ++ tox.ini | 9 +++++---- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/.testr.conf b/.testr.conf index 25d7f8d95..d7b8b32fe 100644 --- a/.testr.conf +++ b/.testr.conf @@ -4,7 +4,7 @@ test_command=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_LOG_CAPTURE=${OS_LOG_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \ OS_DEBUG=${OS_DEBUG:-0} \ - python -m subunit.run discover . $LISTOPT $IDOPTION + ${PYTHON:-python} -m subunit.run discover . $LISTOPT $IDOPTION test_id_option=--load-list $IDFILE test_list_option=--list diff --git a/test-requirements.txt b/test-requirements.txt index f9f2a6b81..a30316a7e 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -12,3 +12,5 @@ oslosphinx>=4.7.0 # Apache-2.0 # releasenotes reno>=1.8.0 # Apache-2.0 + +coverage>=4.0 # Apache-2.0 diff --git a/tox.ini b/tox.ini index 4bd24a3f2..fe0a09e21 100644 --- a/tox.ini +++ b/tox.ini @@ -9,7 +9,7 @@ install_command = pip install -U {opts} {packages} deps= -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt commands= - python setup.py testr --slowest --testr-args='{posargs}' + python setup.py test --slowest --testr-args='{posargs}' [testenv:pep8] commands = @@ -24,9 +24,10 @@ envdir = {toxworkdir}/venv commands = {toxinidir}/tests/run_functests.sh {posargs} [testenv:cover] -setenv = PYTHON=coverage run --source diskimage_builder -commands = bash -c 'if [ ! -d ./.testrepository ] ; then testr init ; fi' - bash -c 'testr run --parallel ; RET=$? ; coverage combine ; coverage html -d ./cover $OMIT && exit $RET' +# NOTE: this is "setup.py test" (*not* testr) which is a pbr wrapper +# around testr. This understands --coverage-package-name which we +# need due to underscore issues. +commands = python setup.py test --coverage --coverage-package-name diskimage_builder --testr-args='{posargs}' [testenv:docs] commands = python setup.py build_sphinx From 0ceb138d877101542934b7729f9497bc4c9a039a Mon Sep 17 00:00:00 2001 From: Ian Wienand Date: Wed, 18 Jan 2017 08:32:10 +1100 Subject: [PATCH 25/31] Don't set base element path in run_functests.sh DIB automatically includes it's base elements in the ELEMENT_PATH; this double-set means the base elements are in the path twice, leading to some tools that walk the element list (package-installs-squash, for example) to do things twice. Change-Id: Icf631427601cfd2abed436711cedb9e002fff8f9 --- tests/run_functests.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/run_functests.sh b/tests/run_functests.sh index 14bd39654..d8459b028 100755 --- a/tests/run_functests.sh +++ b/tests/run_functests.sh @@ -97,7 +97,7 @@ function run_disk_element_test() { if break="after-error" break_outside_target=1 \ break_cmd="cp -v \$TMP_MOUNT_PATH/tmp/dib-test-should-fail ${dest_dir} || true" \ DIB_SHOW_IMAGE_USAGE=1 \ - ELEMENTS_PATH=$DIB_ELEMENTS:$DIB_ELEMENTS/$element/test-elements \ + ELEMENTS_PATH=$DIB_ELEMENTS/$element/test-elements \ $DIB_CMD -x -t tar,qcow2 ${use_tmp_flag} -o $dest_dir/image -n $element $test_element 2>&1 \ | log_with_prefix "${element}/${test_element}"; then From a45ad8eaf56c16c8f77dc8f96fecaf224af4d5bf Mon Sep 17 00:00:00 2001 From: Andrey Shestakov Date: Tue, 6 Dec 2016 23:59:49 +0200 Subject: [PATCH 26/31] Fix dhcp-all-interfaces for ubuntu-minimal xenial The start script of dhcp-all-interfaces currently requires ifup command. ifup command provided by package ifupdown, which is not installed in ubuntu-minimal Xenial. This change adds ifupdown package for debian family, as it required for dhcp-all-interfaces. Closes-bug: #1647853 Change-Id: I6dfc0108ec067f350b22e9fb933b9e8d47b09fde --- elements/dhcp-all-interfaces/package-installs.yaml | 1 + elements/dhcp-all-interfaces/pkg-map | 6 +++++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/elements/dhcp-all-interfaces/package-installs.yaml b/elements/dhcp-all-interfaces/package-installs.yaml index 98034e6c1..f73108729 100644 --- a/elements/dhcp-all-interfaces/package-installs.yaml +++ b/elements/dhcp-all-interfaces/package-installs.yaml @@ -1 +1,2 @@ dhcp-client: +ifupdown: diff --git a/elements/dhcp-all-interfaces/pkg-map b/elements/dhcp-all-interfaces/pkg-map index 6abf6d80f..080f6cd4e 100644 --- a/elements/dhcp-all-interfaces/pkg-map +++ b/elements/dhcp-all-interfaces/pkg-map @@ -8,9 +8,13 @@ }, "suse": { "dhcp-client": "dhcp-client" + }, + "debian": { + "ifupdown": "ifupdown" } }, "default": { - "dhcp-client": "isc-dhcp-client" + "dhcp-client": "isc-dhcp-client", + "ifupdown": "" } } From 90a347707c2c0e411052d9d54e00a3380f3e9ce3 Mon Sep 17 00:00:00 2001 From: Paul Belanger Date: Thu, 12 Jan 2017 11:55:47 -0500 Subject: [PATCH 27/31] Bump fedora release to 25 Now that fedora-25 is released, bump fedora-minimal / fedora elements to use it. Change-Id: Ifdb05255e7a138b99099252fe17041fe4621b6f4 Depends-On: I3cc467a2d08486458b3f625f94ba969532f6cd04 Signed-off-by: Paul Belanger --- .../fedora-minimal/environment.d/10-fedora-distro-name.bash | 2 +- elements/fedora/environment.d/10-fedora-distro-name.bash | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/elements/fedora-minimal/environment.d/10-fedora-distro-name.bash b/elements/fedora-minimal/environment.d/10-fedora-distro-name.bash index 3477f1eac..0ffbc4f4e 100644 --- a/elements/fedora-minimal/environment.d/10-fedora-distro-name.bash +++ b/elements/fedora-minimal/environment.d/10-fedora-distro-name.bash @@ -1,2 +1,2 @@ export DISTRO_NAME=fedora -export DIB_RELEASE=${DIB_RELEASE:-24} +export DIB_RELEASE=${DIB_RELEASE:-25} diff --git a/elements/fedora/environment.d/10-fedora-distro-name.bash b/elements/fedora/environment.d/10-fedora-distro-name.bash index 07b4e60e4..5c83ef2e8 100644 --- a/elements/fedora/environment.d/10-fedora-distro-name.bash +++ b/elements/fedora/environment.d/10-fedora-distro-name.bash @@ -1,5 +1,5 @@ export DISTRO_NAME=fedora -export DIB_RELEASE=${DIB_RELEASE:-24} +export DIB_RELEASE=${DIB_RELEASE:-25} if [ -n "${DIB_FEDORA_DISTRIBUTION_MIRROR:-}" ]; then export DIB_DISTRIBUTION_MIRROR=$DIB_FEDORA_DISTRIBUTION_MIRROR fi From dc8449bf02316f27b556994644f7c4ce18b995a8 Mon Sep 17 00:00:00 2001 From: Matthew Thode Date: Sun, 22 Jan 2017 15:53:02 -0600 Subject: [PATCH 28/31] move post-install.d to finalize.d It looks like I installed the cleanup file in the wrong location. Moving it to the correct location and cleaning some more stuff up should allow for a much smaller image, currently tested at 300M vs 490M. Change-Id: I9d0a24b0fe59d4f2d38aa88ba47e4400e0476b46 --- .../gentoo/{post-install.d => finalise.d}/99-cleanup | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) rename elements/gentoo/{post-install.d => finalise.d}/99-cleanup (71%) diff --git a/elements/gentoo/post-install.d/99-cleanup b/elements/gentoo/finalise.d/99-cleanup similarity index 71% rename from elements/gentoo/post-install.d/99-cleanup rename to elements/gentoo/finalise.d/99-cleanup index 656b0a10d..f8f5507a7 100755 --- a/elements/gentoo/post-install.d/99-cleanup +++ b/elements/gentoo/finalise.d/99-cleanup @@ -7,8 +7,16 @@ set -eu set -o pipefail # make sure system is in a consistant state +echo 'PYTHON_TARGETS="python3_4"' >> /etc/portage/make.conf +eselect python set python3.4 +emerge -C -q dev-lang/python:2.7 dev-vcs/git +USE="-build" emerge -q --jobs=2 --update --newuse --deep --with-bdeps=y @world +# rebuild packages that might need it +USE="-build" emerge -q --jobs=2 --usepkg=n @preserved-rebuild +# remove unneeded packages USE="-build" emerge --verbose=n --depclean -USE="-build" emerge -v --usepkg=n @preserved-rebuild +# rebuild packages that might have somehow depended on the unneeded packages +USE="-build" emerge -q --jobs=2 --usepkg=n @preserved-rebuild # update config files etc-update --automode -5 From ca60b2cf7d30a8978ae9166a23bfee1bca8f6f6a Mon Sep 17 00:00:00 2001 From: Ian Wienand Date: Mon, 23 Jan 2017 14:20:36 +1100 Subject: [PATCH 29/31] ironic-agent: use /sbin for modprobe On redhat/fedora /sbin is a symlink to /usr/sbin, but not on all platforms. This was put in with Ibf74dd1b2678ea76e0676711a7aa5ba6b88d5421 Change-Id: I7847b29503c3c07503430a7d85a5364911894c6c Closes-bug: #1658297 --- .../ironic-agent-source-install/ironic-python-agent.service | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/elements/ironic-agent/install.d/ironic-agent-source-install/ironic-python-agent.service b/elements/ironic-agent/install.d/ironic-agent-source-install/ironic-python-agent.service index da9709660..d6e4ca2a3 100644 --- a/elements/ironic-agent/install.d/ironic-agent-source-install/ironic-python-agent.service +++ b/elements/ironic-agent/install.d/ironic-agent-source-install/ironic-python-agent.service @@ -3,7 +3,7 @@ Description=Ironic Python Agent After=network-online.target [Service] -ExecStartPre=/usr/sbin/modprobe vfat +ExecStartPre=/sbin/modprobe vfat ExecStart=/usr/local/bin/ironic-python-agent Restart=always RestartSec=30s From 11d384e2fa1f550d424f686600d643b3969a40b3 Mon Sep 17 00:00:00 2001 From: Andrey Shestakov Date: Thu, 5 Jan 2017 14:39:25 +0200 Subject: [PATCH 30/31] Add DIB_IPA_COMPRESS_CMD option This option allows to specifiy command for compress built initramfs image for ironic-agent element. This command can be specified with arguments and should read raw data from stdin and write compressed data to stdout. Default if "gzip". Change-Id: I0fdd2ab91d7bf7aaaa6cdd5278e3902d44c5b883 --- elements/ironic-agent/README.rst | 4 ++++ elements/ironic-agent/cleanup.d/99-ramdisk-create | 4 +++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/elements/ironic-agent/README.rst b/elements/ironic-agent/README.rst index a6382a029..112caa3e5 100644 --- a/elements/ironic-agent/README.rst +++ b/elements/ironic-agent/README.rst @@ -18,6 +18,10 @@ Beyond installing the ironic-python-agent, this element does the following: * Install the certificate if any, which is set to the environment variable ``DIB_IPA_CERT`` for validating the authenticity by ironic-python-agent. The certificate can be self-signed certificate or CA certificate. +* Compresses initramfs with command specified in environment variable + ``DIB_IPA_COMPRESS_CMD``, which is 'gzip' by default. This command should listen + for raw data from stdin and write compressed data to stdout. Command can be + with arguments. This element outputs three files: diff --git a/elements/ironic-agent/cleanup.d/99-ramdisk-create b/elements/ironic-agent/cleanup.d/99-ramdisk-create index 5e04a18ea..3db28bc76 100755 --- a/elements/ironic-agent/cleanup.d/99-ramdisk-create +++ b/elements/ironic-agent/cleanup.d/99-ramdisk-create @@ -18,6 +18,8 @@ source $_LIB/img-functions IMAGE_PATH=$(readlink -f $IMAGE_NAME) cd $TARGET_ROOT +DIB_IPA_COMPRESS_CMD="${DIB_IPA_COMPRESS_CMD:-gzip}" + echo "#disabled" > ./tmp/fstab.new sudo mv ./tmp/fstab.new ./etc/fstab sudo ln -s ./sbin/init ./ @@ -42,7 +44,7 @@ sudo find . -xdev \ -path './var/cache/*' -prune -o \ -name '*.pyc' -prune -o \ -name '*.pyo' -prune -o \ - -print | sudo cpio -o -H newc | gzip > ${IMAGE_PATH}.initramfs + -print | sudo cpio -o -H newc | ${DIB_IPA_COMPRESS_CMD} > ${IMAGE_PATH}.initramfs select_boot_kernel_initrd $TARGET_ROOT sudo cp $BOOTDIR/$KERNEL ${IMAGE_PATH}.kernel From 83bdfe179a549e9ae3cd61e4f15383f77d668c11 Mon Sep 17 00:00:00 2001 From: Ian Wienand Date: Tue, 31 Jan 2017 14:20:09 +1100 Subject: [PATCH 31/31] Git ignore coverage stuff One less piece of clutter when you're trying to merge ... Change-Id: I794027280359f5d1e02d4f3b99222d35783075d5 --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index 091b3585a..d2238b44b 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,6 @@ +.coverage +coverage.xml +cover/* *~ .testrepository *.sw?