[ansible] Improve partition images support

- partitioning scheme now closer resembles what's being done by IPA

  - for partition images the root partition is last so that it can grow
  - for whole-disk images, configdrive is created as far as possible

- added support for setting disk label for created partition table,
  supported are 'gpt' and 'msdos' (default)

  - for 'gpt' disks, a bios_grub partiton is prepended to standard ones

- 'parted' module for Ansible was renamed to ``ironic_parted`` to not
  shadow ``parted`` module included in Ansible 2.3
  It was also rewritten to be cleaner and more stable:

  - changed accepted module arguments
  - added support for units (MiB and % for now)
  - added more validations
  - left support for msdos and gpt partition tables only
  - partitions are created one by one, and actual block devices created
    are searched for and reported.
  - module returns only after all partitions are available for write

Change-Id: I4d6d7619c6f3ba25c29263ffe5d778698e598429
This commit is contained in:
Pavlo Shchelokovskyy 2017-06-12 13:39:26 +00:00
parent b963a18c63
commit 1d6b1b89d2
7 changed files with 479 additions and 163 deletions

View File

@ -69,13 +69,15 @@ Supports whole-disk images and partition images:
For partition images the driver will create root partition, and,
if requested, ephemeral and swap partitions as set in node's
``instance_info`` by nova or operator.
Partition table created will be of ``msdos`` type.
Partition table created will be of ``msdos`` type by default,
the node's``disk_label`` capability is honored if it is set in node's
``instance_info``.
Configdrive partition
~~~~~~~~~~~~~~~~~~~~~
Creating a configdrive partition is supported for both whole disk
and partition images.
and partition images, on both ``msdos`` and ``GPT`` labeled disks.
Root device hints
~~~~~~~~~~~~~~~~~
@ -121,7 +123,7 @@ Requirements
============
ironic
Requires ironic of Newton release or newer.
Requires ironic version >= 8.0.0. (Pike release or newer).
Ansible
Tested with and targets Ansible ≥ 2.1
@ -414,6 +416,7 @@ Those values are then accessible in your plays as well
type: <url|file>
location: <URL OR PATH ON CONDUCTOR>
partition_info:
label: <msdos|gpt>
preserve_ephemeral: <bool>
ephemeral_format: <FILESYSTEM TO CREATE ON EPHEMERAL PARTITION>
partitions: <LIST OF PARTITIONS IN FORMAT EXPECTED BY PARTED MODULE>
@ -444,15 +447,23 @@ Some more explanations:
partitions:
- name: <NAME OF PARTITION>
size_mib: <SIZE OF THE PARTITION>
boot: <bool>
swap: <bool>
unit: <UNITS FOR SIZE>
size: <SIZE OF THE PARTITION>
type: <primary|extended|logical>
align: <ONE OF PARTED_SUPPORTED OPTIONS>
format: <PARTITION TYPE TO SET>
flags:
flag_name: <bool>
The driver will populate this list from ``root_gb``, ``swap_mb`` and
``ephemeral_gb`` fields of ``instance_info``.
The driver will also prepend the ``bios_grub``-labeled partition
when deploying on GPT-labeled disk,
and pre-create a 64MiB partiton for configdrive if it is set in
``instance_info``.
Please read the documentation included in the ``parted`` module's source
for more info on the module and its arguments.
Please read the documentation included in the ``ironic_parted`` module's
source for more info on the module and its arguments.
``ironic.partiton_info.ephemeral_format``
Optional. Taken from ``instance_info``, it defines file system to be
@ -481,10 +492,12 @@ You can use these modules in your playbooks as well.
module arguments.
Due to the low level of such operation it is not idempotent.
``parted``
``ironic_parted``
creates partition tables and partitions with ``parted`` utility.
Due to the low level of such operation it is not idempotent.
Please read the documentation included in the module's source
for more information about this module and its arguments.
The name is chosen so that the ``parted`` module included in Ansible 2.3
is not shadowed.
.. _Ironic Python Agent: http://docs.openstack.org/developer/ironic-python-agent

View File

@ -249,31 +249,54 @@ def _parse_partitioning_info(node):
info = node.instance_info
i_info = {}
partitions = []
root_partition = {'name': 'root',
'size_mib': info['root_mb'],
'boot': 'yes',
'swap': 'no'}
partitions.append(root_partition)
i_info['label'] = deploy_utils.get_disk_label(node) or 'msdos'
# prepend 1MiB bios_grub partition for GPT so that grub(2) installs
if i_info['label'] == 'gpt':
bios_partition = {'name': 'bios',
'size': 1,
'unit': 'MiB',
'flags': {'bios_grub': 'yes'}}
partitions.append(bios_partition)
ephemeral_mb = info['ephemeral_mb']
if ephemeral_mb:
i_info['ephemeral_format'] = info['ephemeral_format']
ephemeral_partition = {'name': 'ephemeral',
'size': ephemeral_mb,
'unit': 'MiB',
'format': i_info['ephemeral_format']}
partitions.append(ephemeral_partition)
i_info['preserve_ephemeral'] = (
'yes' if info['preserve_ephemeral'] else 'no')
swap_mb = info['swap_mb']
if swap_mb:
swap_partition = {'name': 'swap',
'size_mib': swap_mb,
'boot': 'no',
'swap': 'yes'}
'size': swap_mb,
'unit': 'MiB',
'format': 'linux-swap'}
partitions.append(swap_partition)
ephemeral_mb = info['ephemeral_mb']
if ephemeral_mb:
ephemeral_partition = {'name': 'ephemeral',
'size_mib': ephemeral_mb,
'boot': 'no',
'swap': 'no'}
partitions.append(ephemeral_partition)
# pre-create partition for configdrive
configdrive = info.get('configdrive')
if configdrive:
configdrive_partition = {'name': 'configdrive',
'size': 64,
'unit': 'MiB',
'format': 'fat32'}
partitions.append(configdrive_partition)
i_info['ephemeral_format'] = info['ephemeral_format']
i_info['preserve_ephemeral'] = (
'yes' if info['preserve_ephemeral'] else 'no')
# NOTE(pas-ha) make the root partition last so that
# e.g. cloud-init can grow it on first start
root_partition = {'name': 'root',
'size': info['root_mb'],
'unit': 'MiB'}
if i_info['label'] == 'msdos':
root_partition['flags'] = {'boot': 'yes'}
partitions.append(root_partition)
i_info['partitions'] = partitions
return {'partition_info': i_info}

View File

@ -0,0 +1,344 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
# NOTE(pas-ha) might not need it when Ansible PullRequest#2971 is accepted
import itertools
try:
import json
except ImportError:
import simplejson as json
PARTITION_TYPES = ('primary', 'logical', 'extended')
SUPPORTED_UNITS = {'%', 'MiB'}
SUPPORTED_ALIGN = {'optimal', 'minimal', 'cylinder', 'none'}
DOCUMENTATION = """
---
module: ironic_parted
short_description: Create disk partition tables and partitions
description: uses GNU parted utility
author: Pavlo Shchelokovskyy @pshchelo
version_added: null
notes:
- IS NOT IDEMPOTENT! partitions and table (if requested) are created anyway
- does not support all the partition labels parted supports, only msdos and gpt
- does not support units other than % and MiB
- check mode is supported by returning emulated list of created block devices
- makes no validation re if given partitions will actually fit the device
- makes some extra validations for appropriate partition types for msdos label
requirements:
- Python >= 2.4 (itertools.groupby available) on the managed node
- 'simplejson' for Python < 2.6
- 'parted' utility installed on the managed node
- 'lsblk' available on managed node
- 'udevadm' available on managed node
options:
device:
description: device to pass to parted
required: true
default: null
choices: []
aliases: []
version_added: null
label:
description: |
type of a partition type to create;
to use an existing partition table, omit it or pass null YAML value
required: false
default: none
choices: [null, msdos, gpt]
aliases: []
version_added: null
dry_run:
description: |
if actually to write changes to disk.
If no, simulated partitions will be reported.
required: false
default: no
choices: [yes, no]
aliases: []
version_added: null
partitions:
description:|
list of partitions. each entry is a dictionary in the form
- size: <int>, required, must be positive non-zero
type: [primary, extended, logical], default is primary
format: a format to pass to parted;
does not actually creates filesystems, only sets
partition ID
name: <str> (optional) name of the partition;
only supported for gpt partitions;
if not set will be reported as 'partN'
unit: 'MiB' or '%' are currently supported,
must be the same for all partitions. default is '%'
align: one of 'optimal', 'cylinder', 'minimal' or 'none';
the default is 'optimal'
flags: <dict> of <flag>: <bool> to (un)set partition flags
required: false
default: null
choices: []
aliases: []
version_added: null
"""
EXAMPLES = """
---
"""
RETURNS = """
---
{"created": {
"<name-as-provided-to-module>": "<device-handle-without-leading-dev>"
}
}
"""
def parse_sizes(module, partitions):
start = 0 if partitions[0]['unit'] == '%' else 1
sizes = {}
for p in partitions:
size = p.get('size')
if not size:
module.fail_json(msg="Partition size must be provided")
try:
p['size'] = int(size)
except ValueError:
module.fail_json(msg="Can not cast partition size to INT.")
if p['size'] <= 0:
module.fail_json(msg="Partition size must be positive.")
end = start + p['size']
sizes[p['name']] = (start, end)
start = end
return sizes
def create_part_args(partition, label, sizes):
parted_args = ['-a', partition['align'],
'--', 'unit', partition['unit'],
'mkpart']
if label == 'msdos':
parted_args.append(partition['type'])
else:
parted_args.append(partition['name'])
if partition['format']:
parted_args.append(partition['format'])
parted_args.extend(["%i" % sizes[partition['name']][0],
"%i" % sizes[partition['name']][1]])
return parted_args
def change_part_args(part_number, partition):
parted_args = []
for flag, state in partition['flags'].items():
parted_args.extend(['set', part_number, flag, state])
return parted_args
def parse_lsblk_output(output):
devices = set()
for line in output.splitlines():
device = line.strip().split('=')[1]
devices.add(device.strip('"'))
return devices
def parse_lsblk_json(output):
def get_names(devices):
names = []
for d in devices:
names.append(d['name'])
names.extend(get_names(d.get('children', [])))
return names
return set(get_names(json.loads(output)['blockdevices']))
def parse_parted_output(output):
partitions = set()
for line in output.splitlines():
out_line = line.strip().split()
if out_line:
try:
int(out_line[0])
except ValueError:
continue
else:
partitions.add(out_line[0])
return partitions
def parse_partitions(module, partitions):
for ind, partition in enumerate(partitions):
# partition name might be an empty string
partition.setdefault('unit', '%')
partition.setdefault('align', 'optimal')
partition['name'] = partition.get('name') or 'part%i' % (ind + 1)
partition.setdefault('type', 'primary')
if partition['type'] not in PARTITION_TYPES:
module.fail_json(msg="Partition type must be one of "
"%s." % PARTITION_TYPES)
if partition['align'] not in SUPPORTED_ALIGN:
module.fail_json("Unsupported partition alignmnet option. "
"Supported are %s" % list(SUPPORTED_ALIGN))
partition['format'] = partition.get('format', None)
# validate and convert partition flags
partition['flags'] = {
k: 'on' if module.boolean(v) else 'off'
for k, v in partition.get('flags', {}).items()
}
# validate name uniqueness
names = [p['name'] for p in partitions]
if len(list(names)) != len(set(names)):
module.fail_json("Partition names must be unique.")
def validate_units(module, partitions):
has_units = set(p['unit'] for p in partitions)
if not has_units.issubset(SUPPORTED_UNITS):
module.fail_json("Unsupported partition size unit. Supported units "
"are %s" % list(SUPPORTED_UNITS))
if len(has_units) > 1:
module.fail_json("All partitions must have the same size unit. "
"Requested units are %s" % list(has_units))
def validate_msdos(module, partitions):
"""Validate limitations of MSDOS partition table"""
p_types = [p['type'] for p in partitions]
# NOTE(pas-ha) no more than 4 primary
if p_types.count('primary') > 4:
module.fail_json("Can not create more than 4 primary partitions "
"on a MSDOS partition table.")
if 'extended' in p_types:
# NOTE(pas-ha) only single extended
if p_types.count('extended') > 1:
module.fail_json("Can not create more than single extended "
"partition on a MSDOS partition table.")
allowed = ['primary', 'extended']
if 'logical' in p_types:
allowed.append('logical')
# NOTE(pas-ha) this produces list with subsequent duplicates
# removed
if [k for k, g in itertools.groupby(p_types)] != allowed:
module.fail_json("Incorrect partitions order: for MSDOS, "
"all primary, single extended, all logical")
elif 'logical' in p_types:
# NOTE(pas-ha) logical has sense only with extended
module.fail_json("Logical partition w/o extended one on MSDOS "
"partition table")
# TODO(pas-ha) add more validation, e.g.
# - add idempotency: first check the already existing partitions
# and do not run anything unless really needed, and only what's needed
# - if only change tags - use specific command
# - allow fuzziness in partition sizes when alligment is 'optimal'
# - estimate and validate available space
# - support more units
# - support negative units?
def main():
module = AnsibleModule(
argument_spec=dict(
device=dict(required=True, type='str'),
label=dict(requred=False, default=None, choices=[None,
"gpt",
"msdos"]),
dry_run=dict(required=False, type='bool', default=False),
partitions=dict(required=False, type='list')
),
supports_check_mode=True
)
device = module.params['device']
label = module.params['label']
partitions = module.params['partitions'] or []
dry_run = module.params['dry_run']
if partitions:
parse_partitions(module, partitions)
if label == 'msdos':
validate_msdos(module, partitions)
validate_units(module, partitions)
sizes = parse_sizes(module, partitions)
else:
sizes = {}
if module.check_mode or dry_run:
short_dev = device.split('/')[-1]
created_partitions = {}
for i, p in enumerate(partitions):
created_partitions[p['name']] = '%s%s' % (short_dev, i + 1)
module.exit_json(changed=dry_run, created=created_partitions)
parted_bin = module.get_bin_path('parted', required=True)
lsblk_bin = module.get_bin_path('lsblk', required=True)
udevadm_bin = module.get_bin_path('udevadm', required=True)
parted = [parted_bin, '-s', device]
# lsblk = [lsblk_bin, '-o', 'NAME', '-P', device]
lsblk = [lsblk_bin, '-J', device]
if label:
module.run_command(parted + ['mklabel', label], check_rc=True)
rc, part_output, err = module.run_command(parted + ['print'],
check_rc=True)
rc, lsblk_output, err = module.run_command(lsblk,
check_rc=True)
part_cache = parse_parted_output(part_output)
dev_cache = parse_lsblk_json(lsblk_output)
created_partitions = {}
for partition in partitions:
# create partition
parted_args = create_part_args(partition, label, sizes)
module.run_command(parted + parted_args, check_rc=True)
rc, part_output, err = module.run_command(parted + ['print'],
check_rc=True)
# get created partition number
part_current = parse_parted_output(part_output)
part_created = part_current - part_cache
part_cache = part_current
# set partition flags
parted_args = change_part_args(part_created.pop(),
partition)
if parted_args:
module.run_command(parted + parted_args, check_rc=True)
# get created block device name
rc, lsblk_output, err = module.run_command(lsblk, check_rc=True)
dev_current = parse_lsblk_json(lsblk_output)
dev_created = dev_current - dev_cache
dev_cache = dev_current
created_partitions[partition['name']] = dev_created.pop()
# NOTE(pas-ha) wait for all partitions to become available for write
for dev_name in created_partitions.values():
module.run_command([udevadm_bin,
'settle',
'--exit-if-exists=/dev/%s' % dev_name])
module.exit_json(changed=True, created=created_partitions)
from ansible.module_utils.basic import * # noqa
if __name__ == '__main__':
main()

View File

@ -1,111 +0,0 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
PARTITION_TYPES = ('primary', 'logical', 'extended')
def construct_parted_args(device):
parted_args = [
'-s', device['device'],
]
if device['label']:
parted_args.extend(['mklabel', device['label']])
partitions = device['partitions']
if partitions:
parted_args.extend(['-a', 'optimal', '--', 'unit', 'MiB'])
start = 1
for ind, partition in enumerate(device['partitions']):
parted_args.extend([
'mkpart', partition['type']])
if partition['swap']:
parted_args.append('linux-swap')
end = start + partition['size_mib']
parted_args.extend(["%i" % start, "%i" % end])
start = end
if partition['boot']:
parted_args.extend([
'set', str(ind + 1), 'boot', 'on'])
return parted_args
def validate_partitions(module, partitions):
for ind, partition in enumerate(partitions):
# partition name might be an empty string
partition['name'] = partition.get('name') or str(ind + 1)
size = partition.get('size_mib', None)
if not size:
module.fail_json(msg="Partition size must be provided")
try:
partition['size_mib'] = int(size)
except ValueError:
module.fail_json(msg="Can not cast partition size to INT.")
partition.setdefault('type', 'primary')
if partition['type'] not in PARTITION_TYPES:
module.fail_json(msg="Partition type must be one of "
"%s." % PARTITION_TYPES)
partition['swap'] = module.boolean(partition.get('swap', False))
partition['boot'] = module.boolean(partition.get('boot', False))
if partition['boot'] and partition['swap']:
module.fail_json(msg="Can not set partition to "
"boot and swap simultaneously.")
# TODO(pas-ha) add more validation, e.g.
# - only one boot partition?
# - no more than 4 primary partitions on msdos table
# - no more that one extended partition on msdos table
# - estimate and validate available space
def main():
module = AnsibleModule(
argument_spec=dict(
device=dict(required=True, type='str'),
dryrun=dict(required=False, default=False, type='bool'),
new_label=dict(required=False, default=False, type='bool'),
label=dict(requred=False, default='msdos', choices=[
"bsd", "dvh", "gpt", "loop", "mac", "msdos", "pc98", "sun"]),
partitions=dict(
required=False, type='list')
),
supports_check_mode=True)
device = module.params['device']
dryrun = module.params['dryrun']
new_label = module.params['new_label']
label = module.params['label']
if not new_label:
label = False
partitions = module.params['partitions'] or []
try:
validate_partitions(module, partitions)
except Exception as e:
module.fail_json(msg="Malformed partitions arguments: %s" % e)
parted_args = construct_parted_args(dict(device=device, label=label,
partitions=partitions))
command = [module.get_bin_path('parted', required=True)]
if not (module.check_mode or dryrun):
command.extend(parted_args)
module.run_command(command, check_rc=True)
partitions_created = {p['name']: '%s%i' % (device, i + 1)
for i, p in enumerate(partitions)}
module.exit_json(changed=not dryrun, created=partitions_created)
from ansible.module_utils.basic import * # noqa
if __name__ == '__main__':
main()

View File

@ -5,28 +5,32 @@
- name: run parted
become: yes
parted:
ironic_parted:
device: "{{ ironic_root_device }}"
label: msdos
new_label: yes
dryrun: "{{ ironic.partition_info.preserve_ephemeral|default('no')|bool }}"
label: "{{ ironic.partition_info.label }}"
dry_run: "{{ ironic.partition_info.preserve_ephemeral|default('no')|bool }}"
partitions: "{{ ironic.partition_info.partitions }}"
register: parts
- name: reset image target to root partition
set_fact:
ironic_image_target: "{{ parts.created.root }}"
ironic_image_target: "/dev/{{ parts.created.root }}"
- name: make swap
become: yes
command: mkswap -L swap1 {{ parts.created.swap }}
command: mkswap -L swap1 /dev/{{ parts.created.swap }}
when: "{{ parts.created.swap is defined }}"
- name: format ephemeral partition
become: yes
filesystem:
dev: "{{ parts.created.ephemeral }}"
dev: "/dev/{{ parts.created.ephemeral }}"
fstype: "{{ ironic.partition_info.ephemeral_format }}"
force: yes
opts: "-L ephemeral0"
when: "{{ parts.created.ephemeral is defined and not ironic.partition_info.preserve_ephemeral|default('no')|bool }}"
- name: save block device for configdrive if partition was created
set_fact:
ironic_configdrive_target: "/dev/{{ parts.created.configdrive }}"
when: "{{ parts.created.configdrive is defined }}"

View File

@ -212,46 +212,57 @@ class TestAnsibleMethods(db_base.DbTestCase):
ansible_deploy.INVENTORY_FILE, '-e', '{"ironic": {"foo": "bar"}}',
'--private-key=/path/to/key')
def test__parse_partitioning_info_root_only(self):
def test__parse_partitioning_info_root_msdos(self):
expected_info = {
'partition_info': {
'label': 'msdos',
'partitions': [
{'name': 'root',
'size_mib': INSTANCE_INFO['root_mb'],
'boot': 'yes',
'swap': 'no'}
{'unit': 'MiB',
'size': INSTANCE_INFO['root_mb'],
'name': 'root',
'flags': {'boot': 'yes'}}
]}}
i_info = ansible_deploy._parse_partitioning_info(self.node)
self.assertEqual(expected_info, i_info)
def test__parse_partitioning_info_all(self):
def test__parse_partitioning_info_all_gpt(self):
in_info = dict(INSTANCE_INFO)
in_info['swap_mb'] = 128
in_info['ephemeral_mb'] = 256
in_info['ephemeral_format'] = 'ext4'
in_info['preserve_ephemeral'] = True
in_info['configdrive'] = 'some-fake-user-data'
in_info['capabilities'] = {'disk_label': 'gpt'}
self.node.instance_info = in_info
self.node.save()
expected_info = {
'partition_info': {
'label': 'gpt',
'ephemeral_format': 'ext4',
'preserve_ephemeral': 'yes',
'partitions': [
{'name': 'root',
'size_mib': INSTANCE_INFO['root_mb'],
'boot': 'yes',
'swap': 'no'},
{'name': 'swap',
'size_mib': 128,
'boot': 'no',
'swap': 'yes'},
{'name': 'ephemeral',
'size_mib': 256,
'boot': 'no',
'swap': 'no'},
{'unit': 'MiB',
'size': 1,
'name': 'bios',
'flags': {'bios_grub': 'yes'}},
{'unit': 'MiB',
'size': 256,
'name': 'ephemeral',
'format': 'ext4'},
{'unit': 'MiB',
'size': 128,
'name': 'swap',
'format': 'linux-swap'},
{'unit': 'MiB',
'size': 64,
'name': 'configdrive',
'format': 'fat32'},
{'unit': 'MiB',
'size': INSTANCE_INFO['root_mb'],
'name': 'root'}
]}}
i_info = ansible_deploy._parse_partitioning_info(self.node)

View File

@ -0,0 +1,32 @@
---
features:
- |
``parted`` module for Ansible was renamed to ``ironic_parted`` to not
shadow the ``parted`` module included in Ansible 2.3.
It was also rewritten to be cleaner and more stable
- changed accepted module arguments
- added support for units (MiB and % for now)
- added more validations
- left support for msdos and gpt partition tables only
- partitions are created one by one, and actual block devices created
are searched for and reported.
- |
Ansible-deploy with local-booted partition images now creates
a partitioning scheme that closer resembles what's being done by
drivers that use ``ironic-python-agent``.
- for partition images the root partition is the last one created
so that it can grow with e.g. when using cloud-init's growroot
- for whole-disk images, configdrive is created as far as possible
upgrade:
- |
``parted`` module previosly provided by ansible-deploy driver was renamed
to ``ironic_parted`` and has significantly changed its interface and
returned values.
**Any out-of-tree playbooks utilizing the ``parted`` module supplied that
was provided by ansible-deploy driver are incompatible with this release and
must be changed accordingly to use the new module name and arguments!**