openstack/openstack-ansible
Code Issues Proposed changes

Add erasure code policy support to swift

Browse Source 
This patch is version 1 (not tested) of adding erasure code support
to swift. It adds the following:

  - Add policy_type, ec_type, ec_num_data_fragments,
    ec_num_parity_fragments and ec_object_segment_size to the
    policy definition.

  - Update the ring.contents.j2 to set replica count for the ring
    to ec_num_parity_fragments + ec_object_segment_size, if using
    the erasure code policy_type.

  - Adds extra EC options to swift.conf for EC policies.

I may have missed something and again this hasn't been tested yet.

NOTE: EC in Swift is strictly _BETA_ and shouldn't be run in
      production, however, we do need to test it!

Change-Id: If2069a95e6ea92e34fb329cb6e0027188f15f0bb
This commit is contained in:
Matthew Oliver 2015-04-27 12:49:54 +10:00 committed by Kevin Carter
parent 39a62de134
commit 951fb85373
3 changed files with 54 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
etc/openstack_deploy/conf.d/swift.yml.example
View File

@ -117,6 +117,28 @@
# Policy index. One policy must include this option with a '0'
# value.
#
# Option: policy_type (optional, string)
# Policy type. Defines if this type of policy, possible values:
# - replication (default); or
# - erasure_coding
#
# Option: ec_type (optional, string)
# Defines the erasure code algorithm to use. This option is required
# when policy_type == erasure_coding.
#
# Option: ec_num_data_fragments (optional, integer)
# Defines the number of data fragments to break object into. This
# option is required when policy_type == erasure_coding.
#
# Option: ec_num_parity_fragments (optional, integer)
# Defines the number of parity fragments to break object into. This
# option is required when policy_type == erasure_coding.
#
# Option: ec_object_segment_size (optional, integer)
# Defines the segment size (btyes). Swift will send the incoming object
# data a segment size at a time in to the erasure code encoder.
# This option is required when policy_type == erasure_coding.
#
# Option: default (optional, boolean)
# Defines the default policy. One policy must include this option
# with a 'True' value.
@ -240,6 +262,14 @@
# index: 1
# repl_number: 3
# deprecated: True
# - policy:
# name: ec10-4
# index: 2
# policy_type: erasure_coding
# ec_type: jerasure_rs_vand
# ec_num_data_fragments: 10
# ec_num_parity_fragments: 4
# ec_object_segment_size: 1048576
# Proxy hosts

6
playbooks/roles/os_swift/templates/ring.contents.j2
View File

@ -22,7 +22,11 @@
{### Lets get the min_part_hours, part_power and repl_number vals #}
{% set min_part_hours = item.min_part_hours | default(swift.min_part_hours | default(swift_default_min_part_hours)) %}
{% set part_power = item.part_power | default(swift.part_power) %}
{% set repl_number = item.repl_number | default(swift.repl_number | default(swift_default_replication_number)) %}
{% if (item.policy_type is defined) and (item.policy_type == "erasure_coding") %}
{% set repl_number = item.ec_num_data_fragments + item.ec_num_parity_fragments %}
{% else %}
{% set repl_number = item.repl_number | default(swift.repl_number | default(swift_default_replication_number)) %}
{% endif %}
{### Create the builder dict #}
{% set builder = {} %}
{### This is a hacky way of updating the builder dict #}

23
playbooks/roles/os_swift/templates/swift.conf.j2
View File

@ -4,16 +4,31 @@
swift_hash_path_suffix = {{ swift_hash_path_suffix }}
swift_hash_path_prefix = {{ swift_hash_path_prefix }}
# Storage Policies
{% for policy in swift.storage_policies %}
{% set swift_policy_type = policy.policy.policy_type|default('replication') %}
[storage-policy:{{ policy.policy.index }}]
name = {{ policy.policy.name }}
{% if policy.policy.deprecated is defined %}
policy_type = {{ swift_policy_type }}
{% if swift_policy_type == 'erasure_coding' %}
ec_type = {{ policy.policy.ec_type }}
ec_num_data_fragments = {{ policy.policy.ec_num_data_fragments }}
ec_num_parity_fragments = {{ policy.policy.ec_num_parity_fragments }}
ec_object_segment_size = {{ policy.policy.ec_object_segment_size }}
{% endif %}
{% if policy.policy.deprecated is defined %}
deprecated = {{ policy.policy.deprecated }}
{% endif %}
{% if policy.policy.default is defined %}
{% endif %}
{% if policy.policy.default is defined %}
default = {{ policy.policy.default }}
{% endif %}
{% endif %}
{% endfor %}
[swift-constraints]