Add support for swift storage policies

This change implements storage policies as defined by swift:
http://docs.openstack.org/developer/swift/overview_policies.html

There are two primary areas of change in this review
- Add storage policy support to the ringbuilder and ring devices, along with
  associated spec tests.
- Adding storage policy support and enforcing rules in swift.conf using the
  swift_storage_policy type and provider. Also: updated spec tests and an
  update to swift acceptance test to use storage policies to configure an
  additional 3 replica based ring.

See release notes and the README update for more details/instructions.

Change-Id: I2b8db751790704df3f1027a14f61e231591537f3
This commit is contained in:
Adam Vinsh 2016-06-08 19:24:19 +00:00
parent 1b8869e6ae
commit 63688a14e5
26 changed files with 1966 additions and 128 deletions

135
README.md
View File

@ -6,7 +6,7 @@ swift
1. [Overview - What is the swift module?](#overview)
2. [Module Description - What does the module do?](#module-description)
3. [Setup - The basics of getting started with swift](#setup)
4. [Reference - The classes, defines,functions and facts available in this module](#reference)
4. [Usage - The classes, defines,functions and facts available in this module](#reference)
5. [Implementation - An under-the-hood peek at what the module is doing](#implementation)
6. [Limitations - OS compatibility, etc.](#limitations)
7. [Development - Guide for contributing to the module](#development)
@ -103,6 +103,139 @@ class { 'swift': swift_hash_path_suffix => 'shared_secret', }
####`swift_hash_path_suffix`
The shared salt used when hashing ring mappings.
### Swift storage policies
### Define: swift::storage::policy
A defined type that is used to configure swift storage policies as defined by swift:
http://docs.openstack.org/developer/swift/overview_policies.html
It is important for the operator to have a solid understanding of storage policies so they understand which parts of this module are needed for the result they seek.
swift::storage::policy is a wrapper to a new swift type/provider called "swift_storage_policy".
Swift storage policies are found in /etc/swift/swift.conf.
ex from swift.conf:
```puppet
[storage-policy:0]
name = Policy-0
aliases = gold, silver, bronze
policy_type = replication
default = true
[storage-policy:1]
name = policy-other
aliases = a, b, c
policy_type = replication
deprecated = No
default = false
```
The swift_storage_policy provider will manage one or more storage policy sections that can be created in swift.conf.
This provider will also enforce the following rules for swift storage policies as defined by the swift project:
http://docs.openstack.org/developer/swift/overview_policies.html#configuring-policies
- No duplicate names or aliases used across all policies.
- There is at least one policy that is marked as the default policy.
- Policy name/alias case/content.
- Policy-0 specifics.
- Deprecated and default can not be declared on the same policy.
- Storage policy policy_type.
- Policy indexes must be unique
#### How to add Policy-0 plus another policy to swift.conf
In this example we have an existing swift ring that is configured to store 1 replica of object data.
This existing ring will be considered "storage-policy:0".
The operator wants to add another storage policy to the cluster for a ring that will be configured to store 3 replicas of object data using 3 different storage devices.
The operator will need to first define storage-policy:0 to match what exists already, then the operator will need to define the new 3 replica storage policy called "storage-policy:1"
Using /spec/acceptance/basic_swift_spec.rb as an example:
The existing storage node and ringbuilder manifest will be:
```puppet
{
# Create storage policy 0 in swift.conf
swift::storage::policy { '0':
policy_name => 'Policy-0',
policy_aliases => 'basic, single, A',
default_policy => true,
policy_type => 'replication'
}
# Build the existing ring
class { '::swift::ringbuilder':
part_power => '14',
replicas => '1',
min_part_hours => 1,
}
swift::storage::node { '0':
mnt_base_dir => '/srv/node',
weight => 1,
zone => '2',
storage_local_net_ip => '127.0.0.1',
require => Swift::Storage::Loopback['2', '3', '4'] ,
}
```
To add the new ring and storage-policy:1 to swift.conf
```puppet
# Create storage policy 1 in swift.conf
swift::storage::policy { '1':
policy_name => '3-Replica-Policy',
policy_aliases => 'extra, triple, B',
default_policy => false,
deprecated => 'No',
}
# Create an object ring for nodes using policy 1
swift::ringbuilder::policy_ring { '1':
part_power => '18',
replicas => '3',
min_part_hours => 1,
}
# ring_object_devices for a storage policy start with the policy id.
# Create 3 ring_object_device starting with "1:" to be
# added to an object-1 ring for storage policy 1.
ring_object_device { "1:127.0.0.1:6000/2":
zone => 2,
weight => 1,
require => Swift::Storage::Loopback['2'],
}
ring_object_device { "1:127.0.0.1:6000/3":
zone => 2,
weight => 1,
require => Swift::Storage::Loopback['3'] ,
}
ring_object_device { "1:127.0.0.1:6000/4":
zone => 2,
weight => 1,
require => Swift::Storage::Loopback['4'] ,
}
```
To remove any section from a storage policy, just set its value to undef.
To remove a storage policy section completely set it to ensure => absent
This will remove the section AND section header.
ex:
```puppet
# Purge storage policy 1 entirely from swift.conf
swift::storage::policy { '1':
ensure => absent,
policy_name => '3-Replica-Policy',
policy_aliases => 'extra, triple, B',
default_policy => false,
deprecated => 'No',
}
```
See swift::storage::policy for additional parameters to set.
#### Storage policies and erasure code support.
Support for erasure code using storage policies is supported using swift::storage::policy.
A future change will enable the swift-object-reconstructor process that is needed for a
cluster that runs erasure code.
### Class swift::proxy
Class that installs and configures the swift proxy server.

View File

@ -6,18 +6,15 @@ Puppet::Type.type(:ring_account_device).provide(
optional_commands :swift_ring_builder => 'swift-ring-builder'
def self.prefetch(resource)
def prefetch(resource)
@my_ring = lookup_ring
end
def self.ring
def ring
@my_ring ||= lookup_ring
end
# TODO maybe this should be a parameter eventually so that
# it can be configurable
def self.builder_file_path
'/etc/swift/account.builder'
def self.builder_file_path(policy_index)
'/etc/swift/account.builder'
end
end

View File

@ -6,18 +6,16 @@ Puppet::Type.type(:ring_container_device).provide(
optional_commands :swift_ring_builder => 'swift-ring-builder'
def self.prefetch(resource)
def prefetch(resource)
@my_ring = lookup_ring
end
def self.ring
def ring
@my_ring ||= lookup_ring
end
# TODO maybe this should be a parameter eventually so that
# it can be configurable
def self.builder_file_path
'/etc/swift/container.builder'
def self.builder_file_path(policy_index)
'/etc/swift/container.builder'
end
end

View File

@ -6,18 +6,21 @@ Puppet::Type.type(:ring_object_device).provide(
optional_commands :swift_ring_builder => 'swift-ring-builder'
def self.prefetch(resource)
def prefetch(resource)
@my_ring = lookup_ring
end
def self.ring
def ring
@my_ring ||= lookup_ring
end
# TODO maybe this should be a parameter eventually so that
# it can be configurable
def self.builder_file_path
'/etc/swift/object.builder'
def self.builder_file_path(policy_index)
if policy_index.nil?
'/etc/swift/object.builder'
elsif policy_index == 0
'/etc/swift/object.builder'
else
"/etc/swift/object-#{policy_index}.builder"
end
end
end

View File

@ -8,7 +8,7 @@ class Puppet::Provider::SwiftRingBuilder < Puppet::Provider
end
end
def self.address_string(address)
def address_string(address)
ip = IPAddr.new(address)
if ip.ipv6?
'[' + ip.to_s + ']'
@ -17,11 +17,11 @@ class Puppet::Provider::SwiftRingBuilder < Puppet::Provider
end
end
def self.lookup_ring
def lookup_ring
object_hash = {}
if File.exists?(builder_file_path)
if File.exists?(builder_file_path(policy_index))
# Swift < 2.2.2 Skip first 4 info lines from swift-ring-builder output
if rows = swift_ring_builder(builder_file_path).split("\n")[4..-1]
if rows = swift_ring_builder(builder_file_path(policy_index)).split("\n")[4..-1]
# Skip "Ring file ... is up-to-date" message, if printed.
if !rows[0].nil? and rows[0] =~ /Ring file\b.*\bis up-to-date/
rows.shift
@ -71,21 +71,32 @@ class Puppet::Provider::SwiftRingBuilder < Puppet::Provider
# Swift 2.9.1+ output example:
if row =~ /^\s*(\d+)\s+(\d+)\s+(\d+)\s+(\S+):(\d+)\s+\S+:\d+\s+(\S+)\s+(\d+\.\d+)\s+(\d+)\s*((-|\s-?)?\d+\.\d+)\s*(\S*)/
address = address_string("#{$4}")
object_hash["#{address}:#{$5}/#{$6}"] = {
:id => $1,
:region => $2,
:zone => $3,
:weight => $7,
:partitions => $8,
:balance => $9,
:meta => $11
if !policy_index.nil?
policy = "#{policy_index}:"
else
policy = ''
end
object_hash["#{policy}#{address}:#{$5}/#{$6}"] = {
:id => $1,
:region => $2,
:zone => $3,
:weight => $7,
:partitions => $8,
:balance => $9,
:meta => $11,
:policy_index => "#{policy_index}"
}
# Swift 1.8+ output example:
elsif row =~ /^\s*(\d+)\s+(\d+)\s+(\d+)\s+(\S+)\s+(\d+)\s+\S+\s+\d+\s+(\S+)\s+(\d+\.\d+)\s+(\d+)\s*((-|\s-?)?\d+\.\d+)\s*(\S*)/
address = address_string("#{$4}")
object_hash["#{address}:#{$5}/#{$6}"] = {
if !policy_index.nil?
policy = "#{policy_index}:"
else
policy = ''
end
object_hash["#{policy}#{address}:#{$5}/#{$6}"] = {
:id => $1,
:region => $2,
:zone => $3,
@ -121,7 +132,6 @@ class Puppet::Provider::SwiftRingBuilder < Puppet::Provider
:balance => $8,
:meta => $9
}
else
Puppet.warning("Unexpected line: #{row}")
end
@ -131,12 +141,8 @@ class Puppet::Provider::SwiftRingBuilder < Puppet::Provider
object_hash
end
def ring
self.class.ring
end
def builder_file_path
self.class.builder_file_path
def builder_file_path(policy_index)
self.class.builder_file_path(policy_index)
end
def exists?
@ -147,13 +153,12 @@ class Puppet::Provider::SwiftRingBuilder < Puppet::Provider
[:zone, :weight].each do |param|
raise(Puppet::Error, "#{param} is required") unless resource[param]
end
if :region == 'none'
# Prior to Swift 1.8.0, regions did not exist.
swift_ring_builder(
builder_file_path,
builder_file_path(policy_index),
'add',
"z#{resource[:zone]}-#{resource[:name]}_#{resource[:meta]}",
"z#{resource[:zone]}-#{device_path}_#{resource[:meta]}",
resource[:weight]
)
else
@ -161,14 +166,30 @@ class Puppet::Provider::SwiftRingBuilder < Puppet::Provider
# Region defaults to 1 if unspecified
resource[:region] ||= 1
swift_ring_builder(
builder_file_path,
builder_file_path(policy_index),
'add',
"r#{resource[:region]}z#{resource[:zone]}-#{resource[:name]}_#{resource[:meta]}",
"r#{resource[:region]}z#{resource[:zone]}-#{device_path}_#{resource[:meta]}",
resource[:weight]
)
end
end
def device_path
if resource[:name].split(/^\d+:/)[1].nil?
return resource[:name]
else
return resource[:name].split(/^\d+:/)[1]
end
end
def policy_index
if resource[:name].split(/^\d+:/)[1].nil?
return nil
else
Integer("#{resource[:name].match(/^\d+/)}")
end
end
def id
ring[resource[:name]][:id]
end
@ -203,7 +224,7 @@ class Puppet::Provider::SwiftRingBuilder < Puppet::Provider
swift_ring_builder(
builder_file_path,
'set_weight',
"d#{ring[resource[:name]][:id]}",
"d#{ring[device_path][:id]}",
resource[:weight]
)
# requires a rebalance
@ -233,10 +254,9 @@ class Puppet::Provider::SwiftRingBuilder < Puppet::Provider
swift_ring_builder(
builder_file_path,
'set_info',
"d#{ring[resource[:name]][:id]}",
"d#{ring[device_path][:id]}",
"_#{resource[:meta]}"
)
end
end

View File

@ -0,0 +1,235 @@
# provider for swift_storage_policy resource.
#
Puppet::Type.type(:swift_storage_policy).provide(:ruby) do
mk_resource_methods
# Policy indexes must be unique, the storage policy resource name is its ID.
# Storage policy section headings in swift.conf are of the format "storage-policy:<policy ID>
def policy_title
"storage-policy:#{name}"
end
def create
@property_flush[:ensure] = :present
end
def destroy
@property_flush[:ensure] = :absent
end
def exists?
# If a storage policy exists if it is ensured absent and is also found in swift.conf.
if (resource[:ensure] == :absent) &&
(self.class.storage_policies.include? policy_title)
return true
end
if self.class.storage_policies.include? policy_title
# Return false if settings are removed from swift.conf that do not exist in the puppet resource.
# This resource will then update swift.conf in flush.
return false if remove_storage_policy_settings?
end
@property_hash[:ensure] == :present
end
def flush
# flush is called when policy state in swift.conf does not match what is defined in puppet.
# get all resource settings and store in @property_flush for use in write_policy
# @property_flush is the state the policy should be in.
# @property_hash is the state of the policy in swift.conf
self.class.policy_settings.each do |property_name, _|
@property_flush[property_name] = resource[property_name]
end
# If this policy is set to absent, call write_policy to remove it from swift.conf.
if @property_flush[:ensure] == :absent
write_policy
return
end
# At this point, this is a new policy to write to disk. Or this is a policy
# containing a setting that needs to be updated on disk.
# Write this policy unless:
# - a policy on disk already has this same policy_name or aliases.
# or a policy default setting conflict is found.
if !self.class.instances.empty?
write_policy unless policy_names_conflict? ||
default_policy_defined?
else
write_policy
end
@property_hash = self.class.storage_policy_properties(policy_title)
end
# A hash of property_name => setting_name. property_name used to access
# resource properties. setting_name used in swift.conf storage policy setting.
def self.policy_settings
settings_hash = { policy_name: 'name',
aliases: 'aliases',
policy_type: 'policy_type',
deprecated: 'deprecated',
default: 'default',
ec_type: 'ec_type',
ec_num_data_fragments: 'ec_num_data_fragments',
ec_num_parity_fragments: 'ec_num_parity_fragments',
ec_object_segment_size: 'ec_object_segment_size' }
settings_hash
end
# Read storage policies from disk found in swift.conf
def self.storage_policies
policies = []
config.section_names.each do |section_name|
policies.push(section_name) if section_name.start_with?('storage-policy')
end
policies.sort
policies
end
# Read storage policy settings and values found swift.conf
def self.storage_policy_properties(policy)
policy_properties = {}
policy_properties[:provider] = :ruby
policy_properties[:name] = policy.split(':')[1]
policy_properties[:ensure] = :present
policy_settings.each do |property_name, setting_name|
unless config.get_value(policy, setting_name).nil?
policy_properties[property_name] = config.get_value(policy, setting_name)
end
end
policy_properties
end
def write_policy
if @property_flush[:ensure] == :absent
remove_storage_policy_section
return
end
self.class.policy_settings.each do |property_name, setting_name|
unless @property_flush[property_name].nil?
config.set_value(policy_title, setting_name, @property_flush[property_name])
end
end
config.save
@config = nil
end
# Removes the entire storage policy section header and settings for this instance.
# TODO push a 'remove_section' method into puppet inifile module for the ini_file util.
def remove_storage_policy_section
# Get all settings for this storage policy section and remove them.
@sections = config.instance_variable_get(:@sections_hash)
@sections[policy_title].instance_variable_get(:@existing_settings).each do |setting, _|
config.remove_setting(policy_title, setting)
end
# ini_file tracks an array of section names 'section_names' and a hash of sections 'sections_hash'
# get array of section names and delete this storage policy from it.
@section_names = config.instance_variable_get(:@section_names)
@section_names.delete(policy_title)
# delete the entire section from the sections_hash then save swift.conf
@sections.delete(policy_title)
config.save
end
# Check for storage policy settings found in swift.conf that are not defined
# in puppet and should be removed from the policy. Return True if settings
# were removed. Removing a setting from a policy declaration will remove that
# setting from swift.conf
def remove_storage_policy_settings?
settings_removed = false
# If storage policy settings are found in swift.conf that are not defined,
# remove the setting line.
self.class.storage_policy_properties(policy_title).each do |key, value|
next unless @resource[key] != value
config.remove_setting(policy_title, key.to_s.delete(':'))
config.save
settings_removed = true
end
@config = nil
settings_removed
end
# Compare current policy alias against existing storage policies in swift.conf
# Duplicate alias are not allowed.
def policy_names_conflict?
self.class.instances.each do |policy|
next if policy.policy_title.eql? "storage-policy:#{name}"
# Split policy aliases into an array and add policy name.
# Split resource aliases into an array and add resource policy_name.
# If any intersecting elements exist raise an error alerting on the
# attempt to set a duplicate name/alias.
policy_names = policy.policy_name.split
unless policy.aliases == :absent
policy_names = policy.aliases.split(', ').concat policy.policy_name.split
end
resource_names = resource[:policy_name].split
unless resource[:aliases].nil?
resource_names = resource[:aliases].split(', ').concat resource[:policy_name].split
end
alias_intersection = policy_names & resource_names
next if alias_intersection.empty?
raise Puppet::Error, "Swift_storage_policy[#{resource[:name]}] trying "\
"to set a duplicate name/alias:#{alias_intersection},"\
"this name/alias exists in #{policy.policy_title}\n"
end
false
end
# Check storage policy default settings across resources to verify that:
# - If any policies are defined, exactly one policy must be declared default.
# - Only one policy can be declared the default.
def default_policy_defined?
self.class.instances.each do |policy|
# Don't compare this instance with its copy found on disk.
next if (policy.policy_title.eql? "storage-policy:#{name}") ||
(policy.default == 'false')
case resource[:default]
when 'true'
raise Puppet::Error, "Swift_storage_policy[#{resource[:name]}] can "\
'not set default = true. '\
"default=true already set in a policy #{policy.policy_title}\n"
when 'false'
return false
end
end
return unless resource[:default] == 'false'
raise Puppet::Error, 'All storage policies have set default = false.. '\
'exactly one policy must be declared default = true'
end
def self.instances
storage_policies.collect do |policy|
policy_properties = storage_policy_properties(policy)
new(policy_properties)
end
end
def self.prefetch(resources)
instances.each do |prov|
if resource = resources[prov.name]
resource.provider = prov
end
end
end
def initialize(value = {})
super(value)
@property_flush = {}
end
private
def self.get_swift_conf_file
if File.exists? '/etc/swift/swift.conf'
file = '/etc/swift/swift.conf'
else
file = '/etc/swift/swift.conf'
end
file
end
def config
self.class.config
end
def self.config
@config ||= Puppet::Util::IniFile.new(get_swift_conf_file)
end
end

View File

@ -6,6 +6,9 @@ Puppet::Type.newtype(:ring_account_device) do
newparam(:name, :namevar => true) do
validate do |value|
if !value.split(/^\d+:/)[1].nil?
raise(Puppet::Error, "Policy_index is not supported on account device")
end
# we have to have URI Scheme so we just add http:// and ignore it later
uri = URI('http://' + value)
address = uri.host

View File

@ -6,6 +6,9 @@ Puppet::Type.newtype(:ring_container_device) do
newparam(:name, :namevar => true) do
validate do |value|
if !value.split(/^\d+:/)[1].nil?
raise(Puppet::Error, "Policy_index is not supported on container device")
end
# we have to have URI Scheme so we just add http:// and ignore it later
uri = URI('http://' + value)
address = uri.host

View File

@ -6,6 +6,11 @@ Puppet::Type.newtype(:ring_object_device) do
newparam(:name, :namevar => true) do
validate do |value|
# If storage policy_index is specified first strip that off.
# Resource name is not required to start with a policy_index
if !value.split(/^\d+:/)[1].nil?
value = value.split(/^\d+:/)[1]
end
# we have to have URI Scheme so we just add http:// and ignore it later
uri = URI('http://' + value)
address = uri.host
@ -26,7 +31,7 @@ Puppet::Type.newtype(:ring_object_device) do
"%.2f" % value
end
end
newproperty(:meta)
[:id, :partitions, :balance].each do |param|

View File

@ -0,0 +1,69 @@
Puppet::Type.newtype(:swift_storage_policy) do
ensurable
newparam(:name, :namevar => :true) do
desc 'Storage policy id digit'
validate do |value|
value_match = /\d+/.match(value)
next unless value_match.nil? || !value_match[0].eql?(value)
fail('swift_storage_policy name must be a postive integer')
end
newvalues(/\d+/)
end
newproperty(:policy_name) do
desc 'Storage policy_name. Policy names are case insensitive, must contain only letters,'\
' digits or a dash and must be unique.'
validate do |value|
value_match = /[a-zA-Z\d\-]+/.match(value)
next unless value_match.nil? || !value_match[0].eql?(value)
fail('policy_name must contain only letters, digits or a dash')
end
newvalues(/[a-zA-Z\d\-]+/)
end
newproperty(:aliases) do
desc 'Storage policy aliases'
validate do |value|
value_match = /([a-zA-Z\d\-]+,\s+)+[a-zA-Z\d\-]+/.match(value)
next unless value_match.nil? || !value_match[0].eql?(value)
fail('aliases must contain only letters, digits or a dash in a comma separated string')
end
newvalues(/([a-zA-Z\d\-]+,\s)+/)
end
newproperty(:policy_type) do
desc 'Type of storage policy, ec or replication'
newvalues(/(replication)|(erasure_coding)/i)
end
newproperty(:default) do
desc 'Set default on storage policy'
newvalues(/(true)|(false)/)
end
newproperty(:deprecated) do
desc 'Set to yes to mark a policy as deprecated'
newvalues(/(yes)|(no)/i)
end
newproperty(:ec_type) do
desc 'Type of erasure code to use'
newvalues(/\w+/)
end
newproperty(:ec_num_data_fragments) do
desc 'The total number of fragments that will be comprised of data.'
newvalues(/\d+/)
end
newproperty(:ec_num_parity_fragments) do
desc 'The total number of fragments that will be comprised of parity'
newvalues(/\d+/)
end
newproperty(:ec_object_segment_size) do
desc 'The amount of data that will be buffered up before feeding a segment into the encoder/decoder'
newvalues(/\d+/)
end
end

View File

@ -25,6 +25,10 @@ class swift::deps {
-> Swift_proxy_config<||>
~> Anchor['swift::config::end']
Anchor['swift::config::begin']
-> Swift_storage_policy<||>
~> Anchor['swift::config::end']
Anchor['swift::config::begin']
-> Swift_object_config<||>
~> Anchor['swift::config::end']

View File

@ -0,0 +1,58 @@
# Used to build an aditional object ring for a storage policy.
# The namevar/name of this class must be an integer.
#
#
# Specifies the following relationship:
# Rings should be created before any devices are added to them
# Rings should be rebalanced if anything changes
# == Parameters
# [*title*] required. Title must be a positive integer. Title of this class
# is used to denote the storage policy ID for the object ring.
#
# [*part_power*] The total number of partitions that should exist in the ring.
# This is expressed as a power of 2.
# [*replicas*] Numer of replicas that should be maintained of each stored object.
# [*min_part_hours*] Minimum amount of time before partitions can be moved.
#
# == Dependencies
#
# Class['swift']
#
# == Examples
#
# == Authors
#
# Dan Bode dan@puppetlabs.com
# Adam Vinsh adam.vinsh@charter.com
#
# == Copyright
#
# Copyright 2011 Puppetlabs Inc, unless otherwise noted.
#
define swift::ringbuilder::policy_ring(
$part_power = undef,
$replicas = undef,
$min_part_hours = undef,
) {
validate_re($title, '^\d+$')
include ::swift::deps
Class['swift'] -> Swift::Ringbuilder::Policy_ring[$title]
if $title == '0' {
$ring_builder = 'object'
} else {
$ring_builder = "object-${title}"
}
swift::ringbuilder::create{ $ring_builder :
part_power => $part_power,
replicas => $replicas,
min_part_hours => $min_part_hours,
}
swift::ringbuilder::rebalance{ $ring_builder: }
Swift::Ringbuilder::Create[$ring_builder] -> Ring_object_device <| |> ~> Swift::Ringbuilder::Rebalance[$ring_builder]
}

View File

@ -38,11 +38,9 @@
# (optional) The IP address of the storage server.
# Defaults to '127.0.0.1'.
#
# ==== DEPRECATED PARAMETERS
#
# [*manage_ring*]
# This parameter is deprecated and does nothing.
#
# [*policy_index*]
# (optional) storage policy index
# Defaults to undef
define swift::storage::node(
$mnt_base_dir,
$zone,
@ -51,8 +49,7 @@ define swift::storage::node(
$group = 'swift',
$max_connections = 25,
$storage_local_net_ip = '127.0.0.1',
# DEPRECATED PARAMETERS
$manage_ring = true
$policy_index = undef,
) {
include ::swift::deps
@ -71,7 +68,14 @@ define swift::storage::node(
type => 'object',
config_file_path => 'object-server.conf',
}
ring_object_device { "${storage_local_net_ip}:60${name}0/${name}":
if !$policy_index {
$ring_device = "${storage_local_net_ip}:60${name}0/${name}"
} else {
$ring_device = "${policy_index}:${storage_local_net_ip}:60${name}0/${name}"
}
ring_object_device { $ring_device:
zone => $zone,
weight => $weight,
}

View File

@ -0,0 +1,99 @@
# Class swift::storage::policy
#
# Setting any optional parameter to undef will remove it
# from the storage policy defined in swift.conf.
#
# == Parameters
# [*ensure*]
# (optional) To ensure a storage policy exists in swift.conf
# set to 'present'. To remove a storage policy from swift.conf
# set to 'absent'.
# Defaults to 'present'
#
# [*policy_name*]
# (required) Storage policy name in swift.conf
# Names and aliases must be unique across all policies.
#
# [*default_policy*]
# (optional) Boolean to specify if this is the default storage policy
# Defaults to true
#
# [*policy_aliases*]
# (optional) A comma separated string of aliases to use for this storage
# policy. ex: 'gold, silver, taco' Names and aliases must be unique across
# all policies.
# Defaults to undef.
#
# [*policy_index*]
# (required) storage policy index. Becomes a storage policy section
# header in swift.conf.
# Defaults to '0'
#
# [*policy_type*]
# (required) Storage policy type. Can be 'replication' or 'erasure_coding'
# Defaults 'replication'.
#
# [*deprecated*]
# (optional) Any policy may be deprecated by setting deprecated = yes.
# Choices are 'yes', 'no', undef
# Defaults to undef
#
# [*ec_type*]
# (optional) Specifies the EC scheme that is to be used
# Defaults to undef
#
# [*ec_num_data_fragments*]
# (optional) The total number of fragments that will be
# comprised of data.
# Defaults to undef
#
# [*ec_num_parity_fragments*]
# (optional) The total number of fragments that will be
# comprised of parity.
# Defaults to undef
#
# [*ec_object_segment_size*]
# (optional) The amount of data that will be buffered up before
# feeding a segment into the encoder/decoder in bytes.
# Defaults to undef
#
define swift::storage::policy(
$policy_name,
$default_policy,
$ensure = 'present',
$policy_aliases = undef,
$policy_index = $name,
$policy_type = 'replication',
$deprecated = undef,
$ec_type = undef,
$ec_num_data_fragments = undef,
$ec_num_parity_fragments = undef,
$ec_object_segment_size = undef,
) {
include ::swift::deps
Swift_storage_policy<| |> ~> Service<| tag == 'swift-service' |>
if (downcase($policy_name) == 'policy-0') and ($policy_index != '0') {
fail('The name Policy-0 can only be used with policy index 0')
}
if ($default_policy == true) and ($deprecated == 'yes') {
fail('a deprecated policy may not also be declared the default')
}
swift_storage_policy { $policy_index:
ensure => $ensure,
policy_name => $policy_name,
aliases => $policy_aliases,
policy_type => $policy_type,
default => bool2str($default_policy),
deprecated => $deprecated,
ec_type => $ec_type,
ec_num_data_fragments => $ec_num_data_fragments,
ec_num_parity_fragments => $ec_num_parity_fragments,
ec_object_segment_size => $ec_object_segment_size,
}
}

View File

@ -0,0 +1,36 @@
---
prelude: >
Add support for swift storage policies.
This change adds storage policy support to the
swift ringbuilder class as well as ring_object_device.
This change also adds a new custom type/provider
called swift_storage_policy that is used to create
and enforce rules for storage policies in swift.conf
features:
- Add support for swift storage policies.
This change adds storage policy support to the
swift ringbuilder class as well as ring_object_device.
The swift ringbuilder provider was modified to accept
ring_object_device with a name that starts with an
integer followed by colon. For example, a ring_object_device
without a storage policy would be named 127.0.0.1:6000/4
A ring_object_device that should be included in
storage-policy:1 would be 1:127.0.0.1:6000/4.
Spec tests were split up and updated to test the
changes to ring_object_device as well.
- This change also adds a new custom type/provider
called swift_storage_policy that is used to create
and enforce rules for storage policies in swift.conf
This provider enforces rules established by the swift
project for storage_policies. This provider uses the
puppet inifile provider to control storage policy
entries in swift.conf. This provider implements a way
to remove/purge a storage policy including it's section
header from swift.conf.
- An upcoming change will enable the use of erasure code
through swift storage policies.
upgrade:
- No action is required by existing users of this
module. To begin using storage policies follow the
example upgrade procedure in the README under the
swift storage policy section.

View File

@ -32,25 +32,62 @@ describe 'basic swift' do
storage_local_net_ip => '127.0.0.1',
}
# create xfs partitions on a loopback device and mounts them
swift::storage::loopback { '2':
swift::storage::loopback { ['2','3','4']:
seek => '200000',
require => Class['swift'],
}
# Create storage policy 0 in swift.conf
swift::storage::policy { '0':
policy_name => 'Policy-0',
policy_aliases => 'basic, single, A',
default_policy => true,
policy_type => 'replication'
}
# Create storage policy 1 in swift.conf
swift::storage::policy { '1':
policy_name => '3-Replica-Policy',
policy_aliases => 'extra, triple, B',
default_policy => false,
deprecated => 'No',
}
# sets up storage nodes which is composed of a single
# device that contains an endpoint for an object, account, and container
swift::storage::node { '2':
mnt_base_dir => '/srv/node',
weight => 1,
manage_ring => true,
zone => '2',
storage_local_net_ip => '127.0.0.1',
require => Swift::Storage::Loopback['2'] ,
policy_index => 0,
require => Swift::Storage::Loopback['2','3','4'] ,
}
# ring_object_devices for a storage policy start with the policy id.
# Create 3 ring_object_device starting with "1:" to be
# added to an object-1 ring for storage policy 1.
ring_object_device { "1:127.0.0.1:6000/2":
zone => 2,
weight => 1,
require => Swift::Storage::Loopback['2'],
}
ring_object_device { "1:127.0.0.1:6000/3":
zone => 2,
weight => 1,
require => Swift::Storage::Loopback['3'] ,
}
ring_object_device { "1:127.0.0.1:6000/4":
zone => 2,
weight => 1,
require => Swift::Storage::Loopback['4'] ,
}
class { '::swift::ringbuilder':
part_power => '18',
replicas => '1',
min_part_hours => 1,
}
swift::ringbuilder::policy_ring { '1':
part_power => '18',
replicas => '3',
min_part_hours => 1,
}
class { '::swift::proxy':
proxy_local_net_ip => '127.0.0.1',
pipeline => ['healthcheck', 'proxy-logging', 'cache', 'authtoken', 'keystone', 'dlo', 'proxy-server'],
@ -113,19 +150,51 @@ describe 'basic swift' do
storage_local_net_ip => '127.0.0.1',
}
# create xfs partitions on a loopback device and mounts them
swift::storage::loopback { '2':
swift::storage::loopback { ['2','3','4']:
seek => '200000',
require => Class['swift'],
}
# Create storage policy 0 in swift.conf
swift::storage::policy { '0':
policy_name => 'Policy-0',
policy_aliases => 'basic, single, A',
default_policy => true,
policy_type => 'replication'
}
# Create storage policy 1 in swift.conf
swift::storage::policy { '1':
policy_name => '3-Replica-Policy',
policy_aliases => 'extra, triple, B',
default_policy => false,
deprecated => 'No',
}
# sets up storage nodes which is composed of a single
# device that contains an endpoint for an object, account, and container
swift::storage::node { '2':
mnt_base_dir => '/srv/node',
weight => 1,
manage_ring => true,
zone => '2',
storage_local_net_ip => '127.0.0.1',
require => Swift::Storage::Loopback['2'] ,
policy_index => 0,
require => Swift::Storage::Loopback['2','3','4'] ,
}
# ring_object_devices for a storage policy start with the policy id.
# Create 3 ring_object_device starting with "1:" to be
# added to an object-1 ring for storage policy 1.
ring_object_device { "1:127.0.0.1:6000/2":
zone => 2,
weight => 1,
require => Swift::Storage::Loopback['2'],
}
ring_object_device { "1:127.0.0.1:6000/3":
zone => 2,
weight => 1,
require => Swift::Storage::Loopback['3'] ,
}
ring_object_device { "1:127.0.0.1:6000/4":
zone => 2,
weight => 1,
require => Swift::Storage::Loopback['4'] ,
}
class { '::swift::storage::account':
service_provider => 'swiftinit',
@ -141,6 +210,11 @@ describe 'basic swift' do
replicas => '1',
min_part_hours => 1,
}
swift::ringbuilder::policy_ring { '1':
part_power => '18',
replicas => '3',
min_part_hours => 1,
}
class { '::swift::proxy':
proxy_local_net_ip => '127.0.0.1',
pipeline => ['healthcheck', 'proxy-logging', 'cache', 'authtoken', 'keystone', 'dlo', 'proxy-server'],

View File

@ -0,0 +1,78 @@
require 'spec_helper'
describe 'swift::ringbuilder::policy_ring' do
let :title do
"1"
end
let :facts do
OSDefaults.get_facts({
:operatingsystem => 'Ubuntu',
:osfamily => 'Debian',
:os_workers => 1,
})
end
describe 'when swift class is not included' do
it 'should fail' do
expect { catalogue }.to raise_error(Puppet::Error)
end
end
describe 'when swift class is included and policy is >= 1' do
let :pre_condition do
"class { memcached: max_memory => 1}
class { swift: swift_hash_path_suffix => string }"
end
it 'should rebalance the object ring' do
is_expected.to contain_swift__ringbuilder__rebalance('object-1')
end
describe 'with default parameters' do
it { is_expected.to contain_swift__ringbuilder__create('object-1').with(
:part_power => '18',
:replicas => '3',
:min_part_hours => '24'
)}
end
describe 'with parameter overrides' do
let :params do
{:part_power => '19',
:replicas => '3',
:min_part_hours => '2'
}
end
it { is_expected.to contain_swift__ringbuilder__create('object-1').with(
:part_power => '19',
:replicas => '3',
:min_part_hours => '2'
)}
end
describe 'when specifying ring devices' do
let :pre_condition do
'class { memcached: max_memory => 1}
class { swift: swift_hash_path_suffix => string }
ring_object_device { "1:127.0.0.1:6000/1":
zone => 1,
weight => 1,
}'
end
it 'should set up all of the correct dependencies' do
is_expected.to contain_swift__ringbuilder__create('object-1').with(
{:before => ['Ring_object_device[1:127.0.0.1:6000/1]']}
)
is_expected.to contain_ring_object_device('1:127.0.0.1:6000/1').with(
{:notify => ['Swift::Ringbuilder::Rebalance[object-1]']}
)
end
end
end
end

View File

@ -1,3 +1,4 @@
require 'spec_helper'
describe 'swift::storage::node' do
let :facts do
@ -8,27 +9,29 @@ describe 'swift::storage::node' do
})
end
let :params do
{
:zone => "1",
:mnt_base_dir => '/srv/node'
}
end
let :title do
"1"
end
describe 'with valid preconditons should contain ring devices' do
let :params do
{
:zone => "1",
:mnt_base_dir => '/srv/node'
}
end
let :pre_condition do
"class { 'swift': swift_hash_path_suffix => 'foo' }
class { 'swift::storage': storage_local_net_ip => '127.0.0.1' }"
end
let :title do
"1"
end
it {
is_expected.to contain_ring_object_device("127.0.0.1:6010/1")
is_expected.to contain_ring_container_device("127.0.0.1:6011/1")
is_expected.to contain_ring_account_device("127.0.0.1:6012/1")
}
let :pre_condition do
"class { 'swift': swift_hash_path_suffix => 'foo' }
class { 'swift::storage': storage_local_net_ip => '127.0.0.1' }"
end
it { is_expected.to contain_ring_object_device("127.0.0.1:6010/1") }
it { is_expected.to contain_ring_container_device("127.0.0.1:6011/1") }
it { is_expected.to contain_ring_account_device("127.0.0.1:6012/1") }
end
context 'when zone is not a number' do
let(:title) { '1' }
@ -39,4 +42,27 @@ describe 'swift::storage::node' do
it_raises 'a Puppet::Error', /The zone parameter must be an integer/
end
describe 'with valid preconditons and policy_index=1 should contain ring devices' do
let :params do
{
:zone => "1",
:mnt_base_dir => '/srv/node',
:policy_index => '1',
}
end
let :title do
"1"
end
let :pre_condition do
"class { 'swift': swift_hash_path_suffix => 'foo' }
class { 'swift::storage': storage_local_net_ip => '127.0.0.1' }"
end
it { is_expected.to contain_ring_object_device("1:127.0.0.1:6010/1") }
it { is_expected.to contain_ring_container_device("127.0.0.1:6011/1") }
it { is_expected.to contain_ring_account_device("127.0.0.1:6012/1") }
end
end

View File

@ -1,21 +1,36 @@
require 'puppet'
require 'mocha'
require File.join(File.dirname(__FILE__), '..', '..', '..', '..', 'lib', 'puppet', 'provider', 'swift_ring_builder')
require File.join(File.dirname(__FILE__), '..', '..', '..', '..', '..', 'lib', 'puppet', 'provider', 'ring_account_device', 'swift_ring_builder')
RSpec.configure do |config|
config.mock_with :mocha
end
provider_class = Puppet::Provider::SwiftRingBuilder
provider_class = Puppet::Type.type(:ring_account_device).provider(:swift_ring_builder)
describe provider_class do
it "should have the SwiftRingBuilder provider class as its baseclass" do
expect(provider_class.superclass).to equal(Puppet::Provider::SwiftRingBuilder)
end
let :builder_file_path do
'/etc/swift/account.builder'
end
it 'should be able to lookup the local ring and build an object 2.9.1+' do
File.expects(:exists?).with(builder_file_path).returns(true)
provider_class.expects(:builder_file_path).twice.returns(builder_file_path)
# Swift 2.9.1+ output
provider_class.expects(:swift_ring_builder).returns(
let :account_builder do
Tempfile.new('account.builder')
end
let :provider do
described_class.new(
Puppet::Type.type(:ring_account_device).new(
:name => '192.168.101.13:6002/1',
:zone => '1',
:weight => '1',
:provider => :swift_ring_builder
)
)
end
let :account_builder_output_2_9_1 do
'/etc/swift/account.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
@ -26,35 +41,63 @@ Devices: id region zone ip address:port replic_ip:replic_port
0 1 3 192.168.101.15:6002 192.168.101.15:6002 1 1.00 262144-100.00 m2
3 1 1 192.168.101.16:6002 192.168.101.16:6002 1 1.00 262144-100.00
'
)
resources = provider_class.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.16:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '200.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql 'm2'
end
describe "with no storage policy_index set on swift 2.9.1+" do
it 'account builder file should be account.builder when object name has no policy_index' do
policy_index = provider.policy_index
expect(provider.builder_file_path(policy_index)).to eq builder_file_path
end
it 'ring_account_device should exist when found in builder file' do
provider.class.stubs(:swift_ring_builder).returns account_builder_output_2_9_1
File.expects(:exists?).with(builder_file_path).returns(true)
expect(provider.exists?).to eq({:id=>"1", :region=>"1", :zone=>"1", :weight=>"1.00", :partitions=>"262144", :balance=>"0.00", :meta=>"", :policy_index=>''})
end
it 'should be able to lookup the local ring' do
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
provider.expects(:swift_ring_builder).returns account_builder_output_2_9_1
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.16:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.13:6002/1'][:policy_index]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '200.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql 'm2'
expect(resources['192.168.101.14:6002/1'][:policy_index]).to eql ''
end
end
describe "with a storage policy_index set on swift 2.9.1+" do
it 'ring_account_device should fail when object name has a policy_index' do
expect {
Puppet::Type.type(:ring_account_device).new(:name => "1:192.168.101.13:6002/1", :zone => '1', :weight => '1', :provider => :swift_ring_builder)}.to raise_error(Puppet::Error, /Policy_index is not supported on account device/)
end
end
it 'should be able to lookup the local ring and build an object 2.2.2+' do
File.expects(:exists?).with(builder_file_path).returns(true)
provider_class.expects(:builder_file_path).twice.returns(builder_file_path)
# Swift 1.8 output
provider_class.expects(:swift_ring_builder).returns(
provider.expects(:swift_ring_builder).returns(
'/etc/swift/account.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
@ -66,7 +109,9 @@ Devices: id region zone ip address port replication ip replicat
3 1 1 192.168.101.16 6002 192.168.101.16 6002 1 1.00 262144-100.00
'
)
resources = provider_class.lookup_ring
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
@ -90,9 +135,7 @@ Devices: id region zone ip address port replication ip replicat
end
it 'should be able to lookup the local ring and build an object 1.8+' do
File.expects(:exists?).with(builder_file_path).returns(true)
provider_class.expects(:builder_file_path).twice.returns(builder_file_path)
# Swift 1.8 output
# Swift 1.8+ output
provider_class.expects(:swift_ring_builder).returns(
'/etc/swift/account.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
@ -104,7 +147,9 @@ Devices: id region zone ip address port replication ip replicat
3 1 1 192.168.101.16 6002 192.168.101.16 6002 1 1.00 262144-100.00
'
)
resources = provider_class.lookup_ring
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
@ -128,8 +173,6 @@ Devices: id region zone ip address port replication ip replicat
end
it 'should be able to lookup the local ring and build an object 1.8.0' do
File.expects(:exists?).with(builder_file_path).returns(true)
provider_class.expects(:builder_file_path).twice.returns(builder_file_path)
# Swift 1.8 output
provider_class.expects(:swift_ring_builder).returns(
'/etc/swift/account.builder, build version 3
@ -142,7 +185,9 @@ Devices: id region zone ip address port name weight partitions b
3 1 1 192.168.101.16 6002 1 1.00 262144-100.00
'
)
resources = provider_class.lookup_ring
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
@ -166,8 +211,6 @@ Devices: id region zone ip address port name weight partitions b
end
it 'should be able to lookup the local ring and build an object 1.7' do
File.expects(:exists?).with(builder_file_path).returns(true)
provider_class.expects(:builder_file_path).twice.returns(builder_file_path)
# Swift 1.7 output
provider_class.expects(:swift_ring_builder).returns(
'/etc/swift/account.builder, build version 3
@ -179,7 +222,9 @@ Devices: id region zone ip address port name weight partitions b
0 1 3 192.168.101.15 6002 1 1.00 262144 0.00
'
)
resources = provider_class.lookup_ring
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
@ -202,8 +247,6 @@ Devices: id region zone ip address port name weight partitions b
end
it 'should be able to lookup the local ring and build an object legacy' do
File.expects(:exists?).with(builder_file_path).returns(true)
provider_class.expects(:builder_file_path).twice.returns(builder_file_path)
provider_class.expects(:swift_ring_builder).returns(
'/etc/swift/account.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
@ -214,7 +257,9 @@ Devices: id zone ip address port name weight partitions balance m
1 1 192.168.101.13 6002 1 1.00 262144 0.00
'
)
resources = provider_class.lookup_ring
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil

View File

@ -0,0 +1,284 @@
require 'puppet'
require 'mocha'
require File.join(File.dirname(__FILE__), '..', '..', '..', '..', '..', 'lib', 'puppet', 'provider', 'ring_container_device', 'swift_ring_builder')
RSpec.configure do |config|
config.mock_with :mocha
end
provider_class = Puppet::Type.type(:ring_container_device).provider(:swift_ring_builder)
describe provider_class do
it "should have the SwiftRingBuilder provider class as its baseclass" do
expect(provider_class.superclass).to equal(Puppet::Provider::SwiftRingBuilder)
end
let :builder_file_path do
'/etc/swift/container.builder'
end
let :provider do
described_class.new(
Puppet::Type.type(:ring_container_device).new(
:name => '192.168.101.13:6002/1',
:zone => '1',
:weight => '1',
:provider => :swift_ring_builder
)
)
end
let :container_builder_output_2_9_1 do
'/etc/swift/container.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
The overload factor is 0.00% (0.000000)
Devices: id region zone ip address:port replic_ip:replic_port name weight partitions balance meta
1 1 1 192.168.101.13:6002 192.168.101.13:6002 1 1.00 262144 0.00
2 1 2 192.168.101.14:6002 192.168.101.14:6002 1 1.00 262144 200.00 m2
0 1 3 192.168.101.15:6002 192.168.101.15:6002 1 1.00 262144-100.00 m2
3 1 1 192.168.101.16:6002 192.168.101.16:6002 1 1.00 262144-100.00
'
end
describe "with no storage policy_index set on swift 2.9.1+" do
it 'container builder file should be container.builder when object name has no policy_index' do
policy_index = provider.policy_index
expect(provider.builder_file_path(policy_index)).to eq builder_file_path
end
it 'ring_container_device should exist when found in builder file' do
provider.expects(:swift_ring_builder).returns container_builder_output_2_9_1
File.expects(:exists?).with(builder_file_path).returns(true)
expect(provider.exists?).to eq({:id=>"1", :region=>"1", :zone=>"1", :weight=>"1.00", :partitions=>"262144", :balance=>"0.00", :meta=>"", :policy_index=>''})
end
it 'should be able to lookup the local ring' do
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
provider.expects(:swift_ring_builder).returns container_builder_output_2_9_1
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.16:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.13:6002/1'][:policy_index]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '200.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql 'm2'
expect(resources['192.168.101.14:6002/1'][:policy_index]).to eql ''
end
end
describe "with a storage policy_index set on swift 2.9.1+" do
it 'ring_container_device should fail when object name has a policy_index' do
expect {
Puppet::Type.type(:ring_container_device).new(
:name => '1:192.168.101.13:6002/1',
:zone => '1',
:weight => '1',
:provider => :swift_ring_builder
)}.to raise_error(Puppet::Error, /Policy_index is not supported on container device/)
end
end
it 'should be able to lookup the local ring and build an object 2.2.2+' do
# Swift 1.8 output
provider.expects(:swift_ring_builder).returns(
'/etc/swift/container.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
The overload factor is 0.00% (0.000000)
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
1 1 1 192.168.101.13 6002 192.168.101.13 6002 1 1.00 262144 0.00
2 1 2 192.168.101.14 6002 192.168.101.14 6002 1 1.00 262144 200.00 m2
0 1 3 192.168.101.15 6002 192.168.101.15 6002 1 1.00 262144-100.00 m2
3 1 1 192.168.101.16 6002 192.168.101.16 6002 1 1.00 262144-100.00
'
)
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.16:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '200.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql 'm2'
end
it 'should be able to lookup the local ring and build an object 1.8+' do
# Swift 1.8+ output
provider_class.expects(:swift_ring_builder).returns(
'/etc/swift/container.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
1 1 1 192.168.101.13 6002 192.168.101.13 6002 1 1.00 262144 0.00
2 1 2 192.168.101.14 6002 192.168.101.14 6002 1 1.00 262144 200.00 m2
0 1 3 192.168.101.15 6002 192.168.101.15 6002 1 1.00 262144-100.00 m2
3 1 1 192.168.101.16 6002 192.168.101.16 6002 1 1.00 262144-100.00
'
)
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.16:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '200.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql 'm2'
end
it 'should be able to lookup the local ring and build an object 1.8.0' do
# Swift 1.8 output
provider_class.expects(:swift_ring_builder).returns(
'/etc/swift/container.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id region zone ip address port name weight partitions balance meta
1 1 1 192.168.101.13 6002 1 1.00 262144 0.00
2 1 2 192.168.101.14 6002 1 1.00 262144 200.00 m2
0 1 3 192.168.101.15 6002 1 1.00 262144-100.00 m2
3 1 1 192.168.101.16 6002 1 1.00 262144-100.00
'
)
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.16:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '200.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql 'm2'
end
it 'should be able to lookup the local ring and build an object 1.7' do
# Swift 1.7 output
provider_class.expects(:swift_ring_builder).returns(
'/etc/swift/container.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id region zone ip address port name weight partitions balance meta
1 1 1 192.168.101.13 6002 1 1.00 262144 0.00
2 1 2 192.168.101.14 6002 1 1.00 262144 0.00
0 1 3 192.168.101.15 6002 1 1.00 262144 0.00
'
)
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql ''
end
it 'should be able to lookup the local ring and build an object legacy' do
provider_class.expects(:swift_ring_builder).returns(
'/etc/swift/container.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id zone ip address port name weight partitions balance meta
2 2 192.168.101.14 6002 1 1.00 262144 0.00
0 3 192.168.101.15 6002 1 1.00 262144 0.00
1 1 192.168.101.13 6002 1 1.00 262144 0.00
'
)
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql 'none'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql 'none'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql ''
end
end

View File

@ -0,0 +1,360 @@
require 'puppet'
require 'mocha'
require File.join(File.dirname(__FILE__), '..', '..', '..', '..', '..', 'lib', 'puppet', 'provider', 'ring_object_device', 'swift_ring_builder')
RSpec.configure do |config|
config.mock_with :mocha
end
provider_class = Puppet::Type.type(:ring_object_device).provider(:swift_ring_builder)
describe provider_class do
it "should have the SwiftRingBuilder provider class as its baseclass" do
expect(provider_class.superclass).to equal(Puppet::Provider::SwiftRingBuilder)
end
let :builder_file_path do
'/etc/swift/object.builder'
end
let :provider do
described_class.new(
Puppet::Type.type(:ring_object_device).new(
:name => '192.168.101.13:6002/1',
:zone => '1',
:weight => '1',
:provider => :swift_ring_builder
)
)
end
let :object_builder_output_2_9_1 do
'/etc/swift/object.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
The overload factor is 0.00% (0.000000)
Devices: id region zone ip address:port replic_ip:replic_port name weight partitions balance meta
1 1 1 192.168.101.13:6002 192.168.101.13:6002 1 1.00 262144 0.00
2 1 2 192.168.101.14:6002 192.168.101.14:6002 1 1.00 262144 200.00 m2
0 1 3 192.168.101.15:6002 192.168.101.15:6002 1 1.00 262144-100.00 m2
3 1 1 192.168.101.16:6002 192.168.101.16:6002 1 1.00 262144-100.00
'
end
describe "with no storage policy_index set on swift 2.9.1+" do
it 'object builder file should be object.builder when object name has no policy_index' do
policy_index = provider.policy_index
expect(provider.builder_file_path(policy_index)).to eq builder_file_path
end
it 'object builder file should be object.builder when object is invoked with policy_index 0' do
expect(provider.builder_file_path(0)).to eq builder_file_path
end
it 'object builder file should not be object-1.builder when object name has no policy_index' do
policy_index = provider.policy_index
expect(provider.builder_file_path(policy_index)).to_not eq builder_file_path_policy1
end
it 'object builder file should not be object-1.builder when object is invoked with policy_index 0' do
expect(provider.builder_file_path(0)).to_not eq builder_file_path_policy1
end
it 'ring_object_device should exist when found in builder file' do
provider.expects(:swift_ring_builder).returns object_builder_output_2_9_1
File.expects(:exists?).with(builder_file_path).returns(true)
expect(provider.exists?).to eq({:id=>"1", :region=>"1", :zone=>"1", :weight=>"1.00", :partitions=>"262144", :balance=>"0.00", :meta=>"", :policy_index=>''})
end
it 'should be able to lookup the local ring' do
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
provider.expects(:swift_ring_builder).returns object_builder_output_2_9_1
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.16:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.13:6002/1'][:policy_index]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '200.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql 'm2'
expect(resources['192.168.101.14:6002/1'][:policy_index]).to eql ''
end
end
let :builder_file_path_policy1 do
'/etc/swift/object-1.builder'
end
let :provider_policy1 do
described_class.new(
Puppet::Type.type(:ring_object_device).new(
:name => '1:192.168.101.13:6002/1',
:zone => '1',
:weight => '1',
:provider => :swift_ring_builder
)
)
end
let :object_builder_policy1_output_2_9_1 do
'/etc/swift/object-1.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
The overload factor is 0.00% (0.000000)
Devices: id region zone ip address:port replic_ip:replic_port name weight partitions balance meta
1 1 1 192.168.101.13:6002 192.168.101.13:6002 1 1.00 262144 0.00
2 1 2 192.168.101.14:6002 192.168.101.14:6002 1 1.00 262144 200.00 m2
0 1 3 192.168.101.15:6002 192.168.101.15:6002 1 1.00 262144-100.00 m2
3 1 1 192.168.101.16:6002 192.168.101.16:6002 1 1.00 262144-100.00
'
end
describe "with a storage policy_index set on swift 2.9.1+" do
it 'object builder file should be object-1.builder when object name has policy_index 1' do
policy_index = provider_policy1.policy_index
expect(provider_policy1.builder_file_path(policy_index)).to eq builder_file_path_policy1
end
it 'object builder file should not be object.builder when object names has policy_index 1' do
policy_index = provider_policy1.policy_index
expect(provider_policy1.builder_file_path(policy_index)).to_not eq builder_file_path
end
it 'ring_object_device should exist when found in builder file with policy_index=1' do
provider_policy1.expects(:swift_ring_builder).returns object_builder_policy1_output_2_9_1
File.expects(:exists?).with(builder_file_path_policy1).returns(true)
expect(provider_policy1.exists?).to eq({:id=>"1", :region=>"1", :zone=>"1", :weight=>"1.00", :partitions=>"262144", :balance=>"0.00", :meta=>"", :policy_index=>"1"})
end
it 'lookup local ring and object resource names should start with policy_index if a policy is set' do
File.expects(:exists?).with(builder_file_path_policy1).returns(true)
provider_policy1.expects(:builder_file_path).twice.returns(builder_file_path_policy1)
provider_policy1.expects(:swift_ring_builder).returns object_builder_output_2_9_1
resources = provider_policy1.lookup_ring
expect(resources['1:192.168.101.13:6002/1']).to_not be_nil
expect(resources['1:192.168.101.14:6002/1']).to_not be_nil
expect(resources['1:192.168.101.15:6002/1']).to_not be_nil
expect(resources['1:192.168.101.16:6002/1']).to_not be_nil
expect(resources['1:192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['1:192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['1:192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['1:192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['1:192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['1:192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['1:192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['1:192.168.101.13:6002/1'][:policy_index]).to eql '1'
expect(resources['1:192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['1:192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['1:192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['1:192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['1:192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['1:192.168.101.14:6002/1'][:balance]).to eql '200.00'
expect(resources['1:192.168.101.14:6002/1'][:meta]).to eql 'm2'
expect(resources['1:192.168.101.14:6002/1'][:policy_index]).to eql '1'
end
end
it 'should be able to lookup the local ring and build an object 2.2.2+' do
# Swift 1.8 output
provider.expects(:swift_ring_builder).returns(
'/etc/swift/object.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
The overload factor is 0.00% (0.000000)
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
1 1 1 192.168.101.13 6002 192.168.101.13 6002 1 1.00 262144 0.00
2 1 2 192.168.101.14 6002 192.168.101.14 6002 1 1.00 262144 200.00 m2
0 1 3 192.168.101.15 6002 192.168.101.15 6002 1 1.00 262144-100.00 m2
3 1 1 192.168.101.16 6002 192.168.101.16 6002 1 1.00 262144-100.00
'
)
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.16:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '200.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql 'm2'
end
it 'should be able to lookup the local ring and build an object 1.8+' do
# Swift 1.8+ output
provider_class.expects(:swift_ring_builder).returns(
'/etc/swift/object.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
1 1 1 192.168.101.13 6002 192.168.101.13 6002 1 1.00 262144 0.00
2 1 2 192.168.101.14 6002 192.168.101.14 6002 1 1.00 262144 200.00 m2
0 1 3 192.168.101.15 6002 192.168.101.15 6002 1 1.00 262144-100.00 m2
3 1 1 192.168.101.16 6002 192.168.101.16 6002 1 1.00 262144-100.00
'
)
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.16:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '200.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql 'm2'
end
it 'should be able to lookup the local ring and build an object 1.8.0' do
# Swift 1.8 output
provider_class.expects(:swift_ring_builder).returns(
'/etc/swift/object.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id region zone ip address port name weight partitions balance meta
1 1 1 192.168.101.13 6002 1 1.00 262144 0.00
2 1 2 192.168.101.14 6002 1 1.00 262144 200.00 m2
0 1 3 192.168.101.15 6002 1 1.00 262144-100.00 m2
3 1 1 192.168.101.16 6002 1 1.00 262144-100.00
'
)
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.16:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '200.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql 'm2'
end
it 'should be able to lookup the local ring and build an object 1.7' do
# Swift 1.7 output
provider_class.expects(:swift_ring_builder).returns(
'/etc/swift/object.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id region zone ip address port name weight partitions balance meta
1 1 1 192.168.101.13 6002 1 1.00 262144 0.00
2 1 2 192.168.101.14 6002 1 1.00 262144 0.00
0 1 3 192.168.101.15 6002 1 1.00 262144 0.00
'
)
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql '1'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql ''
end
it 'should be able to lookup the local ring and build an object legacy' do
provider_class.expects(:swift_ring_builder).returns(
'/etc/swift/object.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id zone ip address port name weight partitions balance meta
2 2 192.168.101.14 6002 1 1.00 262144 0.00
0 3 192.168.101.15 6002 1 1.00 262144 0.00
1 1 192.168.101.13 6002 1 1.00 262144 0.00
'
)
File.expects(:exists?).with(builder_file_path).returns(true)
provider.expects(:builder_file_path).twice.returns(builder_file_path)
resources = provider.lookup_ring
expect(resources['192.168.101.15:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1']).to_not be_nil
expect(resources['192.168.101.14:6002/1']).to_not be_nil
expect(resources['192.168.101.13:6002/1'][:id]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:region]).to eql 'none'
expect(resources['192.168.101.13:6002/1'][:zone]).to eql '1'
expect(resources['192.168.101.13:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.13:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.13:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.13:6002/1'][:meta]).to eql ''
expect(resources['192.168.101.14:6002/1'][:id]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:region]).to eql 'none'
expect(resources['192.168.101.14:6002/1'][:zone]).to eql '2'
expect(resources['192.168.101.14:6002/1'][:weight]).to eql '1.00'
expect(resources['192.168.101.14:6002/1'][:partitions]).to eql '262144'
expect(resources['192.168.101.14:6002/1'][:balance]).to eql '0.00'
expect(resources['192.168.101.14:6002/1'][:meta]).to eql ''
end
end

View File

@ -0,0 +1,209 @@
require 'puppet'
require 'mocha'
require File.join(File.dirname(__FILE__), '..', '..', '..', '..', '..', 'spec', 'fixtures', 'modules', 'inifile', 'lib', 'puppet', 'util', 'ini_file')
require File.join(File.dirname(__FILE__), '..', '..', '..', '..', '..', 'lib', 'puppet', 'provider', 'swift_storage_policy', 'ruby')
RSpec.configure do |config|
config.mock_with :mocha
end
provider_class = Puppet::Type.type(:swift_storage_policy).provider(:ruby)
describe provider_class do
let :swift_conf_no_policy do
'[swift-constraints]
max_header_size=8192
[swift-hash]
swift_hash_path_suffix=secrete
'
end
let :swift_conf_policy0 do
'[swift-constraints]
max_header_size=8192
[swift-hash]
swift_hash_path_suffix=secrete
[storage-policy:0]
name = Policy-0
aliases = gold, silver, taco
policy_type = replication
default = true
'
end
let :swift_conf_policy1 do
'[swift-constraints]
max_header_size=8192
[swift-hash]
swift_hash_path_suffix=secrete
[storage-policy:0]
name = Policy-0
aliases = gold, silver, taco
policy_type = replication
default = true
[storage-policy:1]
name = Policy-1
aliases = a, b, c
policy_type = replication
default = false
'
end
let :swift_conf_policy2 do
'[swift-constraints]
max_header_size=8192
[swift-hash]
swift_hash_path_suffix=secrete
[storage-policy:0]
name = Policy-0
aliases = gold, silver, taco
policy_type = replication
default = true
[storage-policy:1]
name = Policy-1
aliases = a, b, c
policy_type = replication
default = false
[storage-policy:2]
name = Policy-2
aliases = red, green, blue
policy_type = replication
default = false
'
end
def validate_file(expected_content, tmpfile)
expect(File.read(tmpfile)).to eq expected_content
end
let :provider0 do
described_class.new(
Puppet::Type.type(:swift_storage_policy).new(
:name => '0',
:policy_name => 'Policy-0',
:aliases => 'gold, silver, taco',
:policy_type => 'replication',
:default => 'true',
:provider => :ruby
)
)
end
let :swift_conf do
Tempfile.new('swift_conf_file')
end
describe 'defining a new valid swift_storage_policy with a policy in place should succeed' do
before :each do
File.open(swift_conf, 'w') do |fh|
fh.write(swift_conf_no_policy)
end
@swiftconffile = swift_conf.path
provider0.class.stubs(:get_swift_conf_file).returns @swiftconffile
end
it 'the swift_storage_policy 0 resource should contain the correct policy_title and name' do
expect(provider0.policy_title).to eql 'storage-policy:0'
expect(provider0.name).to eql '0'
end
it 'swift_storage_policy resources should create correct items in swift.conf and pass error checking' do
# Testing the complete flow/cases of creating swift_storage_policy resources here in one block.
# Spliting cases up across multiple "it" blocks doesn't appear to work well for a provider that calls
# flush. It appears that the testing needs to be done in one "it" block to duplicate the state puppet
# would find when pre-fetching and flushing resources during a run.
validate_file(swift_conf_no_policy, provider0.class.get_swift_conf_file)
# Create policy 0,flush calls provider "write_policy".
provider0.create
provider0.flush
# storage-policy:0 should exist and be found in swift.conf now.
expect(provider0.exists?).to be_truthy
expect(provider0.class.storage_policies).to eq (["storage-policy:0"])
validate_file(swift_conf_policy0, provider0.class.get_swift_conf_file)
provider1 = described_class.new(
Puppet::Type.type(:swift_storage_policy).new(
:name => '1',
:policy_name => 'Policy-1',
:aliases => 'a, b, c',
:policy_type => 'replication',
:default => 'false',
:provider => :ruby
)
)
provider1.class.stubs(:get_swift_conf_file).returns @swiftconffile
# storage-policy:1 should not yet exist in swift.conf
expect(provider1.exists?).to be_falsey
# Create policy 1,flush calls provider "write_policy"
provider1.create
provider1.flush
# storage-policy:0 and storage-policy:1 should exist
expect(provider0.exists?).to be_truthy
expect(provider1.exists?).to be_truthy
# storage-policy:0 and storage-policy:1 should exist in swift.conf
validate_file(swift_conf_policy1, provider0.class.get_swift_conf_file)
validate_file(swift_conf_policy1, provider1.class.get_swift_conf_file)
provider2 = described_class.new(
Puppet::Type.type(:swift_storage_policy).new(
:name => '2',
:policy_name => 'Policy-2',
:aliases => 'gold, red, green',
:policy_type => 'replication',
:default => 'false',
:provider => :ruby
)
)
provider2.class.stubs(:get_swift_conf_file).returns @swiftconffile
# storage-policy:2 should raise an error for duplicate name/alias conflict with storage-policy:0
provider2.create
expect { provider2.flush }.to raise_error(Puppet::Error, /trying to set a duplicate name/)
# storage-policy:0 and storage-policy:1 should exist in swift.conf
validate_file(swift_conf_policy1, provider0.class.get_swift_conf_file)
validate_file(swift_conf_policy1, provider1.class.get_swift_conf_file)
# Set non duplicate alias on provider2 then create/flush again with no error
provider2 = described_class.new(
Puppet::Type.type(:swift_storage_policy).new(
:name => '2',
:policy_name => 'Policy-2',
:aliases => 'red, green, blue',
:policy_type => 'replication',
:default => 'false',
:provider => :ruby
)
)
provider2.class.stubs(:get_swift_conf_file).returns @swiftconffile
provider2.create
expect { provider2.flush }.not_to raise_error
# storage-policy:0,1,2 should exist in swift.conf
validate_file(swift_conf_policy2, provider0.class.get_swift_conf_file)
# attempt to set storage-policy:1 default=true, expect an error raised on conflict with storage-policy:0
provider1 = described_class.new(
Puppet::Type.type(:swift_storage_policy).new(
:name => '1',
:policy_name => 'Policy-1',
:aliases => 'a, b, c',
:policy_type => 'replication',
:default => 'true',
:provider => :ruby
)
)
provider1.class.stubs(:get_swift_conf_file).returns @swiftconffile
provider1.create
expect { provider1.flush }.to raise_error(Puppet::Error, /default=true already set in a policy storage-policy:0/)
end
end
end

View File

@ -11,4 +11,10 @@ describe Puppet::Type.type(:ring_account_device) do
Puppet::Type.type(:ring_account_device).new(:name => 'foo:80')
}.to raise_error(Puppet::Error, /should contain a device/)
end
it 'should fail if the name contains a policy index' do
expect {
Puppet::Type.type(:ring_account_device).new(:name => '1:192.168.101.13:6002/1')
}.to raise_error(Puppet::Error, /Policy_index is not supported on account device/)
end
end

View File

@ -12,4 +12,11 @@ describe Puppet::Type.type(:ring_container_device) do
Puppet::Type.type(:ring_container_device).new(:name => 'foo:80')
}.to raise_error(Puppet::Error, /should contain a device/)
end
it 'should fail if the name contains a policy index' do
expect {
Puppet::Type.type(:ring_container_device).new(:name => '1:192.168.101.13:6002/1')
}.to raise_error(Puppet::Error, /Policy_index is not supported on container device/)
end
end

View File

@ -11,4 +11,10 @@ describe Puppet::Type.type(:ring_object_device) do
Puppet::Type.type(:ring_object_device).new(:name => 'foo:80')
}.to raise_error(Puppet::Error, /should contain a device/)
end
it 'should not fail if the name contains a policy index' do
expect {
Puppet::Type.type(:ring_object_device).new(:name => '1:192.168.101.13:6002/1')
}.to_not raise_error
end
end

View File

@ -0,0 +1,76 @@
require 'puppet'
describe Puppet::Type.type(:swift_storage_policy) do
it 'should pass if swift_storage_policy name is an integer' do
expect {
Puppet::Type.type(:swift_storage_policy).new(:name => '1')
}.to_not raise_error
end
it 'should fail if swift_storage_policy name is not an integer' do
expect {
Puppet::Type.type(:swift_storage_policy).new(:name => 'a')
}.to raise_error(Puppet::ResourceError, /swift_storage_policy name must be a postive integer/)
end
it 'should pass if policy_name is valid' do
expect {
Puppet::Type.type(:swift_storage_policy).new(:name => '1', :policy_name => 'Policy-0')
}.to_not raise_error
end
it 'should fail if policy_name is not valid' do
expect {
Puppet::Type.type(:swift_storage_policy).new(:name => '1', :policy_name => 'Policy_0')
}.to raise_error(Puppet::ResourceError, /policy_name must contain only letters, digits or a dash/)
end
it 'should pass if aliases is valid' do
expect {
Puppet::Type.type(:swift_storage_policy).new(:name => '1', :policy_name => 'Policy-0', :aliases => 'Gold, Silver, taco')
}.to_not raise_error
end
it 'should fail if aliases is not valid' do
expect {
Puppet::Type.type(:swift_storage_policy).new(:name => '1', :policy_name => 'Policy-0', :aliases => 'Gold, Sil_ver taco')
}.to raise_error(Puppet::ResourceError, /aliases must contain only letters, digits or a dash in a comma separated string/)
end
it 'should pass if policy_type is valid' do
expect {
Puppet::Type.type(:swift_storage_policy).new(:name => '1', :policy_name => 'Policy-0', :policy_type => 'replication')
}.to_not raise_error
end
it 'should fail if policy_type is not valid' do
expect {
Puppet::Type.type(:swift_storage_policy).new(:name => '1', :policy_name => 'Policy-0', :policy_type => 'other')
}.to raise_error(Puppet::ResourceError, /Valid values match \/(replication)|(erasure_coding)/)
end
it 'should pass if default is valid' do
expect {
Puppet::Type.type(:swift_storage_policy).new(:name => '1', :policy_name => 'Policy-0', :default => 'true')
}.to_not raise_error
end
it 'should fail if default is not valid' do
expect {
Puppet::Type.type(:swift_storage_policy).new(:name => '1', :policy_name => 'Policy-0', :default => 'other')
}.to raise_error(Puppet::ResourceError, /Valid values match \/(true)|(false)/)
end
it 'should pass if deprecated is valid' do
expect {
Puppet::Type.type(:swift_storage_policy).new(:name => '1', :policy_name => 'Policy-0', :deprecated => 'no')
}.to_not raise_error
end
it 'should fail if deprecated is not valid' do
expect {
Puppet::Type.type(:swift_storage_policy).new(:name => '1', :policy_name => 'Policy-0', :deprecated => 'other')
}.to raise_error(Puppet::ResourceError, /Valid values match \/(yes)|(no)/)
end
end