openstack/cookbook-openstack-image
Code Issues Proposed changes

initial pull from rcbops-cookbooks/glance, monitoring removed

Browse Source
This commit is contained in:
mattray
2012-06-28 16:23:15 -07:00
parent d506c67183
commit 72b598704c
17 changed files with 1753 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

86
README.md
View File

@@ -1,17 +1,98 @@
Description Description
=========== ===========
Placeholder for the OpenStack Image service `Glance` as part of the OpenStack `Essex` reference architecture using Chef. The http://github.com/opscode/openstack-chef-repo will contain documentation for using this cookbook in the context of a full OpenStack deployment.
Sharing the Nova database will be preferred for simplicity. This cookbook installs the OpenStack Image service **Glance** as part of the OpenStack **Essex** reference deployment Chef for OpenStack. The http://github.com/opscode/openstack-chef-repo contains documentation for using this cookbook in the context of a full OpenStack deployment. Glance is installed from packages, optionally populating the repository with default images.
http://glance.openstack.org/
Requirements
============
Chef 0.10.0 or higher required (for Chef environment use)
Platform
--------
* Ubuntu-12.04
* Fedora-17
Cookbooks
---------
The following cookbooks are dependencies:
* database
* mysql
* keystone
* osops-utils
Recipes Recipes
======= =======
default
-------
-Includes recipes `api`, `registry`
api
------
-Installs the glance-api server
registry
--------
-Includes recipe `mysql:client`
-Installs the glance-registry server
Attributes
==========
* `glance["db"]["name"]` - Name of glance database
* `glance["db"]["user"]` - Username for glance database access
* `glance["db"]["password"]` - Password for glance database access
* `glance["api"]["ip_address"]` - IP address to use for communicating with the glance API
* `glance["api"]["bind_address"]` - IP address for the glance API to bind to
* `glance["api"]["port"]` - Port for the glance API to bind to
* `glance["api"]["adminURL"]` - Used when registering image endpoint with keystone
* `glance["api"]["internalURL"]` - Used when registering image endpoint with keystone
* `glance["api"]["publicURL"]` - Used when registering image endpoint with keystone
* `glance["registry"]["ip_address"]` - IP address to use for communicating with the glance registry
* `glance["registry"]["bind_address"]` - IP address for the glance registry to bind to
* `glance["registry"]["port"]` - IP address for the glance port to bind to
* `glance["service_tenant_name"]` - Tenant name used by glance when interacting with keystone - used in the API and registry paste.ini files
* `glance["service_user"]` - User name used by glance when interacting with keystone - used in the API and registry paste.ini files
* `glance["service_pass"]` - User password used by glance when interacting with keystone - used in the API and registry paste.ini files
* `glance["service_role"]` - User role used by glance when interacting with keystone - used in the API and registry paste.ini files
* `glance["image_upload"]` - Toggles whether to automatically upload images in the `glance["images"]` array
* `glance["images"]` - Default list of images to upload to the glance repository as part of the install
* `glance["image]["<imagename>"]` - URL location of the <imagename> image. There can be multiple instances of this line to define multiple imagess (eg natty, maverick, fedora17 etc)
--- example `glance["image]["natty"]` - "http://c250663.r63.cf1.rackcdn.com/ubuntu-11.04-server-uec-amd64-multinic.tar.gz"
* `glance["api"]["default_store"]` - Toggles the backend storage type. Currently supported is "file" and "swift"
* `glance["api"]["swift"]["store_container"] - Set the container used by glance to store images and snapshots. Defaults to "glance"
* `glance["api"]["swift"]["store_large_object_size"] - Set the size at which glance starts to chunnk files. Defaults to "200" MB
* `glance["api"]["swift"]["store_large_object_chunk_size"] - Set the chunk size for glance. Defaults to "200" MB
Templates
=========
* `glance-api-paste.ini.erb` - Paste config for glance-api middleware
* `glance-api.conf.erb` - Config file for glance-api server
* `glance-registry-paste.ini.erb` - Paste config for glance-registry middleware
* `glance-registry.conf.erb` - Config file for glance-registry server
* `glance-scrubber.conf.erb` - Config file for glance image scrubber service
* `policy.json.erb` - Configuration of ACLs for glance API server
License and Author License and Author
================== ==================
Author:: Justin Shepherd (<justin.shepherd@rackspace.com>)
Author:: Jason Cannavale (<jason.cannavale@rackspace.com>)
Author:: Ron Pedde (<ron.pedde@rackspace.com>)
Author:: Joseph Breu (<joseph.breu@rackspace.com>)
Author:: William Kelly (<william.kelly@rackspace.com>)
Author:: Darren Birkett (<darren.birkett@rackspace.co.uk>)
Author:: Evan Callicoat (<evan.callicoat@rackspace.com>)
Author:: Matt Ray (<matt@opscode.com>) Author:: Matt Ray (<matt@opscode.com>)
Copyright 2012 Rackspace, Inc.
Copyright 2012 Opscode, Inc. Copyright 2012 Opscode, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); Licensed under the Apache License, Version 2.0 (the "License");
@@ -25,4 +106,3 @@ distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and See the License for the specific language governing permissions and
limitations under the License. limitations under the License.

79
attributes/default.rb Normal file
View File

@@ -0,0 +1,79 @@
#
# Cookbook Name:: glance
# Attributes:: glance
#
# Copyright 2009, Rackspace Hosting, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
########################################################################
# Toggles - These can be overridden at the environment level
default["enable_monit"] = false # OS provides packages
default["developer_mode"] = true # we want secure passwords by default
########################################################################
default["glance"]["services"]["api"]["scheme"] = "http"
default["glance"]["services"]["api"]["network"] = "public"
default["glance"]["services"]["api"]["port"] = 9292
default["glance"]["services"]["api"]["path"] = "/v1"
default["glance"]["services"]["registry"]["scheme"] = "http"
default["glance"]["services"]["registry"]["network"] = "public"
default["glance"]["services"]["registry"]["port"] = 9191
default["glance"]["services"]["registry"]["path"] = "/v1"
default["glance"]["db"]["name"] = "glance"
default["glance"]["db"]["username"] = "glance"
# TODO: These may need to be glance-registry specific.. and looked up by glance-api
default["glance"]["service_tenant_name"] = "service"
default["glance"]["service_user"] = "glance"
default["glance"]["service_role"] = "admin"
default["glance"]["api"]["default_store"] = "file"
default["glance"]["api"]["swift"]["store_container"] = "glance"
default["glance"]["api"]["swift"]["store_large_object_size"] = "200"
default["glance"]["api"]["swift"]["store_large_object_chunk_size"] = "200"
default["glance"]["image_upload"] = false
default["glance"]["images"] = [ "tty" ]
default["glance"]["image"]["oneiric"] = "http://c250663.r63.cf1.rackcdn.com/ubuntu-11.10-server-uec-amd64-multinic.tar.gz"
default["glance"]["image"]["natty"] = "http://c250663.r63.cf1.rackcdn.com/ubuntu-11.04-server-uec-amd64-multinic.tar.gz"
default["glance"]["image"]["maverick"] = "http://c250663.r63.cf1.rackcdn.com/ubuntu-10.10-server-uec-amd64-multinic.tar.gz"
#default["glance"]["image"]["tty"] = "http://smoser.brickies.net/ubuntu/ttylinux-uec/ttylinux-uec-amd64-12.1_2.6.35-22_1.tar.gz"
default["glance"]["image"]["tty"] = "http://c250663.r63.cf1.rackcdn.com/ttylinux.tgz"
default["glance"]["image"]["cirros"] = "https://launchpadlibrarian.net/83305869/cirros-0.3.0-x86_64-uec.tar.gz"
# logging attribute
default["glance"]["syslog"]["use"] = true
default["glance"]["syslog"]["facility"] = "LOG_LOCAL2"
# platform-specific settings
case platform
when "fedora"
default["glance"]["platform"] = {
"mysql_python_packages" => [ "MySQL-python" ],
"glance_packages" => [ "openstack-glance", "openstack-swift" ],
"glance_api_service" => "openstack-glance-api",
"glance_registry_service" => "openstack-glance-registry",
"package_overrides" => ""
}
when "ubuntu"
default["glance"]["platform"] = {
"mysql_python_packages" => [ "python-mysqldb" ],
"glance_packages" => [ "glance", "python-swift" ],
"glance_api_service" => "glance-api",
"glance_registry_service" => "glance-registry",
"package_overrides" => "-o Dpkg::Options::='--force-confold' -o Dpkg::Options::='--force-confdef'"
}
end

116
files/default/glance_plugin.py Normal file
View File

@@ -0,0 +1,116 @@
from glance.client import V1Client
from glance.common import exception
import collectd
global NAME, OS_USERNAME, OS_PASSWORD, OS_TENANT_NAME, OS_AUTH_URL, OS_AUTH_STRATEGY, VERBOSE_LOGGING
OS_USERNAME = "username"
OS_PASSWORD = "password"
OS_TENANT_NAME = "tenantname"
OS_AUTH_URL = "http://localhost:5000/v2.0"
OS_AUTH_STRATEGY = "keystone"
VERBOSE_LOGGING = False
def get_stats(user, passwd, tenant, url, host=None):
creds = {"username": user, "password": passwd, "tenant": tenant,"auth_url": url, "strategy": OS_AUTH_STRATEGY}
client = V1Client(host,creds=creds)
try:
image_list = client.get_images_detailed()
except exception.NotAuthenticated:
msg = "Client credentials appear to be invalid"
raise exception.ClientConnectionError(msg)
else:
# TODO(shep): this needs to be rewritten more inline with the keystone|nova plugins
data = dict()
data["count"] = int(len(image_list))
data["bytes"] = 0
data["snapshot.count"] = 0
data["snapshot.bytes"] = 0
data["tenant"] = dict()
for image in image_list:
data["bytes"] += int(image["size"])
if "image_type" in image["properties"] and image["properties"]["image_type"] == "snapshot":
data["snapshot.count"] += 1
data["snapshot.bytes"] += int(image["size"])
uuid = str(image["owner"])
if uuid in data["tenant"]:
data["tenant"][uuid]["count"] += 1
data["tenant"][uuid]["bytes"] += int(image["size"])
if "image_type" in image["properties"] and image["properties"]["image_type"] == "snapshot":
data["tenant"][uuid]["snapshot.count"] += 1
data["tenant"][uuid]["snapshot.bytes"] += int(image["size"])
else:
data["tenant"][uuid] = dict()
data["tenant"][uuid]["count"] = 1
data["tenant"][uuid]["bytes"] = int(image["size"])
data["tenant"][uuid]["snapshot.count"] = 0
data["tenant"][uuid]["snapshot.bytes"] = 0
if "image_type" in image["properties"] and image["properties"]["image_type"] == "snapshot":
data["tenant"][uuid]["snapshot.count"] += 1
data["tenant"][uuid]["snapshot.bytes"] += int(image["size"])
# debug
#for key in data.keys():
# if key == "tenant":
# for uuid in data[key].keys():
# for field in data[key][uuid]:
# print "glance.images.tenant.%s.%s : %i" % (uuid, field, data[key][uuid][field])
# else:
# print "glance.images.%s : %i" % (key, data[key])
##########
return data
def configure_callback(conf):
"""Received configuration information"""
global OS_USERNAME, OS_PASSWORD, OS_TENANT_NAME, OS_AUTH_URL
for node in conf.children:
if node.key == "Username":
OS_USERNAME = node.values[0]
elif node.key == "Password":
OS_PASSWORD = node.values[0]
elif node.key == "TenantName":
OS_TENANT_NAME = node.values[0]
elif node.key == "AuthURL":
OS_AUTH_URL = node.values[0]
elif node.key == "Verbose":
VERBOSE_LOGGING = node.values[0]
else:
logger("warn", "Unknown config key: %s" % node.key)
def read_callback():
logger("verb", "read_callback")
info = get_stats(OS_USERNAME, OS_PASSWORD, OS_TENANT_NAME, OS_AUTH_URL)
if not info:
logger("err", "No information received")
return
for key in info.keys():
if key == "tenant":
for uuid in info[key].keys():
for field in info[key][uuid]:
logger('verb', 'Dispatching glance.images.tenant.%s.%s : %i' % (uuid, field, int(info[key][uuid][field])))
path = 'glance.images.%s.%s' % (uuid, field)
val = collectd.Values(plugin=path)
val.type = 'gauge'
val.values = [int(info[key][uuid][field])]
val.dispatch()
else:
logger('verb', 'Dispatching %s : %i' % (key, int(info[key])))
path = 'glance.images.%s' % (key)
val = collectd.Values(plugin=path)
val.type = 'gauge'
val.values = [int(info[key])]
val.dispatch()
def logger(t, msg):
if t == 'err':
collectd.error('%s: %s' % (NAME, msg))
if t == 'warn':
collectd.warning('%s: %s' % (NAME, msg))
elif t == 'verb' and VERBOSE_LOGGING == True:
collectd.info('%s: %s' % (NAME, msg))
collectd.register_config(configure_callback)
collectd.warning("Initializing glance plugin")
collectd.register_read(read_callback)

21
metadata.rb
View File

@@ -1,6 +1,15 @@
maintainer "Opscode, Inc." maintainer "Opscode, Inc."
maintainer_email "matt@opscode.com" license "Apache 2.0"
license "Apache 2.0" description "The Glance Image Registry and Delivery Service Glance"
description "The OpenStack Image service Glance." long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md')) version "5.0.0"
version "0.0.1" recipe "glance::api", "Installs packages required for a glance api server"
recipe "glance::registry", "Installs packages required for a glance registry server"
%w{ ubuntu fedora }.each do |os|
supports os
end
%w{ database keystone mysql osops-utils }.each do |dep|
depends dep
end

220
recipes/api.rb Normal file
View File

@@ -0,0 +1,220 @@
#
# Cookbook Name:: glance
# Recipe:: api
#
# Copyright 2009-2012, Rackspace Hosting, Inc.
# Copyright 2012, Opscode, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
include_recipe "glance::glance-rsyslog"
platform_options = node["glance"]["platform"]
package "curl" do
action :upgrade
end
package "python-keystone" do
action :install
end
platform_options["glance_packages"].each do |pkg|
package pkg do
action :upgrade
end
end
service "glance-api" do
service_name platform_options["glance_api_service"]
supports :status => true, :restart => true
action :enable
end
# FIXME: this is broken. Joe, Wilk, fix this.
template "/usr/share/pyshared/glance/store/swift.py" do
source "swift.py"
group "root"
owner "root"
mode "0644"
only_if do platform?(%w{debian ubuntu}) end
notifies :restart, resources(:service => "glance-api"), :immediately
end
directory "/etc/glance" do
action :create
group "glance"
owner "glance"
mode "0700"
end
# FIXME: seems like misfeature
template "/etc/glance/policy.json" do
source "policy.json.erb"
owner "root"
group "root"
mode "0644"
notifies :restart, resources(:service => "glance-api"), :immediately
not_if do
File.exists?("/etc/glance/policy.json")
end
end
rabbit_info = get_settings_by_role("rabbitmq-server", "rabbitmq") # FIXME: access
ks_admin_endpoint = get_access_endpoint("keystone", "keystone", "admin-api")
ks_service_endpoint = get_access_endpoint("keystone", "keystone","service-api")
keystone = get_settings_by_role("keystone", "keystone")
glance = get_settings_by_role("glance-api", "glance")
registry_endpoint = get_access_endpoint("glance-registry", "glance", "registry")
api_endpoint = get_bind_endpoint("glance", "api")
template "/etc/glance/glance-api.conf" do
source "glance-api.conf.erb"
owner "root"
group "root"
mode "0644"
variables(
"api_bind_address" => api_endpoint["host"],
"api_bind_port" => api_endpoint["port"],
"registry_ip_address" => registry_endpoint["host"],
"registry_port" => registry_endpoint["port"],
"use_syslog" => node["glance"]["syslog"]["use"],
"log_facility" => node["glance"]["syslog"]["facility"],
"rabbit_ipaddress" => rabbit_info["ipaddress"], #FIXME!
"keystone_api_ipaddress" => ks_admin_endpoint["host"],
"keystone_service_port" => ks_service_endpoint["port"],
"service_user" => glance["service_user"],
"service_pass" => glance["service_pass"],
"service_tenant_name" => glance["service_tenant_name"],
"default_store" => glance["api"]["default_store"],
"swift_large_object_size" => glance["api"]["swift"]["store_large_object_size"],
"swift_large_object_chunk_size" => glance["api"]["swift"]["store_large_object_chunk_size"],
"swift_store_container" => glance["api"]["swift"]["store_container"]
)
notifies :restart, resources(:service => "glance-api"), :immediately
end
template "/etc/glance/glance-api-paste.ini" do
source "glance-api-paste.ini.erb"
owner "root"
group "root"
mode "0644"
variables(
"keystone_api_ipaddress" => ks_admin_endpoint["host"],
"keystone_service_port" => ks_service_endpoint["port"],
"keystone_admin_port" => ks_admin_endpoint["port"],
"keystone_admin_token" => keystone["admin_token"],
"service_tenant_name" => node["glance"]["service_tenant_name"],
"service_user" => node["glance"]["service_user"],
"service_pass" => node["glance"]["service_pass"]
)
notifies :restart, resources(:service => "glance-api"), :immediately
end
template "/etc/glance/glance-scrubber.conf" do
source "glance-scrubber.conf.erb"
owner "root"
group "root"
mode "0644"
variables(
"registry_ip_address" => registry_endpoint["host"],
"registry_port" => registry_endpoint["port"]
)
end
template "/etc/glance/glance-scrubber-paste.ini" do
source "glance-scrubber-paste.ini.erb"
owner "root"
group "root"
mode "0644"
end
# Register Image Service
keystone_register "Register Image Service" do
auth_host ks_admin_endpoint["host"]
auth_port ks_admin_endpoint["port"]
auth_protocol ks_admin_endpoint["scheme"]
api_ver ks_admin_endpoint["path"]
auth_token keystone["admin_token"]
service_name "glance"
service_type "image"
service_description "Glance Image Service"
action :create_service
end
# Register Image Endpoint
keystone_register "Register Image Endpoint" do
auth_host ks_admin_endpoint["host"]
auth_port ks_admin_endpoint["port"]
auth_protocol ks_admin_endpoint["scheme"]
api_ver ks_admin_endpoint["path"]
auth_token keystone["admin_token"]
service_type "image"
endpoint_region "RegionOne"
endpoint_adminurl api_endpoint["uri"]
endpoint_internalurl api_endpoint["uri"]
endpoint_publicurl api_endpoint["uri"]
action :create_endpoint
end
if node["glance"]["image_upload"]
# TODO(breu): the environment needs to be derived from a search
# TODO(shep): this whole bit is super dirty.. and needs some love.
node["glance"]["images"].each do |img|
Chef::Log.info("Checking to see if #{img.to_s}-image should be uploaded.")
keystone_admin_user = keystone["admin_user"]
keystone_admin_password = keystone["users"][keystone_admin_user]["password"]
keystone_tenant = keystone["users"][keystone_admin_user]["default_tenant"]
bash "default image setup for #{img.to_s}" do
cwd "/tmp"
user "root"
environment ({"OS_USERNAME" => keystone_admin_user,
"OS_PASSWORD" => keystone_admin_password,
"OS_TENANT_NAME" => keystone_tenant,
"OS_AUTH_URL" => ks_admin_endpoint["uri"]})
code <<-EOH
set -e
set -x
mkdir -p images/#{img.to_s}
cd images/#{img.to_s}
curl #{node["glance"]["image"][img.to_sym]} | tar -zx
image_name=$(basename #{node["glance"]["image"][img]} .tar.gz)
image_name=${image_name%-multinic}
kernel_file=$(ls *vmlinuz-virtual | head -n1)
if [ ${#kernel_file} -eq 0 ]; then
kernel_file=$(ls *vmlinuz | head -n1)
fi
ramdisk=$(ls *-initrd | head -n1)
if [ ${#ramdisk} -eq 0 ]; then
ramdisk=$(ls *-loader | head -n1)
fi
kernel=$(ls *.img | head -n1)
kid=$(glance --silent-upload add name="${image_name}-kernel" is_public=true disk_format=aki container_format=aki < ${kernel_file} | cut -d: -f2 | sed 's/ //')
rid=$(glance --silent-upload add name="${image_name}-initrd" is_public=true disk_format=ari container_format=ari < ${ramdisk} | cut -d: -f2 | sed 's/ //')
glance --silent-upload add name="#{img.to_s}-image" is_public=true disk_format=ami container_format=ami kernel_id=$kid ramdisk_id=$rid < ${kernel}
EOH
not_if "glance -f -I #{keystone_admin_user} -K #{keystone_admin_password} -T #{keystone_tenant} -N #{ks_admin_endpoint["uri"]} index | grep #{img.to_s}-image"
end
end
end

3
recipes/default.rb
View File

@@ -2,6 +2,7 @@
# Cookbook Name:: glance # Cookbook Name:: glance
# Recipe:: default # Recipe:: default
# #
# Copyright 2009-2012, Rackspace Hosting, Inc.
# Copyright 2012, Opscode, Inc. # Copyright 2012, Opscode, Inc.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
@@ -17,3 +18,5 @@
# limitations under the License. # limitations under the License.
# #
include_recipe "glance::registry"
include_recipe "glance::api"

31
recipes/glance-rsyslog.rb Normal file
View File

@@ -0,0 +1,31 @@
#
# Cookbook Name:: glance
# Recipe:: default
#
# Copyright 2009, Rackspace Hosting, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
if node["glance"]["syslog"]["use"]
template "/etc/rsyslog.d/22-glance.conf" do
source "22-glance.conf.erb"
owner "root"
group "root"
mode "0644"
variables(
"use_syslog" => node["glance"]["syslog"]["use"],
"log_facility" => node["glance"]["syslog"]["facility"]
)
end
end

174
recipes/registry.rb Normal file
View File

@@ -0,0 +1,174 @@
#
# Cookbook Name:: glance
# Recipe:: registry
#
# Copyright 2009, Rackspace Hosting, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
::Chef::Recipe.send(:include, Opscode::OpenSSL::Password)
include_recipe "mysql::client"
include_recipe "glance::glance-rsyslog"
platform_options = node["glance"]["platform"]
# Allow for using a well known db password
if node["developer_mode"]
node.set_unless['glance']['db']['password'] = "glance"
else
node.set_unless['glance']['db']['password'] = secure_password
end
# Set a secure keystone service password
node.set_unless['glance']['service_pass'] = secure_password
package "python-keystone" do
action :install
end
ks_admin_endpoint = get_access_endpoint("keystone", "keystone", "admin-api")
ks_service_endpoint = get_access_endpoint("keystone", "keystone", "service-api")
keystone = get_settings_by_role("keystone", "keystone")
registry_endpoint = get_bind_endpoint("glance", "registry")
#creates db and user
#returns connection info
#defined in osops-utils/libraries
mysql_info = create_db_and_user("mysql",
node["glance"]["db"]["name"],
node["glance"]["db"]["username"],
node["glance"]["db"]["password"])
package "curl" do
action :install
end
platform_options["mysql_python_packages"].each do |pkg|
package pkg do
action :install
end
end
platform_options["glance_packages"].each do |pkg|
package pkg do
action :upgrade
end
end
service "glance-registry" do
service_name platform_options["glance_registry_service"]
supports :status => true, :restart => true
action :enable
end
execute "glance-manage db_sync" do
command "sudo -u glance glance-manage db_sync"
action :nothing
notifies :restart, resources(:service => "glance-registry"), :immediately
end
# Having to manually version the database because of Ubuntu bug
# https://bugs.launchpad.net/ubuntu/+source/glance/+bug/981111
execute "glance-manage version_control" do
command "sudo -u glance glance-manage version_control 0"
action :nothing
not_if "sudo -u glance glance-manage db_version"
notifies :run, resources(:execute => "glance-manage db_sync"), :immediately
end
file "/var/lib/glance/glance.sqlite" do
action :delete
end
# Register Service Tenant
keystone_register "Register Service Tenant" do
auth_host ks_admin_endpoint["host"]
auth_port ks_admin_endpoint["port"]
auth_protocol ks_admin_endpoint["scheme"]
api_ver ks_admin_endpoint["path"]
auth_token keystone["admin_token"]
tenant_name node["glance"]["service_tenant_name"]
tenant_description "Service Tenant"
tenant_enabled "true" # Not required as this is the default
action :create_tenant
end
# Register Service User
keystone_register "Register Service User" do
auth_host ks_admin_endpoint["host"]
auth_port ks_admin_endpoint["port"]
auth_protocol ks_admin_endpoint["scheme"]
api_ver ks_admin_endpoint["path"]
auth_token keystone["admin_token"]
tenant_name node["glance"]["service_tenant_name"]
user_name node["glance"]["service_user"]
user_pass node["glance"]["service_pass"]
user_enabled "true" # Not required as this is the default
action :create_user
end
## Grant Admin role to Service User for Service Tenant ##
keystone_register "Grant 'admin' Role to Service User for Service Tenant" do
auth_host ks_admin_endpoint["host"]
auth_port ks_admin_endpoint["port"]
auth_protocol ks_admin_endpoint["scheme"]
api_ver ks_admin_endpoint["path"]
auth_token keystone["admin_token"]
tenant_name node["glance"]["service_tenant_name"]
user_name node["glance"]["service_user"]
role_name node["glance"]["service_role"]
action :grant_role
end
directory "/etc/glance" do
action :create
group "glance"
owner "glance"
mode "0700"
end
template "/etc/glance/glance-registry.conf" do
source "glance-registry.conf.erb"
owner "root"
group "root"
mode "0644"
variables(
"registry_bind_address" => registry_endpoint["host"],
"registry_port" => registry_endpoint["port"],
"db_ip_address" => mysql_info["bind_address"],
"db_user" => node["glance"]["db"]["username"],
"db_password" => node["glance"]["db"]["password"],
"db_name" => node["glance"]["db"]["name"],
"use_syslog" => node["glance"]["syslog"]["use"],
"log_facility" => node["glance"]["syslog"]["facility"]
)
notifies :run, resources(:execute => "glance-manage version_control"), :immediately
end
template "/etc/glance/glance-registry-paste.ini" do
source "glance-registry-paste.ini.erb"
owner "root"
group "root"
mode "0644"
variables(
"keystone_api_ipaddress" => ks_admin_endpoint["host"],
"keystone_service_port" => ks_service_endpoint["port"],
"keystone_admin_port" => ks_admin_endpoint["port"],
"service_tenant_name" => node["glance"]["service_tenant_name"],
"service_user" => node["glance"]["service_user"],
"service_pass" => node["glance"]["service_pass"]
)
notifies :restart, resources(:service => "glance-registry"), :immediately
end

7
templates/default/22-glance.conf.erb Normal file
View File

@@ -0,0 +1,7 @@
$DirGroup adm
$DirCreateMode 0755
$FileGroup adm
$template GlanceLog, "/var/log/glance/glance.log"
local2.* -?GlanceLog

76
templates/default/glance-api-paste.ini.erb Normal file
View File

@@ -0,0 +1,76 @@
# Default minimal pipeline
[pipeline:glance-api]
pipeline = versionnegotiation context apiv1app
# Use the following pipeline for keystone auth
# i.e. in glance-api.conf:
# [paste_deploy]
# flavor = keystone
#
[pipeline:glance-api-keystone]
pipeline = versionnegotiation authtoken context apiv1app
# Use the following pipeline to enable transparent caching of image files
# i.e. in glance-api.conf:
# [paste_deploy]
# flavor = caching
#
[pipeline:glance-api-caching]
pipeline = versionnegotiation context cache apiv1app
# Use the following pipeline for keystone auth with caching
# i.e. in glance-api.conf:
# [paste_deploy]
# flavor = keystone+caching
#
[pipeline:glance-api-keystone+caching]
pipeline = versionnegotiation authtoken context cache apiv1app
# Use the following pipeline to enable the Image Cache Management API
# i.e. in glance-api.conf:
# [paste_deploy]
# flavor = cachemanagement
#
[pipeline:glance-api-cachemanagement]
pipeline = versionnegotiation context cache cachemanage apiv1app
# Use the following pipeline for keystone auth with cache management
# i.e. in glance-api.conf:
# [paste_deploy]
# flavor = keystone+cachemanagement
#
[pipeline:glance-api-keystone+cachemanagement]
pipeline = versionnegotiation authtoken context cache cachemanage apiv1app
[app:apiv1app]
paste.app_factory = glance.common.wsgi:app_factory
glance.app_factory = glance.api.v1.router:API
[filter:versionnegotiation]
paste.filter_factory = glance.common.wsgi:filter_factory
glance.filter_factory = glance.api.middleware.version_negotiation:VersionNegotiationFilter
[filter:cache]
paste.filter_factory = glance.common.wsgi:filter_factory
glance.filter_factory = glance.api.middleware.cache:CacheFilter
[filter:cachemanage]
paste.filter_factory = glance.common.wsgi:filter_factory
glance.filter_factory = glance.api.middleware.cache_manage:CacheManageFilter
[filter:context]
paste.filter_factory = glance.common.wsgi:filter_factory
glance.filter_factory = glance.common.context:ContextMiddleware
[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = <%= @keystone_api_ipaddress %>
service_port = <%= @keystone_service_port %>
auth_host = <%= @keystone_api_ipaddress %>
auth_port = <%= @keystone_admin_port %>
auth_protocol = http
auth_uri = http://<%= @keystone_api_ipaddress %>:<%= @keystone_service_port %>/
admin_tenant_name = <%= @service_tenant_name %>
admin_user = <%= @service_user %>
admin_password = <%= @service_pass %>

244
templates/default/glance-api.conf.erb Normal file
View File

@@ -0,0 +1,244 @@
[DEFAULT]
# Show more verbose log output (sets INFO log level output)
verbose = True
# Show debugging output in logs (sets DEBUG log level output)
debug = False
# Which backend store should Glance use by default is not specified
# in a request to add a new image to Glance? Default: 'file'
# Available choices are 'file', 'swift', and 's3'
default_store = file
# Address to bind the API server
bind_host = <%= @api_bind_address %>
# Port the bind the API server to
bind_port = <%= @api_bind_port %>
# Log to this file. Make sure you do not set the same log
# file for both the API and registry servers!
log_file = /var/log/glance/api.log
# Backlog requests when creating socket
backlog = 4096
# Number of Glance API worker processes to start.
# On machines with more than one CPU increasing this value
# may improve performance (especially if using SSL with
# compression turned on). It is typically recommended to set
# this value to the number of CPUs present on your machine.
workers = 0
# Role used to identify an authenticated user as administrator
#admin_role = admin
# ================= Syslog Options ============================
# Send logs to syslog (/dev/log) instead of to file specified
# by `log_file`
use_syslog = <%= @use_syslog %>
<% if @use_syslog == true %>
# Facility to use. If unset defaults to LOG_USER.
syslog_log_facility = <%= @log_facility %>
<% else %>
# syslog_log_facility = LOG_USER
<% end %>
# ================= SSL Options ===============================
# Certificate file to use when starting API server securely
# cert_file = /path/to/certfile
# Private key file to use when starting API server securely
# key_file = /path/to/keyfile
# ================= Security Options ==========================
# AES key for encrypting store 'location' metadata, including
# -- if used -- Swift or S3 credentials
# Should be set to a random string of length 16, 24 or 32 bytes
# metadata_encryption_key = <16, 24 or 32 char registry metadata key>
# ============ Registry Options ===============================
# Address to find the registry server
registry_host = <%= @registry_ip_address %>
# Port the registry server is listening on
registry_port = <%= @registry_port %>
# What protocol to use when connecting to the registry server?
# Set to https for secure HTTP communication
registry_client_protocol = http
# The path to the key file to use in SSL connections to the
# registry server, if any. Alternately, you may set the
# GLANCE_CLIENT_KEY_FILE environ variable to a filepath of the key file
# registry_client_key_file = /path/to/key/file
# The path to the cert file to use in SSL connections to the
# registry server, if any. Alternately, you may set the
# GLANCE_CLIENT_CERT_FILE environ variable to a filepath of the cert file
# registry_client_cert_file = /path/to/cert/file
# The path to the certifying authority cert file to use in SSL connections
# to the registry server, if any. Alternately, you may set the
# GLANCE_CLIENT_CA_FILE environ variable to a filepath of the CA cert file
# registry_client_ca_file = /path/to/ca/file
# ============ Notification System Options =====================
# Notifications can be sent when images are create, updated or deleted.
# There are three methods of sending notifications, logging (via the
# log_file directive), rabbit (via a rabbitmq queue), qpid (via a Qpid
# message queue), or noop (no notifications sent, the default)
notifier_strategy = noop
# Configuration options if sending notifications via rabbitmq (these are
# the defaults)
rabbit_host = <%= @rabbit_ipaddress %>
rabbit_port = 5672
rabbit_use_ssl = false
rabbit_userid = guest
rabbit_password = guest
rabbit_virtual_host = /
rabbit_notification_exchange = glance
rabbit_notification_topic = glance_notifications
# Configuration options if sending notifications via Qpid (these are
# the defaults)
qpid_notification_exchange = glance
qpid_notification_topic = glance_notifications
qpid_host = localhost
qpid_port = 5672
qpid_username =
qpid_password =
qpid_sasl_mechanisms =
qpid_reconnect_timeout = 0
qpid_reconnect_limit = 0
qpid_reconnect_interval_min = 0
qpid_reconnect_interval_max = 0
qpid_reconnect_interval = 0
qpid_heartbeat = 5
# Set to 'ssl' to enable SSL
qpid_protocol = tcp
qpid_tcp_nodelay = True
# ============ Filesystem Store Options ========================
# Directory that the Filesystem backend store
# writes image data to
filesystem_store_datadir = /var/lib/glance/images/
# ============ Swift Store Options =============================
# Address where the Swift authentication service lives
# Valid schemes are 'http://' and 'https://'
# If no scheme specified, default to 'https://'
swift_store_auth_address = 127.0.0.1:8080/v1.0/
# User to authenticate against the Swift authentication service
# If you use Swift authentication service, set it to 'account':'user'
# where 'account' is a Swift storage account and 'user'
# is a user in that account
swift_store_user = jdoe:jdoe
# Auth key for the user authenticating against the
# Swift authentication service
swift_store_key = a86850deb2742ec3cb41518e26aa2d89
# Container within the account that the account should use
# for storing images in Swift
swift_store_container = glance
# Do we create the container if it does not exist?
swift_store_create_container_on_put = False
# What size, in MB, should Glance start chunking image files
# and do a large object manifest in Swift? By default, this is
# the maximum object size in Swift, which is 5GB
swift_store_large_object_size = 5120
# When doing a large object manifest, what size, in MB, should
# Glance write chunks to Swift? This amount of data is written
# to a temporary disk buffer during the process of chunking
# the image file, and the default is 200MB
swift_store_large_object_chunk_size = 200
# Whether to use ServiceNET to communicate with the Swift storage servers.
# (If you aren't RACKSPACE, leave this False!)
#
# To use ServiceNET for authentication, prefix hostname of
# `swift_store_auth_address` with 'snet-'.
# Ex. https://example.com/v1.0/ -> https://snet-example.com/v1.0/
swift_enable_snet = False
# ============ S3 Store Options =============================
# Address where the S3 authentication service lives
# Valid schemes are 'http://' and 'https://'
# If no scheme specified, default to 'http://'
s3_store_host = 127.0.0.1:8080/v1.0/
# User to authenticate against the S3 authentication service
s3_store_access_key = <20-char AWS access key>
# Auth key for the user authenticating against the
# S3 authentication service
s3_store_secret_key = <40-char AWS secret key>
# Container within the account that the account should use
# for storing images in S3. Note that S3 has a flat namespace,
# so you need a unique bucket name for your glance images. An
# easy way to do this is append your AWS access key to "glance".
# S3 buckets in AWS *must* be lowercased, so remember to lowercase
# your AWS access key if you use it in your bucket name below!
s3_store_bucket = <lowercased 20-char aws access key>glance
# Do we create the bucket if it does not exist?
s3_store_create_bucket_on_put = False
# When sending images to S3, the data will first be written to a
# temporary buffer on disk. By default the platform's temporary directory
# will be used. If required, an alternative directory can be specified here.
# s3_store_object_buffer_dir = /path/to/dir
# ============ RBD Store Options =============================
# Ceph configuration file path
# If using cephx authentication, this file should
# include a reference to the right keyring
# in a client.<USER> section
rbd_store_ceph_conf = /etc/ceph/ceph.conf
# RADOS user to authenticate as (only applicable if using cephx)
rbd_store_user = glance
# RADOS pool in which images are stored
rbd_store_pool = images
# Images will be chunked into objects of this size (in megabytes).
# For best performance, this should be a power of two
rbd_store_chunk_size = 8
# ============ Delayed Delete Options =============================
# Turn on/off delayed delete
delayed_delete = False
# Delayed delete time in seconds
scrub_time = 43200
# Directory that the scrubber will use to remind itself of what to delete
# Make sure this is also set in glance-scrubber.conf
scrubber_datadir = /var/lib/glance/scrubber
# =============== Image Cache Options =============================
# Base directory that the Image Cache uses
image_cache_dir = /var/lib/glance/image-cache/
[paste_deploy]
flavor = keystone

33
templates/default/glance-registry-paste.ini.erb Normal file
View File

@@ -0,0 +1,33 @@
# Default minimal pipeline
[pipeline:glance-registry]
pipeline = context registryapp
# Use the following pipeline for keystone auth
# i.e. in glance-registry.conf:
# [paste_deploy]
# flavor = keystone
#
[pipeline:glance-registry-keystone]
pipeline = authtoken context registryapp
[app:registryapp]
paste.app_factory = glance.common.wsgi:app_factory
glance.app_factory = glance.registry.api.v1:API
[filter:context]
context_class = glance.registry.context.RequestContext
paste.filter_factory = glance.common.wsgi:filter_factory
glance.filter_factory = glance.common.context:ContextMiddleware
[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = <%= @keystone_api_ipaddress %>
service_port = <%= @keystone_service_port %>
auth_host = <%= @keystone_api_ipaddress %>
auth_port = <%= @keystone_admin_port %>
auth_protocol = http
auth_uri = http://<%= @keystone_api_ipaddress %>:<%= @keystone_service_port %>/
admin_tenant_name = <%= @service_tenant_name %>
admin_user = <%= @service_user %>
admin_password = <%= @service_pass %>

65
templates/default/glance-registry.conf.erb Normal file
View File

@@ -0,0 +1,65 @@
[DEFAULT]
# Show more verbose log output (sets INFO log level output)
verbose = True
# Show debugging output in logs (sets DEBUG log level output)
debug = False
# Address to bind the registry server
bind_host = <%= @registry_bind_address %>
# Port the bind the registry server to
bind_port = <%= @registry_port %>
# Log to this file. Make sure you do not set the same log
# file for both the API and registry servers!
log_file = /var/log/glance/registry.log
# Backlog requests when creating socket
backlog = 4096
# SQLAlchemy connection string for the reference implementation
# registry server. Any valid SQLAlchemy connection string is fine.
# See: http://www.sqlalchemy.org/docs/05/reference/sqlalchemy/connections.html#sqlalchemy.create_engine
sql_connection = mysql://<%= @db_user %>:<%= @db_password %>@<%= @db_ip_address %>/<%= @db_name %>
# Period in seconds after which SQLAlchemy should reestablish its connection
# to the database.
#
# MySQL uses a default `wait_timeout` of 8 hours, after which it will drop
# idle connections. This can result in 'MySQL Gone Away' exceptions. If you
# notice this, you can lower this value to ensure that SQLAlchemy reconnects
# before MySQL can drop the connection.
sql_idle_timeout = 3600
# Limit the api to return `param_limit_max` items in a call to a container. If
# a larger `limit` query param is provided, it will be reduced to this value.
api_limit_max = 1000
# If a `limit` query param is not provided in an api request, it will
# default to `limit_param_default`
limit_param_default = 25
# ================= Syslog Options ============================
# Send logs to syslog (/dev/log) instead of to file specified
# by `log_file`
use_syslog = <%= @use_syslog %>
<% if @use_syslog == true %>
# Facility to use. If unset defaults to LOG_USER.
syslog_log_facility = <%= @log_facility %>
<% else %>
# syslog_log_facility = LOG_USER
<% end %>
# ================= SSL Options ===============================
# Certificate file to use when starting registry server securely
# cert_file = /path/to/certfile
# Private key file to use when starting registry server securely
# key_file = /path/to/keyfile
[paste_deploy]
flavor = keystone

3
templates/default/glance-scrubber-paste.ini.erb Normal file
View File

@@ -0,0 +1,3 @@
[app:glance-scrubber]
paste.app_factory = glance.common.wsgi:app_factory
glance.app_factory = glance.store.scrubber:Scrubber

35
templates/default/glance-scrubber.conf.erb Normal file
View File

@@ -0,0 +1,35 @@
[DEFAULT]
# Show more verbose log output (sets INFO log level output)
verbose = True
# Show debugging output in logs (sets DEBUG log level output)
debug = False
# Log to this file. Make sure you do not set the same log
# file for both the API and registry servers!
log_file = /var/log/glance/scrubber.log
# Send logs to syslog (/dev/log) instead of to file specified by `log_file`
use_syslog = False
# Should we run our own loop or rely on cron/scheduler to run us
daemon = False
# Loop time between checking the db for new items to schedule for delete
wakeup_time = 300
# Directory that the scrubber will use to remind itself of what to delete
# Make sure this is also set in glance-api.conf
scrubber_datadir = /var/lib/glance/scrubber
# Only one server in your deployment should be designated the cleanup host
cleanup_scrubber = False
# pending_delete items older than this time are candidates for cleanup
cleanup_scrubber_time = 86400
# Address to find the registry server for cleanups
registry_host = <%= @registry_ip_address %>
# Port the registry server is listening on
registry_port = <%= @registry_port %>

4
templates/default/policy.json.erb Normal file
View File

@@ -0,0 +1,4 @@
{
"default": [],
"manage_image_cache": [["role:admin"]]
}

565
templates/default/swift.py Normal file
View File

@@ -0,0 +1,565 @@
# vim: tabstop=4 shiftwidth=4 softtabstop=4
# Copyright 2010-2011 OpenStack, LLC
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Storage backend for SWIFT"""
from __future__ import absolute_import
import hashlib
import httplib
import logging
import math
import urlparse
from glance.common import cfg
from glance.common import exception
import glance.store
import glance.store.base
import glance.store.location
try:
from swift.common import client as swift_client
except ImportError:
pass
DEFAULT_CONTAINER = 'glance'
DEFAULT_LARGE_OBJECT_SIZE = 5 * 1024 # 5GB
DEFAULT_LARGE_OBJECT_CHUNK_SIZE = 200 # 200M
ONE_MB = 1000 * 1024
logger = logging.getLogger('glance.store.swift')
class StoreLocation(glance.store.location.StoreLocation):
"""
Class describing a Swift URI. A Swift URI can look like any of
the following:
swift://user:pass@authurl.com/container/obj-id
swift://account:user:pass@authurl.com/container/obj-id
swift+http://user:pass@authurl.com/container/obj-id
swift+https://user:pass@authurl.com/container/obj-id
The swift+http:// URIs indicate there is an HTTP authentication URL.
The default for Swift is an HTTPS authentication URL, so swift:// and
swift+https:// are the same...
"""
def process_specs(self):
self.scheme = self.specs.get('scheme', 'swift+https')
self.user = self.specs.get('user')
self.key = self.specs.get('key')
self.authurl = self.specs.get('authurl')
self.container = self.specs.get('container')
self.obj = self.specs.get('obj')
def _get_credstring(self):
if self.user:
return '%s:%s@' % (self.user, self.key)
return ''
def get_uri(self):
authurl = self.authurl
if authurl.startswith('http://'):
authurl = authurl[7:]
elif authurl.startswith('https://'):
authurl = authurl[8:]
credstring = self._get_credstring()
authurl = authurl.strip('/')
container = self.container.strip('/')
obj = self.obj.strip('/')
return '%s://%s%s/%s/%s' % (self.scheme, credstring, authurl,
container, obj)
def parse_uri(self, uri):
"""
Parse URLs. This method fixes an issue where credentials specified
in the URL are interpreted differently in Python 2.6.1+ than prior
versions of Python. It also deals with the peculiarity that new-style
Swift URIs have where a username can contain a ':', like so:
swift://account:user:pass@authurl.com/container/obj
"""
# Make sure that URIs that contain multiple schemes, such as:
# swift://user:pass@http://authurl.com/v1/container/obj
# are immediately rejected.
if uri.count('://') != 1:
reason = _(
"URI cannot contain more than one occurrence of a scheme."
"If you have specified a URI like "
"swift://user:pass@http://authurl.com/v1/container/obj"
", you need to change it to use the swift+http:// scheme, "
"like so: "
"swift+http://user:pass@authurl.com/v1/container/obj"
)
raise exception.BadStoreUri(uri, reason)
pieces = urlparse.urlparse(uri)
assert pieces.scheme in ('swift', 'swift+http', 'swift+https')
self.scheme = pieces.scheme
netloc = pieces.netloc
path = pieces.path.lstrip('/')
if netloc != '':
# > Python 2.6.1
if '@' in netloc:
creds, netloc = netloc.split('@')
else:
creds = None
else:
# Python 2.6.1 compat
# see lp659445 and Python issue7904
if '@' in path:
creds, path = path.split('@')
else:
creds = None
netloc = path[0:path.find('/')].strip('/')
path = path[path.find('/'):].strip('/')
if creds:
cred_parts = creds.split(':')
# User can be account:user, in which case cred_parts[0:2] will be
# the account and user. Combine them into a single username of
# account:user
if len(cred_parts) == 1:
reason = (_("Badly formed credentials '%(creds)s' in Swift "
"URI") % locals())
raise exception.BadStoreUri(uri, reason)
elif len(cred_parts) == 3:
user = ':'.join(cred_parts[0:2])
else:
user = cred_parts[0]
key = cred_parts[-1]
self.user = user
self.key = key
else:
self.user = None
path_parts = path.split('/')
try:
self.obj = path_parts.pop()
self.container = path_parts.pop()
if not netloc.startswith('http'):
# push hostname back into the remaining to build full authurl
path_parts.insert(0, netloc)
self.authurl = '/'.join(path_parts)
except IndexError:
reason = _("Badly formed Swift URI")
raise exception.BadStoreUri(uri, reason)
@property
def swift_auth_url(self):
"""
Creates a fully-qualified auth url that the Swift client library can
use. The scheme for the auth_url is determined using the scheme
included in the `location` field.
HTTPS is assumed, unless 'swift+http' is specified.
"""
if self.scheme in ('swift+https', 'swift'):
auth_scheme = 'https://'
else:
auth_scheme = 'http://'
full_url = ''.join([auth_scheme, self.authurl.rstrip("/"), "/"])
return full_url
class Store(glance.store.base.Store):
"""An implementation of the swift backend adapter."""
EXAMPLE_URL = "swift://<USER>:<KEY>@<AUTH_ADDRESS>/<CONTAINER>/<FILE>"
CHUNKSIZE = 65536
opts = [
cfg.BoolOpt('swift_enable_snet', default=False),
cfg.StrOpt('swift_store_auth_address'),
cfg.StrOpt('swift_store_user', secret=True),
cfg.StrOpt('swift_store_key', secret=True),
cfg.StrOpt('swift_store_auth_version', default='2'),
cfg.StrOpt('swift_store_container',
default=DEFAULT_CONTAINER),
cfg.IntOpt('swift_store_large_object_size',
default=DEFAULT_LARGE_OBJECT_SIZE),
cfg.IntOpt('swift_store_large_object_chunk_size',
default=DEFAULT_LARGE_OBJECT_CHUNK_SIZE),
cfg.BoolOpt('swift_store_create_container_on_put', default=False),
]
def configure(self):
self.conf.register_opts(self.opts)
self.snet = self.conf.swift_enable_snet
self.auth_version = self._option_get('swift_store_auth_version')
def configure_add(self):
"""
Configure the Store to use the stored configuration options
Any store that needs special configuration should implement
this method. If the store was not able to successfully configure
itself, it should raise `exception.BadStoreConfiguration`
"""
self.auth_address = self._option_get('swift_store_auth_address')
self.user = self._option_get('swift_store_user')
self.key = self._option_get('swift_store_key')
self.container = self.conf.swift_store_container
try:
# The config file has swift_store_large_object_*size in MB, but
# internally we store it in bytes, since the image_size parameter
# passed to add() is also in bytes.
self.large_object_size = \
self.conf.swift_store_large_object_size * ONE_MB
self.large_object_chunk_size = \
self.conf.swift_store_large_object_chunk_size * ONE_MB
except cfg.ConfigFileValueError, e:
reason = _("Error in configuration conf: %s") % e
logger.error(reason)
raise exception.BadStoreConfiguration(store_name="swift",
reason=reason)
self.scheme = 'swift+https'
if self.auth_address.startswith('http://'):
self.scheme = 'swift+http'
self.full_auth_address = self.auth_address
elif self.auth_address.startswith('https://'):
self.full_auth_address = self.auth_address
else: # Defaults https
self.full_auth_address = 'https://' + self.auth_address
self.full_auth_address = ''.join([self.full_auth_address.rstrip("/"), '/'])
def get(self, location):
"""
Takes a `glance.store.location.Location` object that indicates
where to find the image file, and returns a tuple of generator
(for reading the image file) and image_size
:param location `glance.store.location.Location` object, supplied
from glance.store.location.get_location_from_uri()
:raises `glance.exception.NotFound` if image does not exist
"""
loc = location.store_location
swift_conn = self._make_swift_connection(
auth_url=loc.swift_auth_url, user=loc.user, key=loc.key)
try:
(resp_headers, resp_body) = swift_conn.get_object(
container=loc.container, obj=loc.obj,
resp_chunk_size=self.CHUNKSIZE)
except swift_client.ClientException, e:
if e.http_status == httplib.NOT_FOUND:
uri = location.get_store_uri()
raise exception.NotFound(_("Swift could not find image at "
"uri %(uri)s") % locals())
else:
raise
class ResponseIndexable(glance.store.Indexable):
def another(self):
try:
return self.wrapped.next()
except StopIteration:
return ''
length = resp_headers.get('content-length')
return (ResponseIndexable(resp_body, length), length)
def get_size(self, location):
"""
Takes a `glance.store.location.Location` object that indicates
where to find the image file, and returns the image_size (or 0
if unavailable)
:param location `glance.store.location.Location` object, supplied
from glance.store.location.get_location_from_uri()
"""
loc = location.store_location
swift_conn = self._make_swift_connection(
auth_url=loc.swift_auth_url, user=loc.user, key=loc.key)
try:
resp_headers = swift_conn.head_object(container=loc.container,
obj=loc.obj)
return resp_headers.get('content-length', 0)
except Exception:
return 0
def _make_swift_connection(self, auth_url, user, key):
"""
Creates a connection using the Swift client library.
"""
snet = self.snet
auth_version = self.auth_version
logger.debug(_("Creating Swift connection with "
"(auth_address=%(auth_url)s, user=%(user)s, "
"snet=%(snet)s, auth_version=%(auth_version)s)") %
locals())
return swift_client.Connection(
authurl=auth_url, user=user, key=key, snet=snet,
auth_version=auth_version)
def _option_get(self, param):
result = getattr(self.conf, param)
if not result:
reason = (_("Could not find %(param)s in configuration "
"options.") % locals())
logger.error(reason)
raise exception.BadStoreConfiguration(store_name="swift",
reason=reason)
return result
def add(self, image_id, image_file, image_size):
"""
Stores an image file with supplied identifier to the backend
storage system and returns an `glance.store.ImageAddResult` object
containing information about the stored image.
:param image_id: The opaque image identifier
:param image_file: The image data to write, as a file-like object
:param image_size: The size of the image data to write, in bytes
:retval `glance.store.ImageAddResult` object
:raises `glance.common.exception.Duplicate` if the image already
existed
Swift writes the image data using the scheme:
``swift://<USER>:<KEY>@<AUTH_ADDRESS>/<CONTAINER>/<ID>`
where:
<USER> = ``swift_store_user``
<KEY> = ``swift_store_key``
<AUTH_ADDRESS> = ``swift_store_auth_address``
<CONTAINER> = ``swift_store_container``
<ID> = The id of the image being added
:note Swift auth URLs by default use HTTPS. To specify an HTTP
auth URL, you can specify http://someurl.com for the
swift_store_auth_address config option
:note Swift cannot natively/transparently handle objects >5GB
in size. So, if the image is greater than 5GB, we write
chunks of image data to Swift and then write an manifest
to Swift that contains information about the chunks.
This same chunking process is used by default for images
of an unknown size, as pushing them directly to swift would
fail if the image turns out to be greater than 5GB.
"""
swift_conn = self._make_swift_connection(
auth_url=self.full_auth_address, user=self.user, key=self.key)
create_container_if_missing(self.container, swift_conn, self.conf)
obj_name = str(image_id)
location = StoreLocation({'scheme': self.scheme,
'container': self.container,
'obj': obj_name,
'authurl': self.auth_address,
'user': self.user,
'key': self.key})
logger.debug(_("Adding image object '%(obj_name)s' "
"to Swift") % locals())
try:
if image_size > 0 and image_size < self.large_object_size:
# Image size is known, and is less than large_object_size.
# Send to Swift with regular PUT.
obj_etag = swift_conn.put_object(self.container, obj_name,
image_file,
content_length=image_size)
else:
# Write the image into Swift in chunks.
chunk_id = 1
if image_size > 0:
total_chunks = str(int(
math.ceil(float(image_size) /
float(self.large_object_chunk_size))))
else:
# image_size == 0 is when we don't know the size
# of the image. This can occur with older clients
# that don't inspect the payload size.
logger.debug(_("Cannot determine image size. Adding as a "
"segmented object to Swift."))
total_chunks = '?'
checksum = hashlib.md5()
combined_chunks_size = 0
while True:
chunk_size = self.large_object_chunk_size
if image_size == 0:
content_length = None
else:
left = image_size - combined_chunks_size
if left == 0:
break
if chunk_size > left:
chunk_size = left
content_length = chunk_size
chunk_name = "%s-%05d" % (obj_name, chunk_id)
reader = ChunkReader(image_file, checksum, chunk_size)
chunk_etag = swift_conn.put_object(
self.container, chunk_name, reader,
content_length=content_length)
bytes_read = reader.bytes_read
logger.debug(_("Wrote chunk %(chunk_id)d/"
"%(total_chunks)s of length %(bytes_read)d "
"to Swift returning MD5 of content: "
"%(chunk_etag)s")
% locals())
if bytes_read == 0:
# Delete the last chunk, because it's of zero size.
# This will happen if image_size == 0.
logger.debug(_("Deleting final zero-length chunk"))
swift_conn.delete_object(self.container, chunk_name)
break
chunk_id += 1
combined_chunks_size += bytes_read
# In the case we have been given an unknown image size,
# set the image_size to the total size of the combined chunks.
if image_size == 0:
image_size = combined_chunks_size
# Now we write the object manifest and return the
# manifest's etag...
manifest = "%s/%s" % (self.container, obj_name)
headers = {'ETag': hashlib.md5("").hexdigest(),
'X-Object-Manifest': manifest}
# The ETag returned for the manifest is actually the
# MD5 hash of the concatenated checksums of the strings
# of each chunk...so we ignore this result in favour of
# the MD5 of the entire image file contents, so that
# users can verify the image file contents accordingly
swift_conn.put_object(self.container, obj_name,
None, headers=headers)
obj_etag = checksum.hexdigest()
# NOTE: We return the user and key here! Have to because
# location is used by the API server to return the actual
# image data. We *really* should consider NOT returning
# the location attribute from GET /images/<ID> and
# GET /images/details
return (location.get_uri(), image_size, obj_etag)
except swift_client.ClientException, e:
if e.http_status == httplib.CONFLICT:
raise exception.Duplicate(_("Swift already has an image at "
"location %s") % location.get_uri())
msg = (_("Failed to add object to Swift.\n"
"Got error from Swift: %(e)s") % locals())
logger.error(msg)
raise glance.store.BackendException(msg)
def delete(self, location):
"""
Takes a `glance.store.location.Location` object that indicates
where to find the image file to delete
:location `glance.store.location.Location` object, supplied
from glance.store.location.get_location_from_uri()
:raises NotFound if image does not exist
"""
loc = location.store_location
swift_conn = self._make_swift_connection(
auth_url=loc.swift_auth_url, user=loc.user, key=loc.key)
try:
# We request the manifest for the object. If one exists,
# that means the object was uploaded in chunks/segments,
# and we need to delete all the chunks as well as the
# manifest.
manifest = None
try:
headers = swift_conn.head_object(loc.container, loc.obj)
manifest = headers.get('x-object-manifest')
except swift_client.ClientException, e:
if e.http_status != httplib.NOT_FOUND:
raise
if manifest:
# Delete all the chunks before the object manifest itself
obj_container, obj_prefix = manifest.split('/', 1)
for segment in swift_conn.get_container(obj_container,
prefix=obj_prefix)[1]:
# TODO(jaypipes): This would be an easy area to parallelize
# since we're simply sending off parallelizable requests
# to Swift to delete stuff. It's not like we're going to
# be hogging up network or file I/O here...
swift_conn.delete_object(obj_container, segment['name'])
else:
swift_conn.delete_object(loc.container, loc.obj)
except swift_client.ClientException, e:
if e.http_status == httplib.NOT_FOUND:
uri = location.get_store_uri()
raise exception.NotFound(_("Swift could not find image at "
"uri %(uri)s") % locals())
else:
raise
class ChunkReader(object):
def __init__(self, fd, checksum, total):
self.fd = fd
self.checksum = checksum
self.total = total
self.bytes_read = 0
def read(self, i):
left = self.total - self.bytes_read
if i > left:
i = left
result = self.fd.read(i)
self.bytes_read += len(result)
self.checksum.update(result)
return result
def create_container_if_missing(container, swift_conn, conf):
"""
Creates a missing container in Swift if the
``swift_store_create_container_on_put`` option is set.
:param container: Name of container to create
:param swift_conn: Connection to Swift
:param conf: Option mapping
"""
try:
swift_conn.head_container(container)
except swift_client.ClientException, e:
if e.http_status == httplib.NOT_FOUND:
if conf.swift_store_create_container_on_put:
try:
swift_conn.put_container(container)
except swift_client.ClientException, e:
msg = _("Failed to add container to Swift.\n"
"Got error from Swift: %(e)s") % locals()
raise glance.store.BackendException(msg)
else:
msg = (_("The container %(container)s does not exist in "
"Swift. Please set the "
"swift_store_create_container_on_put option"
"to add container to Swift automatically.")
% locals())
raise glance.store.BackendException(msg)
else:
raise
glance.store.register_store(__name__, ['swift', 'swift+http', 'swift+https'])