30 Commits

Author SHA1 Message Date
Russell Bryant
2fa1ed0da1 Implement blueprint add-qpid-support.
This patch implements a new notification strategy that sends
notifications to a message queue via Qpid.

Change-Id: Ie2640139dcf735e428af63e9e87b3667a2b22eb5
2012-02-10 15:36:13 -05:00
Jay Pipes
180ae8964d Adds documentation for policy files.
Implements interim-glance-authz-service

Change-Id: Ifefa4174aa45a29fd58ab40c74b5eaf6fecb4f83
2012-01-24 21:19:26 -05:00
Stuart McLaren
e893b248a2 Multi-process Glance API server support.
Implements blueprint multi-process-server. Allows several Glance API
worker processes to be started, which can increase performance on machines
with more than one CPU.

Change-Id: I1cbb48945fd23afd71de3a30b80836b590c023a1
2012-01-19 09:45:21 -08:00
Eoghan Glynn
097ced8cb2 Make paste.ini file location configurable.
Add an optional paste_deploy.config_file variable to identify
a non-default location for the paste.ini file (the default for
/path/to/component.conf is /path/to/component-paste.ini).

Change-Id: Ibd7aefe95706fb50e4e353c14eb6dab0f948331f
2012-01-17 12:00:51 +00:00
Eoghan Glynn
5835b30cc2 Avoid the need for users to manually edit PasteDeploy config in order to switch pipelines.
Define multiple pipelines in glace-api.conf to reflect the
various supported deployment flavors (minimal, with caching,
with cache management, with keystone-based auth etc.).

Add an optional paste_deploy.flavor config variable to allow the
user select the appropriate pipeline without having to edit the
paste config (i.e. uncommenting lines as before). For example
in glance-api.conf, a setting of:

  [paste_deploy]
  flavor = keystone+caching

identifies the following pipeline in glace-api-paste.ini:

  [pipeline:glance-api-keystone+caching]
  pipeline = versionnegotiation authtoken auth-context cache apiv1app

the advantage being that the user need not be concerned with
the precise sequence of filters required to realize the QoS
they desire.

Modify the functional tests that patch configuration (i.e. the
keystone and caching tests) to use the new deployment_flavor
mechanism.

Extend the TestConfigOpts to support option groups.

Change-Id: Ide843ada11bce115b7dc650440397853c6409b03
2012-01-17 12:00:33 +00:00
Eoghan Glynn
14593a3b96 Split out paste deployment config from the core glance
*.conf files into corresponding *-paste.ini files.

Fixes LP #815208

Change-Id: I5acb8488cc25bb3c7dd735c8c2b0fe62fcea8e5e
2012-01-17 11:40:26 +00:00
Chris Behrens
195e6670c4 Implement retries in notify_kombu
Fixes bug 915817

Copies code from nova and includes improvements pending for nova as well

Change-Id: Ib596f4bacc5f58c4507f4111cdfac273029018bc
2012-01-12 23:49:41 -08:00
Ewan Mellor
48449171a7 Bug #909533: Swift uploads through Glance using ridiculously small chunks
Remove the use of a temporary disk buffer when streaming to Swift.  This
was added on the assumption that it was not possible to stream chunks to
Swift direct from webob.Request.body_file.  That's not true -- a simple
file-like object wrapping body_file does the job perfectly.

This removes the need for swift_store_object_buffer_dir in the config file.
It is also significantly cheaper, since there is one copy fewer.

Fix the parsing of the swift_store_large_object_size and
swift_store_large_object_chunk_size options.  These are
specified in MB in the config file, but needs to be in bytes internally,
because they are compared against the image_size parameter given to add().

Update the unit tests to match.  This includes an additional check that
put_object is called the correct number of times.  The unit tests missed the
latter problem problem because there was no check that a small object is only
uploaded in one chunk, and despite the comment to the contrary there was no
check that the right number of chunks were written in the large-object case
either.  Added these in all places where store.add is called.

Change-Id: Ieb4cf68516b53bd16d2671e49e805b26118b3671
2012-01-06 19:44:49 -05:00
Johannes Erdfelt
fa1b0b1f78 Synchronize notification queue setup between nova and glance
Fixes bug 901376

Glance used a SimpleQueue which would end up with a direct queue with an
exchange named the same as the queue. This was different than Nova, which
uses a topic queue with an exchange named 'nova'.

This change makes Glance use a topic queue with a configurable exchange
name to match Nova.

Change-Id: Ia014e4c00060abc2345289a54e45bbfdc6b7e8e5
2011-12-29 00:03:19 +00:00
Mark McLoughlin
ba44d1c384 Add generic PasteDeploy app and filter factories
These generic factories allow us to dump the copied and pasted
app_factory and filter_factory methods in the codebase.

The main difference is the paste configuration changes from:

  [app:apiv1app]
  paste.app_factory = glance.api.v1:app_factory
  ...
  [filter:cache]
  paste.filter_factory = glance.api.middleware.cache:filter_factory

to this:

  [app:apiv1app]
  paste.app_factory = glance.common.wsgi:app_factory
  glance.app_factory = glance.api.v1:API
  ...
  [filter:cache]
  paste.filter_factory = glance.common.wsgi:filter_factory
  glance.filter_factory = glance.api.middleware.cache:CacheFilter

Apart from reducing code duplication, this will also allow us to have
the generic factories inject other data into the apps and filters.

Change-Id: I1d2be5630ab61d29b8948ff88d58e6e1b11c2e5f
2011-12-07 12:42:03 +00:00
Lorin Hochstein
6bf61e8ce1 Added some examples for "glance add"
Fixed some warnings.

Change-Id: I9658c7458f3073b4bab93d0105d6ba174a4d0ca1
2011-11-29 13:10:50 -05:00
Paul Bourke
49d67bd505 Adds option to set custom data buffer dir
Implements blueprint custom-disk-buffer

When pushing an image large enough to be chunked into either a Swift
or S3 backend, the chunks are buffered to /tmp.

This change allows a user to define an alternate directory via the
configuration files.

Change-Id: I2660b6e5a76cbfd43834384b129550b66bb9afce
2011-11-22 17:08:32 +00:00
Jay Pipes
2a5a0e9a35 Fixes LP Bug#878411 - No docs for image cache
Adds documentation on managing the image cache

Adds a simple CLI program to queue an image for
caching.

Change-Id: I2b4c9708c521fdcfaf1c6b8f07505e130923c4b7
2011-11-09 13:40:49 -05:00
Jay Pipes
2418efd7b7 Fixes LP Bug#882185 - Document Swift HTTPS default
Documentation should mention Swift auth address
HTTPS by default.

Change-Id: Ibe0fc1c3590961eb4e025a2fc86b54ac82d9d63d
2011-11-01 12:32:53 -04:00
Jay Pipes
39c8557434 Adds Driver Layer to Image Cache
Fixes LP Bug#879136 - keyerror: 'image' when doing nova image-list
Fixes LP Bug#819936 - New image cache breaks Glance on Windows

This patch refactors the image cache further by adding an
adaptable driver layer to the cache. The existing filesystem-based
driver that depended on python-xattr and conditional fstab support
has been moved to /glance/image_cache/drivers/xattr.py, and a new
default driver is now based on SQLite and has no special requirements.

The image cache now contains a simple interface for pruning the
cache. Instead of the logic being contained in
/glance/image_cache/pruner.py, now the prune logic is self-contained
within the ImageCache.prune() method, with pruning calling the
simple well-defined driver methods of get_least_recently_accessed()
and get_cache_size().

Adds a functional test case for the caching middleware and adds
documentation on how to configure the image cache drivers.

TODO: cache-manage middleware...
TODO: cache management docs

Change-Id: Id7ae73549d6bb39222eb7ac0427b0083fd1af3ec
2011-10-25 12:54:31 -04:00
Jay Pipes
ad9e9ca3f7 Overhauls the image cache to be truly optional
Fixes LP Bug#874580 - keyerror 'location' when fetch errors
Fixes LP Bug#817570 - Make new image cache a true extension
Fixes LP Bug#872372 - Image cache has virtually no unit test coverage

* Adds unit tests for the image cache (coverage goes from 26% to 100%)
* Removes caching logic from the images controller and places it into
  a removeable transparent caching middleware
* Adds a functional test case that verifies caching of an image
  and subsequent cache hits
* Removes the image_cache_enabled configuration variable, since it's
  now enabled by simply including the cache in the application
  pipeline
* Adds a singular glance-cache.conf to etc/ that replaces the
  multiple glance-pruner.conf, glance-reaper.conf and
  glance-prefetcher.conf files
* Adds documentation on enabling and configuring the image cache

TODO: Add documentation on the image cache utilities, like reaper,
      prefetcher, etc.

Change-Id: I58845871deee26f81ffabe1750adc472ce5b3797
2011-10-19 16:35:35 -04:00
Jay Pipes
eec5c1afa1 Adds SSL configuration params to the client
* Adds SSL configuration params to all client classes
* Adds unit test for bad SSL client configuration
* Refactors the registry methods to no longer need
  configuration options passed, and to configure the
  registry client once, on images.Controller.__init__
* Adds glance-api.conf options for SSL support in
  registry client connections.
* Adds SSL CA file socket wrappers via a client auth
  HTTPS connection class
* Adds server SSL support, but not functional tests
  for SSL support yet. Still need to research self-signed
  cert generation for tests...
* Updates documentation for bind and startup options
* Adds functional test case for secure communication
  with API server stood up with SSL supprt. Note it is
  not very DRY. There is some DRY cleanup to do in the
  future...

TODO: Integrate options with bin/glance CLI tool

Change-Id: Ie9fcd36337cc93fd5beeabb9186ad5e93ae2a0f0
2011-10-11 16:40:06 -04:00
Josh Durgin
42300518ea Add RBD store backend.
To use this you'll need the latest version of Ceph from git,
which will become version 0.37.

Change-Id: Ia2126395525759eb8c11df055bdc6f286dea97b9
2011-09-29 13:54:19 -07:00
Adam Gandelman
706a53396c Remove PWD from possible config_file_dirs.
Resolves bug #828719, which can potentially cause problems depending on
what $PWD commands are executed in.

Amendment: Also update documentation accordingly.

Change-Id: Ib2ec4bb07222e51c86b7d8f8ce1dd0e6bb342a64
2011-09-02 13:29:15 -07:00
Jay Pipes
501b14bf94 Fixes LP Bug #827660 - Swift driver fail 5G upload
Fixes LP Bug #827660 - Swift driver fails to
upload image files greater than 5GB. The swift
store driver was not using the large object
manifest methods required to upload objects
greater than 5GB into Swift.

Adds configuration options to specify the
size in MB that an image file should be considered
for large object manifest, and a configurable
chunk size. We use a disk buffer of this size,
adding chunks of the image file into Swift and
then after all chunks are saved, add the manifest
to Swift.

Change-Id: I4b4421aa1bba584ab242016a041e864ef19d0214
2011-08-24 14:38:43 -04:00
Johannes Erdfelt
091aae8a6d Add notifications for uploads, updates and deletes
Change-Id: I372f77fe2d1a575f2108c9b8d1f69301c0d5eb5e
2011-08-15 19:58:57 +00:00
jaypipes@gmail.com
dfc71f44f2 Typo in error condition for create_bucket_on_put, make body seekable in req object, and remove +glance from docs and configs 2011-07-26 10:26:19 -04:00
jaypipes@gmail.com
684d89117e Fixed review stuff from Brian 2011-07-25 13:45:35 -04:00
jaypipes@gmail.com
94b9c19c40 Fix merge conflict... 2011-05-11 19:52:06 -04:00
jaypipes@gmail.com
7f6944c816 Doc and docstring fixes from Dan's review 2011-05-10 12:26:29 -04:00
jaypipes@gmail.com
1359d6e2cc Fix up find_config_file() to accept an app_name arg. Update all documentation referencing config files. 2011-05-09 14:55:46 -04:00
jaypipes@gmail.com
4db9df77d0 Improve logging configuration docs... 2011-04-26 16:10:33 -07:00
jaypipes@gmail.com
4eaf065305 Updates glance-manage to use configuration files as well
as command line options.

Completes documentation on configuring Glance's servers
and CLI tools.
2011-04-04 13:00:26 -04:00
jaypipes@gmail.com
75575b41c6 Adds documentation on configuring logging and a test that log_file works. It didn't, so this also inludes fixes for setting up log handling :) 2011-03-08 16:52:38 -05:00
jaypipes@gmail.com
66f4b78234 Adds bin/glance-control program server daemonization wrapper
program based on Swift's swift-init script.

Adds a crap-ton of documentation on installing Glance and
controlling Glance's server programs.

This was necessary since when glance.common.server was removed
from Glance and paste.deploy used, we lost the ability to control
server daemonization with CLI options. This adds back some of that
functionality while still using paste.deploy like Swift does.
2011-02-04 18:59:52 -05:00