Commit Graph

71 Commits

Author SHA1 Message Date
Tom Hancock
d1f8e8b009 Add insecure option to registry https client
This fixes LP bug #1063793.

This adds a new API config option registry_client_insecure
(default false) which optionally allows not to specify a ca
file for registry connections from the API server.
The default is false so existing behaviour is
unchanged unless this option is specified.
Adding this option makes the registry symmetric wrt the API
with which you can connect using clients with --insecure

Change-Id: I4fa462cd68afaf14f7d7de6c572980a41c0849d6
2012-10-09 10:43:57 +00:00
Eoghan Glynn
eafed25ba6 Add rabbit_durable_queues config option.
Fixes bug 1054183

Avoid AMQPChannelException: (406, "PRECONDITION_FAILED...") failures
due to a mismatch between the durability of rabbitmq exchange/queue
declared by glance and ceilometer.

Change-Id: I4e25986a1f503782e701aa1168c4eb231ff25d06
2012-09-21 18:26:33 +01:00
Eoghan Glynn
72f54521e4 Enable multi-processing by default.
In realistic production environments, multi-processing is likely
to be enabled.

A default setting of one worker would allow the multi-processing
codepath to be followed by default, without changing the number
of processes actually servicing API requests (i.e. the single
parent process will just wait on the child).

For most functional tests, we continue to explicitly configure
zero workers, so as to avoid spawning twice as many glance-api
processes during a test run.

Change-Id: I1c327805f8fb975a3175e9a8b40aeddd21d5ef91
2012-09-19 20:50:38 +01:00
Mark Washenberger
173d821463 Add region configuration for swift
Addresses bug 1050034

Change-Id: Ib83f5dbb2c1fbeacd741e0cb92b619e67b2243c7
2012-09-13 17:43:10 +00:00
Brian Waldon
4de8670bcb Add s3_store_bucket_url_format config option
Swift's implementation of the S3 interface doesn't support subdomain
access to containers. It requires that the bucket be prepended to the
request path. The option 's3_store_bucket_url_format' can be set to
either 'path' or 'subdomain' (default) to control how boto forms
the bucket url.

Fixes bug 997658

Change-Id: Ia6e1e7356eef7ac2267f7738e2f4a7c70dc12eeb
2012-09-11 11:03:59 -07:00
Brian Waldon
8e17990406 Remove extraneous whitespace in config files
One can not simply remove the hash mark in front of a commented-out
config option if there is a space following it. This patch removes
that extraneous whitespace.

Change-Id: Id084eef34a9e894de3f67b2919d55879cd9a715e
2012-09-05 16:37:30 -07:00
isethi
39700637e3 Makes deployed APIs configurable
Uses config options enable_v1_api and enable_v2_api
to control which APIs have been deployed.
(Thanks markwash for all the help)

Fixes bug 1043497

Change-Id: I42dc1d88f94e61b9550c5f114ffc1abad25be7ff
2012-09-05 21:01:25 +00:00
Brian Waldon
9736d299a4 Clean up comments in paste files
* Reduce comments in paste files to just what is necessary
* Add a 'cachemanage' pipeline to complete our set of paste
  deploy flavors in glance-api-paste.ini
* Add a sample [paste_deploy] section to both glance-api and
  glance-registry configs

Related to bp glance-folsom-docs-cleanup.

Change-Id: I0208b12f6861b48e52260b25802c47ebf6316b9a
2012-09-03 12:51:56 -07:00
Brian Waldon
b7197041fe Move authtoken config out of paste
Since we can now configure Keystone's auth_token middleware using
our own config files, we should ship our example config to show
how this is done. This moves the auth_* variables out of
glance-api-paste.ini and glance-registry-paste.ini into
glance-api.conf and glance-registry.conf.

Fixes bug 1043490

Change-Id: Ibdaf8b2ef0f55e5e892fbd1b994dceb323f07c7b
2012-08-29 12:24:52 -07:00
Brian Waldon
3a80eee8ee Make max image size upload configurable
Add conf option image_size_cap to represent what the internal
IMAGE_SIZE_CAP constant used to cover in the v1 API. The default
is reduced to 1 TB from 50 PB.

Thoroughly test this on image create and update through the v1 API.

Fixes bug 1038994

Change-Id: Ie0985b62228f8b28c005a8847049d7b68d9b959a
2012-08-20 08:01:00 -04:00
Brian Waldon
006254c505 Allow server-side validation of client ssl certs
This adds a 'ca_file' config option that points to a local
CA cert that will be used to verify certs provided by
connecting clients. The 'ca_file' option is only used if the
server is already properly configured to to use SSL - that
means having a valid 'cert_file' and 'key_file'. If no 'ca_file'
is provided, the behavior will remain the same - the server
will still provide its cert to clients, but it will ignore
certs sent back from those clients.

Fixes bug 1032451

Change-Id: Ie48646b0fc5398ba7cda2fb627b820f533482e00
2012-08-06 11:45:34 -07:00
Jenkins
ab8bc2ab3b Merge "Make TCP_KEEPIDLE configurable" 2012-08-03 18:15:50 +00:00
Dan Prince
8a4f3773ac Add swift_store_admin_tenants option.
Add a swift storage option to configure a default list of admin
tenant IDs which will be granted write access on any swift
images.

This setting might be useful when a user choose not to use a global
swift admin role when managing glance images. This might also
be useful when implementing delayed delete for glance swift
multi-tenant.

Partially implements blueprint: swift-tenant-specific-storage.

Change-Id: Id65e1c8d114c9551b99f6a18c527ba26aefd341f
2012-08-01 13:43:38 -04:00
Stuart McLaren
efb30dd640 Make TCP_KEEPIDLE configurable
Addresses LP bug #1031794.

Change-Id: I9a3a6a23b1bcec5229b15234512e04cc47c3ffd9
2012-08-01 15:04:40 +00:00
Dan Prince
8b2d038185 Adds multi tenant support for swift backend.
Updates the swift store to support multiple tenants:

 * Added configuration option for swift_store_multi_tenant.

 * Updated the swift connection creation logic so that in multi-tenant
   mode the token and storage URL from the service catalog are used
   to create swift connection.

 * When in multi-tenant mode locations URL's (stored in the DB) do
   not contain hard coded swift credentials.

Includes unit tests to verify multi-tenant swift storage URLs.

Partially implements blueprint: swift-tenant-specific-storage.

Change-Id: I45fc97027e6f211ac353513c2d9d6da51ccf4489
2012-07-20 10:12:02 -04:00
Dan Prince
15c204af75 Update default values for known_stores config.
Updates the default value for the 'known_stores' config
variable to include the full list of storage classes
supported in Glance.

Also, removes the known_stores config section from the
example config file since it is arguably an advanced config
that won't get used by most users. Removing this config avoids
the overhead of maintaining internal class names in config files.

Fixes LP Bug #1008698.

Change-Id: I0117376aa4de3103410ecb1a36df6998fcd0d5b5
2012-06-08 15:37:29 -04:00
Brian Waldon
2a01e6ef75 Enable anonymous access through context middleware
Certain deployments need to allow anonymous access to its
images. This patch allows a user that has failed to authenticate with
Keystone to access the API in read-only context.

* Configure through 'allow_anonymous_access' option, defaults to False
* Implements bp api-v2-anonymous-access

Change-Id: Ia8f57e54bd141a2da1ca4600d1970558fb497f67
2012-05-28 20:44:14 +01:00
Dan Prince
a92b01fe58 Adds sql_ config settings to glance-api.conf.
The Glance v2 API makes use of the DB directly. As such
we should include the sql_ configs in the default glance-api.conf
file.

Fixes LP Bug #1003504.

Change-Id: I5e287dafcd65864f56789ae22bab3dcc6b353a47
2012-05-23 11:56:10 -04:00
Joshua Harlow
ef475e17d5 Implements blueprint import-dynamic-stores.
Make glance more pluggable with regard to stores.

Change-Id: I7b264d1b047a321f7b60857bb73154f831b82a7b
2012-05-16 09:39:49 -07:00
Chmouel Boudjnah
fe3ae1d73b Update glance to pass properly tenant_name.
- Update glance with the latest changes in swift.common.client.
- Use 5000 for port.
- Properly test v2 in tests.
- Fixes bug 980196.
- Test swift against a specific stable version like for keystone.

Change-Id: Iaff536efc92e78e51d8042010ce0eb407c4a71b2
2012-05-02 13:31:09 +01:00
Juerg Haefliger
633f83109f Add support for auth version 2.
Implements version 2 authentication for the Swift backend.
Fixes bug 944946.

Change-Id: I614ad03374a2e31a945f12fb9588947022203f1f
2012-03-28 10:29:40 +02:00
Brian Waldon
eff04c5612 Add admin_role option
* Allow a deployer to define the role that should give an authenticated
  user admin privileges. Defaults to 'admin'
* Fixes bug 956547

Change-Id: I018b9c438a12c527833ed8e2e6c377a93fde9130
2012-03-16 11:50:20 -07:00
Russell Bryant
2fa1ed0da1 Implement blueprint add-qpid-support.
This patch implements a new notification strategy that sends
notifications to a message queue via Qpid.

Change-Id: Ie2640139dcf735e428af63e9e87b3667a2b22eb5
2012-02-10 15:36:13 -05:00
Stuart McLaren
e893b248a2 Multi-process Glance API server support.
Implements blueprint multi-process-server. Allows several Glance API
worker processes to be started, which can increase performance on machines
with more than one CPU.

Change-Id: I1cbb48945fd23afd71de3a30b80836b590c023a1
2012-01-19 09:45:21 -08:00
Eoghan Glynn
14593a3b96 Split out paste deployment config from the core glance
*.conf files into corresponding *-paste.ini files.

Fixes LP #815208

Change-Id: I5acb8488cc25bb3c7dd735c8c2b0fe62fcea8e5e
2012-01-17 11:40:26 +00:00
Jenkins
883bdd8564 Merge "Fixes incorrect URI scheme for s3 backend" 2012-01-10 02:51:01 +00:00
Jenkins
e27b0e1c27 Merge "Add comments for swift options in glance-api.conf." 2012-01-10 02:41:04 +00:00
Ewan Mellor
48449171a7 Bug #909533: Swift uploads through Glance using ridiculously small chunks
Remove the use of a temporary disk buffer when streaming to Swift.  This
was added on the assumption that it was not possible to stream chunks to
Swift direct from webob.Request.body_file.  That's not true -- a simple
file-like object wrapping body_file does the job perfectly.

This removes the need for swift_store_object_buffer_dir in the config file.
It is also significantly cheaper, since there is one copy fewer.

Fix the parsing of the swift_store_large_object_size and
swift_store_large_object_chunk_size options.  These are
specified in MB in the config file, but needs to be in bytes internally,
because they are compared against the image_size parameter given to add().

Update the unit tests to match.  This includes an additional check that
put_object is called the correct number of times.  The unit tests missed the
latter problem problem because there was no check that a small object is only
uploaded in one chunk, and despite the comment to the contrary there was no
check that the right number of chunks were written in the large-object case
either.  Added these in all places where store.add is called.

Change-Id: Ieb4cf68516b53bd16d2671e49e805b26118b3671
2012-01-06 19:44:49 -05:00
Hengqing Hu
d466cec357 Fixes incorrect URI scheme for s3 backend
Fixes bug #912094.

URI scheme for s3 backend should be 's3+https', not 'swift+https'.

Use s3.amazonaws.com instead of s3service.com to follow RFC2606.

Indent reason for BadStoreUri exception in s3.py.

Comments added for the following option in etc/glance-api.conf
    's3_store_host'

Change-Id: Id456b287d82fe61711f0d738e61ccb599f0b5684
2012-01-05 19:24:42 +08:00
Hengqing Hu
9633697ae0 Add comments for swift options in glance-api.conf.
Fixes bug #771465.

Comments added for the following two options in
glance-api.conf
    'swift_store_auth_address'
    'swift_store_user'

Indent reason for BadStoreUri exception in swift.py.

Change-Id: Ife28e01a4e2344cf1e5a179752d244b68e45d348
2012-01-05 16:29:31 +08:00
Brian Waldon
add97aa9f3 Add ability to specify syslog facility
Addresses bug 887596.

Change-Id: Id5ae9bc2f37b39cdd4236d5df24273cda058e759
2012-01-03 08:55:14 -05:00
Johannes Erdfelt
fa1b0b1f78 Synchronize notification queue setup between nova and glance
Fixes bug 901376

Glance used a SimpleQueue which would end up with a direct queue with an
exchange named the same as the queue. This was different than Nova, which
uses a topic queue with an exchange named 'nova'.

This change makes Glance use a topic queue with a configurable exchange
name to match Nova.

Change-Id: Ia014e4c00060abc2345289a54e45bbfdc6b7e8e5
2011-12-29 00:03:19 +00:00
Mark McLoughlin
ba44d1c384 Add generic PasteDeploy app and filter factories
These generic factories allow us to dump the copied and pasted
app_factory and filter_factory methods in the codebase.

The main difference is the paste configuration changes from:

  [app:apiv1app]
  paste.app_factory = glance.api.v1:app_factory
  ...
  [filter:cache]
  paste.filter_factory = glance.api.middleware.cache:filter_factory

to this:

  [app:apiv1app]
  paste.app_factory = glance.common.wsgi:app_factory
  glance.app_factory = glance.api.v1:API
  ...
  [filter:cache]
  paste.filter_factory = glance.common.wsgi:filter_factory
  glance.filter_factory = glance.api.middleware.cache:CacheFilter

Apart from reducing code duplication, this will also allow us to have
the generic factories inject other data into the apps and filters.

Change-Id: I1d2be5630ab61d29b8948ff88d58e6e1b11c2e5f
2011-12-07 12:42:03 +00:00
Jay Pipes
1ab63ff5c9 Fixes LP Bug#861650 - Glance client deps
This patch addresses the dependency proliferation in
the glance client stuff. It removes references to modules
that contain non-client-necessary libraries (like
eventlet, xattr, sqlalchemy-migrate and sqlalchemy)
by restructuring the modules slightly.

Note that the additional httplib2 dependency is because
that is used in the authentication strategy stuff in
glance.common.auth. This could be rewritten to use httplib
instead, further reducing the dependencies of the client lib.

IMPORTANT NOTE: This patch changes the default entrypoint
for the Images API router application, and therefore this
should be merged along with the packaging changes in this
branch:

https://code.launchpad.net/~jaypipes/glance/ubuntu/+merge/82318

Change-Id: I5dbc8584fb77e3e011fb6ff3532f792f5103e242
2011-11-30 09:41:22 -05:00
Paul Bourke
49d67bd505 Adds option to set custom data buffer dir
Implements blueprint custom-disk-buffer

When pushing an image large enough to be chunked into either a Swift
or S3 backend, the chunks are buffered to /tmp.

This change allows a user to define an alternate directory via the
configuration files.

Change-Id: I2660b6e5a76cbfd43834384b129550b66bb9afce
2011-11-22 17:08:32 +00:00
Rick Harris
94dcf3acd0 Using Keystone's new port number 35357.
Fixes bug 888753

Change-Id: I22894a31ee54edd6d00ac3458ea4a2f8842aa4a0
2011-11-10 23:33:45 +00:00
Mark McLoughlin
2c66c6972d Remove unused versions pipeline from PasteDeploy config
Since the version negotiation middleware was introduced, versions
pipeline app in the config is unused. Consequently, the versions
app factory is unused too.

Change-Id: I2d50432de215e51c1e52f8d68eaa0820e9b5bd0b
2011-11-09 22:56:19 +00:00
Stuart McLaren
5e6fb33b22 Adds option to encrypt 'location' metadata.
Implements blueprint swift-location-credentials

When the new option is enabled the location metadata
(which may include user credentials) is encrypted
before being sent to the registry server.

Change-Id: I072e3f5c81f187435b1c156627076d5fde087af5
2011-11-03 17:25:01 +00:00
Jay Pipes
39c8557434 Adds Driver Layer to Image Cache
Fixes LP Bug#879136 - keyerror: 'image' when doing nova image-list
Fixes LP Bug#819936 - New image cache breaks Glance on Windows

This patch refactors the image cache further by adding an
adaptable driver layer to the cache. The existing filesystem-based
driver that depended on python-xattr and conditional fstab support
has been moved to /glance/image_cache/drivers/xattr.py, and a new
default driver is now based on SQLite and has no special requirements.

The image cache now contains a simple interface for pruning the
cache. Instead of the logic being contained in
/glance/image_cache/pruner.py, now the prune logic is self-contained
within the ImageCache.prune() method, with pruning calling the
simple well-defined driver methods of get_least_recently_accessed()
and get_cache_size().

Adds a functional test case for the caching middleware and adds
documentation on how to configure the image cache drivers.

TODO: cache-manage middleware...
TODO: cache management docs

Change-Id: Id7ae73549d6bb39222eb7ac0427b0083fd1af3ec
2011-10-25 12:54:31 -04:00
Jay Pipes
ad9e9ca3f7 Overhauls the image cache to be truly optional
Fixes LP Bug#874580 - keyerror 'location' when fetch errors
Fixes LP Bug#817570 - Make new image cache a true extension
Fixes LP Bug#872372 - Image cache has virtually no unit test coverage

* Adds unit tests for the image cache (coverage goes from 26% to 100%)
* Removes caching logic from the images controller and places it into
  a removeable transparent caching middleware
* Adds a functional test case that verifies caching of an image
  and subsequent cache hits
* Removes the image_cache_enabled configuration variable, since it's
  now enabled by simply including the cache in the application
  pipeline
* Adds a singular glance-cache.conf to etc/ that replaces the
  multiple glance-pruner.conf, glance-reaper.conf and
  glance-prefetcher.conf files
* Adds documentation on enabling and configuring the image cache

TODO: Add documentation on the image cache utilities, like reaper,
      prefetcher, etc.

Change-Id: I58845871deee26f81ffabe1750adc472ce5b3797
2011-10-19 16:35:35 -04:00
Jenkins
d3a31b372e Merge "Adds SSL configuration params to the client" 2011-10-18 14:48:01 +00:00
Kevin L. Mitchell
5b27c663b9 Better document using Glance with Keystone.
Addresses bug 871803 by expounding on what configuration should
look like.  Also fixes the example config files, which generally
had the authentication setup completely wrong.

Change-Id: I44b2b8bd340ca95b5a2c2e9408797b0308000a65
2011-10-13 17:03:39 +00:00
Jay Pipes
eec5c1afa1 Adds SSL configuration params to the client
* Adds SSL configuration params to all client classes
* Adds unit test for bad SSL client configuration
* Refactors the registry methods to no longer need
  configuration options passed, and to configure the
  registry client once, on images.Controller.__init__
* Adds glance-api.conf options for SSL support in
  registry client connections.
* Adds SSL CA file socket wrappers via a client auth
  HTTPS connection class
* Adds server SSL support, but not functional tests
  for SSL support yet. Still need to research self-signed
  cert generation for tests...
* Updates documentation for bind and startup options
* Adds functional test case for secure communication
  with API server stood up with SSL supprt. Note it is
  not very DRY. There is some DRY cleanup to do in the
  future...

TODO: Integrate options with bin/glance CLI tool

Change-Id: Ie9fcd36337cc93fd5beeabb9186ad5e93ae2a0f0
2011-10-11 16:40:06 -04:00
Josh Durgin
42300518ea Add RBD store backend.
To use this you'll need the latest version of Ceph from git,
which will become version 0.37.

Change-Id: Ia2126395525759eb8c11df055bdc6f286dea97b9
2011-09-29 13:54:19 -07:00
Jason Kölker
27b3df281f * Scrubber now uses registry client to communicate with registry
* glance-api writes out to a scrubber "queue" dir on delete
* Scrubber determines images to deleted from "queue" dir not db

Change-Id: Ia5574fc75f1a9c763bdef0f5773c2c182932b68a
2011-09-12 18:29:19 -05:00
Dan Prince
22d7864a41 Update paste config files with keystone examples.
see ticket: lp839559

Change-Id: Ie5d5011a601962bcc79cf69ef8db11029fcb866e
2011-09-02 09:32:54 -04:00
Jay Pipes
501b14bf94 Fixes LP Bug #827660 - Swift driver fail 5G upload
Fixes LP Bug #827660 - Swift driver fails to
upload image files greater than 5GB. The swift
store driver was not using the large object
manifest methods required to upload objects
greater than 5GB into Swift.

Adds configuration options to specify the
size in MB that an image file should be considered
for large object manifest, and a configurable
chunk size. We use a disk buffer of this size,
adding chunks of the image file into Swift and
then after all chunks are saved, add the manifest
to Swift.

Change-Id: I4b4421aa1bba584ab242016a041e864ef19d0214
2011-08-24 14:38:43 -04:00
Johannes Erdfelt
091aae8a6d Add notifications for uploads, updates and deletes
Change-Id: I372f77fe2d1a575f2108c9b8d1f69301c0d5eb5e
2011-08-15 19:58:57 +00:00
Rick Harris
c451de83b9 Adds syslog support
Change-Id: I05ef04c77b68f0c68d86ae49a5fee68c2f86c80a
2011-08-11 20:15:33 +00:00
Rick Harris
09a67e4fb4 Adds swift_enable_snet config
Change-Id: I7b798e2b4256a1b6c4694f0d126d2fa4604c95aa
2011-08-05 11:54:52 -05:00