openstack
/
tempest
Code Issues Proposed changes

Migrate service_clients to tempest.lib 

Migrate the service_clients module to tempest.lib.services.clients.
Migrate related unit tests as well.

The clients module atm imports plugin.py from Tempest which is not
allowed via hacking to avoid cirtular dependencies.
If there is no way around this, I will have to remove the self
registration of the service clients from plugins, and ask the
plugins to do the registration themselves - which is a pity. Ideas?

Change-Id: I40e3478f69af62a7cdc14fa65ed21dcfbbe10e72
This commit is contained in:
Andrea Frittoli (andreaf) 2016-08-05 07:27:02 +01:00
parent 40c9d6b8b0
commit e07579c603
10 changed files with 640 additions and 478 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/source/library.rst
View File

@ -67,3 +67,4 @@ Current Library APIs
library/utils
library/api_microversion_testing
library/auth
library/clients

24
doc/source/library/clients.rst Normal file
View File

@ -0,0 +1,24 @@
.. _clients:
Service Clients Usage
=====================
Tests make requests against APIs using service clients. Service clients are
specializations of the ``RestClient`` class. The service clients that cover the
APIs exposed by a service should be grouped in a service clients module.
A service clients module is python module where all service clients are
defined. If major API versions are available, submodules should be defined,
one for each version.
The ``ClientsFactory`` class helps initializing all clients of a specific
service client module from a set of shared parameters.
The ``ServiceClients`` class provides a convenient way to get access to all
available service clients initialized with a provided set of credentials.
------------------
The clients module
------------------
.. automodule:: tempest.lib.services.clients
:members:

141
doc/source/plugin.rst
View File

@ -111,8 +111,9 @@ you would do something like the following::
class MyPlugin(plugins.TempestPlugin):
Then you need to ensure you locally define all of the methods in the abstract
class, you can refer to the api doc below for a reference of what that entails.
Then you need to ensure you locally define all of the mandatory methods in the
abstract class, you can refer to the api doc below for a reference of what that
entails.
Abstract Plugin Class
---------------------
@ -164,6 +165,142 @@ When adding configuration options the ``register_opts`` method gets passed the
CONF object from tempest. This enables the plugin to add options to both
existing sections and also create new configuration sections for new options.
Service Clients
---------------
If a plugin defines a service client, it is beneficial for it to implement the
``get_service_clients`` method in the plugin class. All service clients which
are exposed via this interface will be automatically configured and be
available in any instance of the service clients class, defined in
``tempest.lib.services.clients.ServiceClients``. In case multiple plugins are
installed, all service clients from all plugins will be registered, making it
easy to write tests which rely on multiple APIs whose service clients are in
different plugins.
Example implementation of ``get_service_clients``::
def get_service_clients(self):
# Example implementation with two service clients
my_service1_config = config.service_client_config('my_service')
params_my_service1 = {
'name': 'my_service_v1',
'service_version': 'my_service.v1',
'module_path': 'plugin_tempest_tests.services.my_service.v1',
'client_names': ['API1Client', 'API2Client'],
}
params_my_service1.update(my_service_config)
my_service2_config = config.service_client_config('my_service')
params_my_service2 = {
'name': 'my_service_v2',
'service_version': 'my_service.v2',
'module_path': 'plugin_tempest_tests.services.my_service.v2',
'client_names': ['API1Client', 'API2Client'],
}
params_my_service2.update(my_service2_config)
return [params_my_service1, params_my_service2]
Parameters:
* **name**: Name of the attribute used to access the ``ClientsFactory`` from
the ``ServiceClients`` instance. See example below.
* **service_version**: Tempest enforces a single implementation for each
service client. Available service clients are held in a ``ClientsRegistry``
singleton, and registered with ``service_version``, which means that
``service_version`` must be unique and it should represent the service API
and version implemented by the service client.
* **module_path**: Relative to the service client module, from the root of the
plugin.
* **client_names**: Name of the classes that implement service clients in the
service clients module.
Example usage of the the service clients in tests::
# my_creds is instance of tempest.lib.auth.Credentials
# identity_uri is v2 or v3 depending on the configuration
from tempest.lib.services import clients
my_clients = clients.ServiceClients(my_creds, identity_uri)
my_service1_api1_client = my_clients.my_service_v1.API1Client()
my_service2_api1_client = my_clients.my_service_v2.API1Client(my_args='any')
Automatic configuration and registration of service clients imposes some extra
constraints on the structure of the configuration options exposed by the
plugin.
First ``service_version`` should be in the format `service_config[.version]`.
The `.version` part is optional, and should only be used if there are multiple
versions of the same API available. The `service_config` must match the name of
a configuration options group defined by the plugin. Different versions of one
API must share the same configuration group.
Second the configuration options group `service_config` must contain the
following options:
* `catalog_type`: corresponds to `service` in the catalog
* `endpoint_type`
The following options will be honoured if defined, but they are not mandatory,
as they do not necessarily apply to all service clients.
* `region`: default to identity.region
* `build_timeout` : default to compute.build_timeout
* `build_interval`: default to compute.build_interval
Third the service client classes should inherit from ``RestClient``, should
accept generic keyword arguments, and should pass those arguments to the
``__init__`` method of ``RestClient``. Extra arguments can be added. For
instance::
class MyAPIClient(rest_client.RestClient):
def __init__(self, auth_provider, service, region,
my_arg, my_arg2=True, **kwargs):
super(MyAPIClient, self).__init__(
auth_provider, service, region, **kwargs)
self.my_arg = my_arg
self.my_args2 = my_arg
Finally the service client should be structured in a python module, so that all
service client classes are importable from it. Each major API version should
have its own module.
The following folder and module structure is recommended for a single major
API version::
plugin_dir/
services/
__init__.py
client_api_1.py
client_api_2.py
The content of __init__.py module should be::
from client_api_1.py import API1Client
from client_api_2.py import API2Client
__all__ = ['API1Client', 'API2Client']
The following folder and module structure is recommended for multiple major
API version::
plugin_dir/
services/
v1/
__init__.py
client_api_1.py
client_api_2.py
v2/
__init__.py
client_api_1.py
client_api_2.py
The content each of __init__.py module under vN should be::
from client_api_1.py import API1Client
from client_api_2.py import API2Client
__all__ = ['API1Client', 'API2Client']
Using Plugins
=============

18
releasenotes/notes/clients_module-16f3025f515bf9ec.yaml Normal file
View File

@ -0,0 +1,18 @@
---
features:
- The Tempest plugin interface contains a new optional method, which allows
plugins to declare and automatically register any service client defined
in the plugin.
- tempest.lib exposes a new stable interface, the clients module and
ServiceClients class, which provides a convinient way for plugin tests to
access service clients defined in Tempest as well as service clients
defined in all loaded plugins.
The new ServiceClients class only exposes for now the service clients
which are in tempest.lib, i.e. compute, network and image. The remaing
service clients (identity, volume and object-storage) will be added in
future updates.
deprecations:
- The new clients module provides a stable alternative to tempest classes
manager.Manager and clients.Manager. manager.Manager only exists now
to smoothen the transition of plugins to the new interface, but it will
be removed shortly without further notice.

10
tempest/clients.py
View File

@ -14,15 +14,13 @@
# under the License.
import copy
from oslo_log import log as logging
from tempest.common import negative_rest_client
from tempest import config
from tempest import exceptions
from tempest.lib import auth
from tempest.lib import exceptions as lib_exc
from tempest import service_clients
from tempest.lib.services import clients
from tempest.services import baremetal
from tempest.services import data_processing
from tempest.services import identity
@ -34,7 +32,7 @@ CONF = config.CONF
LOG = logging.getLogger(__name__)
class Manager(service_clients.ServiceClients):
class Manager(clients.ServiceClients):
"""Top level manager for OpenStack tempest clients"""
default_params = config.service_client_config()
@ -109,8 +107,8 @@ class Manager(service_clients.ServiceClients):
# Setup the parameters for all Tempest services.
# NOTE(andreaf) Since client.py is an internal module of Tempest,
# it doesn't have to consider plugin configuration.
all_tempest_modules = (set(service_clients.tempest_modules()) |
service_clients._tempest_internal_modules())
all_tempest_modules = (set(clients.tempest_modules()) |
clients._tempest_internal_modules())
for service in all_tempest_modules:
try:
# NOTE(andreaf) Use the unversioned service name to fetch

415
tempest/lib/services/clients.py
View File

@ -1,3 +1,4 @@
# Copyright 2012 OpenStack Foundation
# Copyright (c) 2016 Hewlett-Packard Enterprise Development Company, L.P.
# All Rights Reserved.
#
@ -13,8 +14,92 @@
# License for the specific language governing permissions and limitations
# under the License.
import copy
import importlib
import inspect
import logging
import six
from tempest.lib import auth
from tempest.lib.common.utils import misc
from tempest.lib import exceptions
from tempest.lib.services import compute
from tempest.lib.services import image
from tempest.lib.services import network
LOG = logging.getLogger(__name__)
def tempest_modules():
"""Dict of service client modules available in Tempest.
Provides a dict of stable service modules available in Tempest, with
``service_version`` as key, and the module object as value.
"""
return {
'compute': compute,
'image.v1': image.v1,
'image.v2': image.v2,
'network': network
}
def _tempest_internal_modules():
# Set of unstable service clients available in Tempest
# NOTE(andreaf) This list will exists only as long the remain clients
# are migrated to tempest.lib, and it will then be deleted without
# deprecation or advance notice
return set(['identity.v2', 'identity.v3', 'object-storage', 'volume.v1',
'volume.v2', 'volume.v3'])
def available_modules():
"""Set of service client modules available in Tempest and plugins
Set of stable service clients from Tempest and service clients exposed
by plugins. This set of available modules can be used for automatic
configuration.
:raise PluginRegistrationException: if a plugin exposes a service_version
already defined by Tempest or another plugin.
Examples:
>>> from tempest import config
>>> params = {}
>>> for service_version in available_modules():
>>> service = service_version.split('.')[0]
>>> params[service] = config.service_client_config(service)
>>> service_clients = ServiceClients(creds, identity_uri,
>>> client_parameters=params)
"""
extra_service_versions = set([])
_tempest_modules = set(tempest_modules())
plugin_services = ClientsRegistry().get_service_clients()
for plugin_name in plugin_services:
plug_service_versions = set([x['service_version'] for x in
plugin_services[plugin_name]])
# If a plugin exposes a duplicate service_version raise an exception
if plug_service_versions:
if not plug_service_versions.isdisjoint(extra_service_versions):
detailed_error = (
'Plugin %s is trying to register a service %s already '
'claimed by another one' % (plugin_name,
extra_service_versions &
plug_service_versions))
raise exceptions.PluginRegistrationException(
name=plugin_name, detailed_error=detailed_error)
if not plug_service_versions.isdisjoint(_tempest_modules):
detailed_error = (
'Plugin %s is trying to register a service %s already '
'claimed by a Tempest one' % (plugin_name,
_tempest_modules &
plug_service_versions))
raise exceptions.PluginRegistrationException(
name=plugin_name, detailed_error=detailed_error)
extra_service_versions |= plug_service_versions
return _tempest_modules | extra_service_versions
@misc.singleton
@ -34,3 +119,333 @@ class ClientsRegistry(object):
def get_service_clients(self):
return self._service_clients
class ClientsFactory(object):
"""Builds service clients for a service client module
This class implements the logic of feeding service client parameters
to service clients from a specific module. It allows setting the
parameters once and obtaining new instances of the clients without the
need of passing any parameter.
ClientsFactory can be used directly, or consumed via the `ServiceClients`
class, which manages the authorization part.
"""
def __init__(self, module_path, client_names, auth_provider, **kwargs):
"""Initialises the client factory
:param module_path: Path to module that includes all service clients.
All service client classes must be exposed by a single module.
If they are separated in different modules, defining __all__
in the root module can help, similar to what is done by service
clients in tempest.
:param client_names: List or set of names of the service client
classes.
:param auth_provider: The auth provider used to initialise client.
:param kwargs: Parameters to be passed to all clients. Parameters
values can be overwritten when clients are initialised, but
parameters cannot be deleted.
:raise ImportError if the specified module_path cannot be imported
Example:
>>> # Get credentials and an auth_provider
>>> clients = ClientsFactory(
>>> module_path='my_service.my_service_clients',
>>> client_names=['ServiceClient1', 'ServiceClient2'],
>>> auth_provider=auth_provider,
>>> service='my_service',
>>> region='region1')
>>> my_api_client = clients.MyApiClient()
>>> my_api_client_region2 = clients.MyApiClient(region='region2')
"""
# Import the module. If it's not importable, the raised exception
# provides good enough information about what happened
_module = importlib.import_module(module_path)
# If any of the classes is not in the module we fail
for class_name in client_names:
# TODO(andreaf) This always passes all parameters to all clients.
# In future to allow clients to specify the list of parameters
# that they accept based out of a list of standard ones.
# Obtain the class
klass = self._get_class(_module, class_name)
final_kwargs = copy.copy(kwargs)
# Set the function as an attribute of the factory
setattr(self, class_name, self._get_partial_class(
klass, auth_provider, final_kwargs))
def _get_partial_class(self, klass, auth_provider, kwargs):
# Define a function that returns a new class instance by
# combining default kwargs with extra ones
def partial_class(alias=None, **later_kwargs):
"""Returns a callable the initialises a service client
Builds a callable that accepts kwargs, which are passed through
to the __init__ of the service client, along with a set of defaults
set in factory at factory __init__ time.
Original args in the service client can only be passed as kwargs.
It accepts one extra parameter 'alias' compared to the original
service client. When alias is provided, the returned callable will
also set an attribute called with a name defined in 'alias', which
contains the instance of the service client.
:param alias: str Name of the attribute set on the factory once
the callable is invoked which contains the initialised
service client. If None, no attribute is set.
:param later_kwargs: kwargs passed through to the service client
__init__ on top of defaults set at factory level.
"""
kwargs.update(later_kwargs)
_client = klass(auth_provider=auth_provider, **kwargs)
if alias:
setattr(self, alias, _client)
return _client
return partial_class
@classmethod
def _get_class(cls, module, class_name):
klass = getattr(module, class_name, None)
if not klass:
msg = 'Invalid class name, %s is not found in %s'
raise AttributeError(msg % (class_name, module))
if not inspect.isclass(klass):
msg = 'Expected a class, got %s of type %s instead'
raise TypeError(msg % (klass, type(klass)))
return klass
class ServiceClients(object):
"""Service client provider class
The ServiceClients object provides a useful means for tests to access
service clients configured for a specified set of credentials.
It hides some of the complexity from the authorization and configuration
layers.
Examples:
>>> from tempest.lib.services import clients
>>> johndoe = cred_provider.get_creds_by_role(['johndoe'])
>>> johndoe_clients = clients.ServiceClients(johndoe,
>>> identity_uri)
>>> johndoe_servers = johndoe_clients.servers_client.list_servers()
"""
# NOTE(andreaf) This class does not depend on tempest configuration
# and its meant for direct consumption by external clients such as tempest
# plugins. Tempest provides a wrapper class, `clients.Manager`, that
# initialises this class using values from tempest CONF object. The wrapper
# class should only be used by tests hosted in Tempest.
def __init__(self, credentials, identity_uri, region=None, scope='project',
disable_ssl_certificate_validation=True, ca_certs=None,
trace_requests='', client_parameters=None):
"""Service Clients provider
Instantiate a `ServiceClients` object, from a set of credentials and an
identity URI. The identity version is inferred from the credentials
object. Optionally auth scope can be provided.
A few parameters can be given a value which is applied as default
for all service clients: region, dscv, ca_certs, trace_requests.
Parameters dscv, ca_certs and trace_requests all apply to the auth
provider as well as any service clients provided by this manager.
Any other client parameter must be set via client_parameters.
The list of available parameters is defined in the service clients
interfaces. For reference, most clients will accept 'region',
'service', 'endpoint_type', 'build_timeout' and 'build_interval', which
are all inherited from RestClient.
The `config` module in Tempest exposes an helper function
`service_client_config` that can be used to extract from configuration
a dictionary ready to be injected in kwargs.
Exceptions are:
- Token clients for 'identity' have a very different interface
- Volume client for 'volume' accepts 'default_volume_size'
- Servers client from 'compute' accepts 'enable_instance_password'
Examples:
>>> identity_params = config.service_client_config('identity')
>>> params = {
>>> 'identity': identity_params,
>>> 'compute': {'region': 'region2'}}
>>> manager = lib_manager.Manager(
>>> my_creds, identity_uri, client_parameters=params)
:param credentials: An instance of `auth.Credentials`
:param identity_uri: URI of the identity API. This should be a
mandatory parameter, and it will so soon.
:param region: Default value of region for service clients.
:param scope: default scope for tokens produced by the auth provider
:param disable_ssl_certificate_validation: Applies to auth and to all
service clients.
:param ca_certs: Applies to auth and to all service clients.
:param trace_requests: Applies to auth and to all service clients.
:param client_parameters: Dictionary with parameters for service
clients. Keys of the dictionary are the service client service
name, as declared in `service_clients.available_modules()` except
for the version. Values are dictionaries of parameters that are
going to be passed to all clients in the service client module.
Examples:
>>> params_service_x = {'param_name': 'param_value'}
>>> client_parameters = { 'service_x': params_service_x }
>>> params_service_y = config.service_client_config('service_y')
>>> client_parameters['service_y'] = params_service_y
"""
self._registered_services = set([])
self.credentials = credentials
self.identity_uri = identity_uri
if not identity_uri:
raise exceptions.InvalidCredentials(
'ServiceClients requires a non-empty identity_uri.')
self.region = region
# Check if passed or default credentials are valid
if not self.credentials.is_valid():
raise exceptions.InvalidCredentials()
# Get the identity classes matching the provided credentials
# TODO(andreaf) Define a new interface in Credentials to get
# the API version from an instance
identity = [(k, auth.IDENTITY_VERSION[k][1]) for k in
auth.IDENTITY_VERSION.keys() if
isinstance(self.credentials, auth.IDENTITY_VERSION[k][0])]
# Zero matches or more than one are both not valid.
if len(identity) != 1:
raise exceptions.InvalidCredentials()
self.auth_version, auth_provider_class = identity[0]
self.dscv = disable_ssl_certificate_validation
self.ca_certs = ca_certs
self.trace_requests = trace_requests
# Creates an auth provider for the credentials
self.auth_provider = auth_provider_class(
self.credentials, self.identity_uri, scope=scope,
disable_ssl_certificate_validation=self.dscv,
ca_certs=self.ca_certs, trace_requests=self.trace_requests)
# Setup some defaults for client parameters of registered services
client_parameters = client_parameters or {}
self.parameters = {}
# Parameters are provided for unversioned services
all_modules = available_modules() | _tempest_internal_modules()
unversioned_services = set(
[x.split('.')[0] for x in all_modules])
for service in unversioned_services:
self.parameters[service] = self._setup_parameters(
client_parameters.pop(service, {}))
# Check that no client parameters was supplied for unregistered clients
if client_parameters:
raise exceptions.UnknownServiceClient(
services=list(client_parameters.keys()))
# Register service clients owned by tempest
for service, module in six.iteritems(tempest_modules()):
attribute = service.replace('.', '_')
configs = service.split('.')[0]
self.register_service_client_module(
attribute, service, module.__name__,
module.__all__, **self.parameters[configs])
# Register service clients from plugins
clients_registry = ClientsRegistry()
plugin_service_clients = clients_registry.get_service_clients()
for plugin in plugin_service_clients:
service_clients = plugin_service_clients[plugin]
# Each plugin returns a list of service client parameters
for service_client in service_clients:
# NOTE(andreaf) If a plugin cannot register, stop the
# registration process, log some details to help
# troubleshooting, and re-raise
try:
self.register_service_client_module(**service_client)
except Exception:
LOG.exception(
'Failed to register service client from plugin %s '
'with parameters %s' % (plugin, service_client))
raise
def register_service_client_module(self, name, service_version,
module_path, client_names, **kwargs):
"""Register a service client module
Initiates a client factory for the specified module, using this
class auth_provider, and accessible via a `name` attribute in the
service client.
:param name: Name used to access the client
:param service_version: Name of the service complete with version.
Used to track registered services. When a plugin implements it,
it can be used by other plugins to obtain their configuration.
:param module_path: Path to module that includes all service clients.
All service client classes must be exposed by a single module.
If they are separated in different modules, defining __all__
in the root module can help, similar to what is done by service
clients in tempest.
:param client_names: List or set of names of service client classes.
:param kwargs: Extra optional parameters to be passed to all clients.
ServiceClient provides defaults for region, dscv, ca_certs and
trace_requests.
:raise ServiceClientRegistrationException: if the provided name is
already in use or if service_version is already registered.
:raise ImportError: if module_path cannot be imported.
"""
if hasattr(self, name):
using_name = getattr(self, name)
detailed_error = 'Module name already in use: %s' % using_name
raise exceptions.ServiceClientRegistrationException(
name=name, service_version=service_version,
module_path=module_path, client_names=client_names,
detailed_error=detailed_error)
if service_version in self.registered_services:
detailed_error = 'Service %s already registered.' % service_version
raise exceptions.ServiceClientRegistrationException(
name=name, service_version=service_version,
module_path=module_path, client_names=client_names,
detailed_error=detailed_error)
params = dict(region=self.region,
disable_ssl_certificate_validation=self.dscv,
ca_certs=self.ca_certs,
trace_requests=self.trace_requests)
params.update(kwargs)
# Instantiate the client factory
_factory = ClientsFactory(module_path=module_path,
client_names=client_names,
auth_provider=self.auth_provider,
**params)
# Adds the client factory to the service_client
setattr(self, name, _factory)
# Add the name of the new service in self.SERVICES for discovery
self._registered_services.add(service_version)
@property
def registered_services(self):
return self._registered_services | _tempest_internal_modules()
def _setup_parameters(self, parameters):
"""Setup default values for client parameters
Region by default is the region passed as an __init__ parameter.
Checks that no parameter for an unknown service is provided.
"""
_parameters = {}
# Use region from __init__
if self.region:
_parameters['region'] = self.region
# Update defaults with specified parameters
_parameters.update(parameters)
# If any parameter is left, parameters for an unknown service were
# provided as input. Fail rather than ignore silently.
return _parameters

8
tempest/manager.py
View File

@ -15,15 +15,15 @@
from oslo_log import log as logging
from tempest import clients
from tempest import clients as tempest_clients
from tempest import config
from tempest import service_clients
from tempest.lib.services import clients
CONF = config.CONF
LOG = logging.getLogger(__name__)
class Manager(service_clients.ServiceClients):
class Manager(clients.ServiceClients):
"""Service client manager class for backward compatibility
The former manager.Manager is not a stable interface in Tempest,
@ -37,7 +37,7 @@ class Manager(service_clients.ServiceClients):
"soon as the client manager becomes available in tempest.lib.")
LOG.warning(msg)
dscv = CONF.identity.disable_ssl_certificate_validation
_, uri = clients.get_auth_provider_class(credentials)
_, uri = tempest_clients.get_auth_provider_class(credentials)
super(Manager, self).__init__(
credentials=credentials, scope=scope,
identity_uri=uri,

431
tempest/service_clients.py
View File

@ -1,431 +0,0 @@
# Copyright 2012 OpenStack Foundation
# Copyright (c) 2016 Hewlett-Packard Enterprise Development Company, L.P.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import copy
import importlib
import inspect
import logging
import six
from tempest.lib import auth
from tempest.lib import exceptions
from tempest.lib.services import clients
from tempest.lib.services import compute
from tempest.lib.services import image
from tempest.lib.services import network
LOG = logging.getLogger(__name__)
def tempest_modules():
"""Dict of service client modules available in Tempest.
Provides a dict of stable service modules available in Tempest, with
``service_version`` as key, and the module object as value.
"""
return {
'compute': compute,
'image.v1': image.v1,
'image.v2': image.v2,
'network': network
}
def _tempest_internal_modules():
# Set of unstable service clients available in Tempest
# NOTE(andreaf) This list will exists only as long the remain clients
# are migrated to tempest.lib, and it will then be deleted without
# deprecation or advance notice
return set(['identity.v2', 'identity.v3', 'object-storage', 'volume.v1',
'volume.v2', 'volume.v3'])
def available_modules():
"""Set of service client modules available in Tempest and plugins
Set of stable service clients from Tempest and service clients exposed
by plugins. This set of available modules can be used for automatic
configuration.
:raise PluginRegistrationException: if a plugin exposes a service_version
already defined by Tempest or another plugin.
Examples:
>>> from tempest import config
>>> params = {}
>>> for service_version in available_modules():
>>> service = service_version.split('.')[0]
>>> params[service] = config.service_client_config(service)
>>> service_clients = ServiceClients(creds, identity_uri,
>>> client_parameters=params)
"""
extra_service_versions = set([])
_tempest_modules = set(tempest_modules())
plugin_services = clients.ClientsRegistry().get_service_clients()
for plugin_name in plugin_services:
plug_service_versions = set([x['service_version'] for x in
plugin_services[plugin_name]])
# If a plugin exposes a duplicate service_version raise an exception
if plug_service_versions:
if not plug_service_versions.isdisjoint(extra_service_versions):
detailed_error = (
'Plugin %s is trying to register a service %s already '
'claimed by another one' % (plugin_name,
extra_service_versions &
plug_service_versions))
raise exceptions.PluginRegistrationException(
name=plugin_name, detailed_error=detailed_error)
if not plug_service_versions.isdisjoint(_tempest_modules):
detailed_error = (
'Plugin %s is trying to register a service %s already '
'claimed by a Tempest one' % (plugin_name,
_tempest_modules &
plug_service_versions))
raise exceptions.PluginRegistrationException(
name=plugin_name, detailed_error=detailed_error)
extra_service_versions |= plug_service_versions
return _tempest_modules | extra_service_versions
class ClientsFactory(object):
"""Builds service clients for a service client module
This class implements the logic of feeding service client parameters
to service clients from a specific module. It allows setting the
parameters once and obtaining new instances of the clients without the
need of passing any parameter.
ClientsFactory can be used directly, or consumed via the `ServiceClients`
class, which manages the authorization part.
"""
def __init__(self, module_path, client_names, auth_provider, **kwargs):
"""Initialises the client factory
:param module_path: Path to module that includes all service clients.
All service client classes must be exposed by a single module.
If they are separated in different modules, defining __all__
in the root module can help, similar to what is done by service
clients in tempest.
:param client_names: List or set of names of the service client
classes.
:param auth_provider: The auth provider used to initialise client.
:param kwargs: Parameters to be passed to all clients. Parameters
values can be overwritten when clients are initialised, but
parameters cannot be deleted.
:raise ImportError if the specified module_path cannot be imported
Example:
>>> # Get credentials and an auth_provider
>>> clients = ClientsFactory(
>>> module_path='my_service.my_service_clients',
>>> client_names=['ServiceClient1', 'ServiceClient2'],
>>> auth_provider=auth_provider,
>>> service='my_service',
>>> region='region1')
>>> my_api_client = clients.MyApiClient()
>>> my_api_client_region2 = clients.MyApiClient(region='region2')
"""
# Import the module. If it's not importable, the raised exception
# provides good enough information about what happened
_module = importlib.import_module(module_path)
# If any of the classes is not in the module we fail
for class_name in client_names:
# TODO(andreaf) This always passes all parameters to all clients.
# In future to allow clients to specify the list of parameters
# that they accept based out of a list of standard ones.
# Obtain the class
klass = self._get_class(_module, class_name)
final_kwargs = copy.copy(kwargs)
# Set the function as an attribute of the factory
setattr(self, class_name, self._get_partial_class(
klass, auth_provider, final_kwargs))
def _get_partial_class(self, klass, auth_provider, kwargs):
# Define a function that returns a new class instance by
# combining default kwargs with extra ones
def partial_class(alias=None, **later_kwargs):
"""Returns a callable the initialises a service client
Builds a callable that accepts kwargs, which are passed through
to the __init__ of the service client, along with a set of defaults
set in factory at factory __init__ time.
Original args in the service client can only be passed as kwargs.
It accepts one extra parameter 'alias' compared to the original
service client. When alias is provided, the returned callable will
also set an attribute called with a name defined in 'alias', which
contains the instance of the service client.
:param alias: str Name of the attribute set on the factory once
the callable is invoked which contains the initialised
service client. If None, no attribute is set.
:param later_kwargs: kwargs passed through to the service client
__init__ on top of defaults set at factory level.
"""
kwargs.update(later_kwargs)
_client = klass(auth_provider=auth_provider, **kwargs)
if alias:
setattr(self, alias, _client)
return _client
return partial_class
@classmethod
def _get_class(cls, module, class_name):
klass = getattr(module, class_name, None)
if not klass:
msg = 'Invalid class name, %s is not found in %s'
raise AttributeError(msg % (class_name, module))
if not inspect.isclass(klass):
msg = 'Expected a class, got %s of type %s instead'
raise TypeError(msg % (klass, type(klass)))
return klass
class ServiceClients(object):
"""Service client provider class
The ServiceClients object provides a useful means for tests to access
service clients configured for a specified set of credentials.
It hides some of the complexity from the authorization and configuration
layers.
Examples:
>>> from tempest import service_clients
>>> johndoe = cred_provider.get_creds_by_role(['johndoe'])
>>> johndoe_clients = service_clients.ServiceClients(johndoe,
>>> identity_uri)
>>> johndoe_servers = johndoe_clients.servers_client.list_servers()
"""
# NOTE(andreaf) This class does not depend on tempest configuration
# and its meant for direct consumption by external clients such as tempest
# plugins. Tempest provides a wrapper class, `clients.Manager`, that
# initialises this class using values from tempest CONF object. The wrapper
# class should only be used by tests hosted in Tempest.
def __init__(self, credentials, identity_uri, region=None, scope='project',
disable_ssl_certificate_validation=True, ca_certs=None,
trace_requests='', client_parameters=None):
"""Service Clients provider
Instantiate a `ServiceClients` object, from a set of credentials and an
identity URI. The identity version is inferred from the credentials
object. Optionally auth scope can be provided.
A few parameters can be given a value which is applied as default
for all service clients: region, dscv, ca_certs, trace_requests.
Parameters dscv, ca_certs and trace_requests all apply to the auth
provider as well as any service clients provided by this manager.
Any other client parameter must be set via client_parameters.
The list of available parameters is defined in the service clients
interfaces. For reference, most clients will accept 'region',
'service', 'endpoint_type', 'build_timeout' and 'build_interval', which
are all inherited from RestClient.
The `config` module in Tempest exposes an helper function
`service_client_config` that can be used to extract from configuration
a dictionary ready to be injected in kwargs.
Exceptions are:
- Token clients for 'identity' have a very different interface
- Volume client for 'volume' accepts 'default_volume_size'
- Servers client from 'compute' accepts 'enable_instance_password'
Examples:
>>> identity_params = config.service_client_config('identity')
>>> params = {
>>> 'identity': identity_params,
>>> 'compute': {'region': 'region2'}}
>>> manager = lib_manager.Manager(
>>> my_creds, identity_uri, client_parameters=params)
:param credentials: An instance of `auth.Credentials`
:param identity_uri: URI of the identity API. This should be a
mandatory parameter, and it will so soon.
:param region: Default value of region for service clients.
:param scope: default scope for tokens produced by the auth provider
:param disable_ssl_certificate_validation: Applies to auth and to all
service clients.
:param ca_certs: Applies to auth and to all service clients.
:param trace_requests: Applies to auth and to all service clients.
:param client_parameters: Dictionary with parameters for service
clients. Keys of the dictionary are the service client service
name, as declared in `service_clients.available_modules()` except
for the version. Values are dictionaries of parameters that are
going to be passed to all clients in the service client module.
Examples:
>>> params_service_x = {'param_name': 'param_value'}
>>> client_parameters = { 'service_x': params_service_x }
>>> params_service_y = config.service_client_config('service_y')
>>> client_parameters['service_y'] = params_service_y
"""
self._registered_services = set([])
self.credentials = credentials
self.identity_uri = identity_uri
if not identity_uri:
raise exceptions.InvalidCredentials(
'ServiceClients requires a non-empty identity_uri.')
self.region = region
# Check if passed or default credentials are valid
if not self.credentials.is_valid():
raise exceptions.InvalidCredentials()
# Get the identity classes matching the provided credentials
# TODO(andreaf) Define a new interface in Credentials to get
# the API version from an instance
identity = [(k, auth.IDENTITY_VERSION[k][1]) for k in
auth.IDENTITY_VERSION.keys() if
isinstance(self.credentials, auth.IDENTITY_VERSION[k][0])]
# Zero matches or more than one are both not valid.
if len(identity) != 1:
raise exceptions.InvalidCredentials()
self.auth_version, auth_provider_class = identity[0]
self.dscv = disable_ssl_certificate_validation
self.ca_certs = ca_certs
self.trace_requests = trace_requests
# Creates an auth provider for the credentials
self.auth_provider = auth_provider_class(
self.credentials, self.identity_uri, scope=scope,
disable_ssl_certificate_validation=self.dscv,
ca_certs=self.ca_certs, trace_requests=self.trace_requests)
# Setup some defaults for client parameters of registered services
client_parameters = client_parameters or {}
self.parameters = {}
# Parameters are provided for unversioned services
all_modules = available_modules() | _tempest_internal_modules()
unversioned_services = set(
[x.split('.')[0] for x in all_modules])
for service in unversioned_services:
self.parameters[service] = self._setup_parameters(
client_parameters.pop(service, {}))
# Check that no client parameters was supplied for unregistered clients
if client_parameters:
raise exceptions.UnknownServiceClient(
services=list(client_parameters.keys()))
# Register service clients owned by tempest
for service, module in six.iteritems(tempest_modules()):
attribute = service.replace('.', '_')
configs = service.split('.')[0]
self.register_service_client_module(
attribute, service, module.__name__,
module.__all__, **self.parameters[configs])
# Register service clients from plugins
clients_registry = clients.ClientsRegistry()
plugin_service_clients = clients_registry.get_service_clients()
for plugin in plugin_service_clients:
service_clients = plugin_service_clients[plugin]
# Each plugin returns a list of service client parameters
for service_client in service_clients:
# NOTE(andreaf) If a plugin cannot register, stop the
# registration process, log some details to help
# troubleshooting, and re-raise
try:
self.register_service_client_module(**service_client)
except Exception:
LOG.exception(
'Failed to register service client from plugin %s '
'with parameters %s' % (plugin, service_client))
raise
def register_service_client_module(self, name, service_version,
module_path, client_names, **kwargs):
"""Register a service client module
Initiates a client factory for the specified module, using this
class auth_provider, and accessible via a `name` attribute in the
service client.
:param name: Name used to access the client
:param service_version: Name of the service complete with version.
Used to track registered services. When a plugin implements it,
it can be used by other plugins to obtain their configuration.
:param module_path: Path to module that includes all service clients.
All service client classes must be exposed by a single module.
If they are separated in different modules, defining __all__
in the root module can help, similar to what is done by service
clients in tempest.
:param client_names: List or set of names of service client classes.
:param kwargs: Extra optional parameters to be passed to all clients.
ServiceClient provides defaults for region, dscv, ca_certs and
trace_requests.
:raise ServiceClientRegistrationException: if the provided name is
already in use or if service_version is already registered.
:raise ImportError: if module_path cannot be imported.
"""
if hasattr(self, name):
using_name = getattr(self, name)
detailed_error = 'Module name already in use: %s' % using_name
raise exceptions.ServiceClientRegistrationException(
name=name, service_version=service_version,
module_path=module_path, client_names=client_names,
detailed_error=detailed_error)
if service_version in self.registered_services:
detailed_error = 'Service %s already registered.' % service_version
raise exceptions.ServiceClientRegistrationException(
name=name, service_version=service_version,
module_path=module_path, client_names=client_names,
detailed_error=detailed_error)
params = dict(region=self.region,
disable_ssl_certificate_validation=self.dscv,
ca_certs=self.ca_certs,
trace_requests=self.trace_requests)
params.update(kwargs)
# Instantiate the client factory
_factory = ClientsFactory(module_path=module_path,
client_names=client_names,
auth_provider=self.auth_provider,
**params)
# Adds the client factory to the service_client
setattr(self, name, _factory)
# Add the name of the new service in self.SERVICES for discovery
self._registered_services.add(service_version)
@property
def registered_services(self):
return self._registered_services | _tempest_internal_modules()
def _setup_parameters(self, parameters):
"""Setup default values for client parameters
Region by default is the region passed as an __init__ parameter.
Checks that no parameter for an unknown service is provided.
"""
_parameters = {}
# Use region from __init__
if self.region:
_parameters['region'] = self.region
# Update defaults with specified parameters
_parameters.update(parameters)
# If any parameter is left, parameters for an unknown service were
# provided as input. Fail rather than ignore silently.
return _parameters

4
tempest/test_discover/plugins.py
View File

@ -88,7 +88,7 @@ class TempestPlugin(object):
>>> 'client_names': ['API1Client', 'API2Client'],
>>> }
>>> params.update(myservice_config)
>>> return [params]
>>> return [params]
>>> # Example implementation with two service clients
>>> foo1_config = config.service_client_config('foo')
@ -107,7 +107,7 @@ class TempestPlugin(object):
>>> 'client_names': ['API1Client', 'API2Client'],
>>> }
>>> params_foo2.update(foo2_config)
>>> return [params_foo1, params_foo2]
>>> return [params_foo1, params_foo2]
"""
return []

66
tempest/tests/test_service_clients.py → tempest/tests/lib/services/test_clients.py
View File

@ -19,7 +19,7 @@ import types
from tempest.lib import auth
from tempest.lib import exceptions
from tempest import service_clients
from tempest.lib.services import clients
from tempest.tests import base
from tempest.tests.lib import fake_auth_provider
from tempest.tests.lib import fake_credentials
@ -54,14 +54,14 @@ class TestClientsFactory(base.TestCase):
def test___init___one_class(self):
fake_partial = 'fake_partial'
partial_mock = self.useFixture(fixtures.MockPatch(
'tempest.service_clients.ClientsFactory._get_partial_class',
'tempest.lib.services.clients.ClientsFactory._get_partial_class',
return_value=fake_partial)).mock
class_names = ['FakeServiceClient1']
mock_importlib = self._setup_fake_module(class_names=class_names)
auth_provider = fake_auth_provider.FakeAuthProvider()
params = {'k1': 'v1', 'k2': 'v2'}
factory = service_clients.ClientsFactory('fake_path', class_names,
auth_provider, **params)
factory = clients.ClientsFactory('fake_path', class_names,
auth_provider, **params)
# Assert module has been imported
mock_importlib.assert_called_once_with('fake_path')
# All attributes have been created
@ -77,14 +77,14 @@ class TestClientsFactory(base.TestCase):
def test___init___two_classes(self):
fake_partial = 'fake_partial'
partial_mock = self.useFixture(fixtures.MockPatch(
'tempest.service_clients.ClientsFactory._get_partial_class',
'tempest.lib.services.clients.ClientsFactory._get_partial_class',
return_value=fake_partial)).mock
class_names = ['FakeServiceClient1', 'FakeServiceClient2']
mock_importlib = self._setup_fake_module(class_names=class_names)
auth_provider = fake_auth_provider.FakeAuthProvider()
params = {'k1': 'v1', 'k2': 'v2'}
factory = service_clients.ClientsFactory('fake_path', class_names,
auth_provider, **params)
factory = clients.ClientsFactory('fake_path', class_names,
auth_provider, **params)
# Assert module has been imported
mock_importlib.assert_called_once_with('fake_path')
# All attributes have been created
@ -100,8 +100,8 @@ class TestClientsFactory(base.TestCase):
auth_provider = fake_auth_provider.FakeAuthProvider()
class_names = ['FakeServiceClient1', 'FakeServiceClient2']
with testtools.ExpectedException(ImportError, '.*fake_module.*'):
service_clients.ClientsFactory('fake_module', class_names,
auth_provider)
clients.ClientsFactory('fake_module', class_names,
auth_provider)
def test___init___not_a_class(self):
class_names = ['FakeServiceClient1', 'FakeServiceClient2']
@ -111,8 +111,8 @@ class TestClientsFactory(base.TestCase):
auth_provider = fake_auth_provider.FakeAuthProvider()
expected_msg = '.*not_really_a_class.*str.*'
with testtools.ExpectedException(TypeError, expected_msg):
service_clients.ClientsFactory('fake_module', extended_class_names,
auth_provider)
clients.ClientsFactory('fake_module', extended_class_names,
auth_provider)
def test___init___class_not_found(self):
class_names = ['FakeServiceClient1', 'FakeServiceClient2']
@ -121,15 +121,15 @@ class TestClientsFactory(base.TestCase):
auth_provider = fake_auth_provider.FakeAuthProvider()
expected_msg = '.*not_really_a_class.*fake_service_client.*'
with testtools.ExpectedException(AttributeError, expected_msg):
service_clients.ClientsFactory('fake_module', extended_class_names,
auth_provider)
clients.ClientsFactory('fake_module', extended_class_names,
auth_provider)
def test__get_partial_class_no_later_kwargs(self):
expected_fake_client = 'not_really_a_client'
self._setup_fake_module(class_names=[])
auth_provider = fake_auth_provider.FakeAuthProvider()
params = {'k1': 'v1', 'k2': 'v2'}
factory = service_clients.ClientsFactory(
factory = clients.ClientsFactory(
'fake_path', [], auth_provider, **params)
klass_mock = mock.Mock(return_value=expected_fake_client)
partial = factory._get_partial_class(klass_mock, auth_provider, params)
@ -147,7 +147,7 @@ class TestClientsFactory(base.TestCase):
auth_provider = fake_auth_provider.FakeAuthProvider()
params = {'k1': 'v1', 'k2': 'v2'}
later_params = {'k2': 'v4', 'k3': 'v3'}
factory = service_clients.ClientsFactory(
factory = clients.ClientsFactory(
'fake_path', [], auth_provider, **params)
klass_mock = mock.Mock(return_value=expected_fake_client)
partial = factory._get_partial_class(klass_mock, auth_provider, params)
@ -167,7 +167,7 @@ class TestClientsFactory(base.TestCase):
auth_provider = fake_auth_provider.FakeAuthProvider()
params = {'k1': 'v1', 'k2': 'v2'}
later_params = {'k2': 'v4', 'k3': 'v3'}
factory = service_clients.ClientsFactory(
factory = clients.ClientsFactory(
'fake_path', [], auth_provider, **params)
klass_mock = mock.Mock(return_value=expected_fake_client)
partial = factory._get_partial_class(klass_mock, auth_provider, params)
@ -188,9 +188,9 @@ class TestServiceClients(base.TestCase):
def setUp(self):
super(TestServiceClients, self).setUp()
self.useFixture(fixtures.MockPatch(
'tempest.service_clients.tempest_modules', return_value={}))
'tempest.lib.services.clients.tempest_modules', return_value={}))
self.useFixture(fixtures.MockPatch(
'tempest.service_clients._tempest_internal_modules',
'tempest.lib.services.clients._tempest_internal_modules',
return_value=set(['fake_service1'])))
def test___init___creds_v2_uri(self):
@ -198,7 +198,7 @@ class TestServiceClients(base.TestCase):
# is required to run the test successfully
creds = fake_credentials.FakeKeystoneV2Credentials()
uri = 'fake_uri'
_manager = service_clients.ServiceClients(creds, identity_uri=uri)
_manager = clients.ServiceClients(creds, identity_uri=uri)
self.assertIsInstance(_manager.auth_provider,
auth.KeystoneV2AuthProvider)
@ -207,7 +207,7 @@ class TestServiceClients(base.TestCase):
# is required to run the test successfully
creds = fake_credentials.FakeKeystoneV3Credentials()
uri = 'fake_uri'
_manager = service_clients.ServiceClients(creds, identity_uri=uri)
_manager = clients.ServiceClients(creds, identity_uri=uri)
self.assertIsInstance(_manager.auth_provider,
auth.KeystoneV3AuthProvider)
@ -215,14 +215,14 @@ class TestServiceClients(base.TestCase):
creds = fake_credentials.FakeCredentials()
uri = 'fake_uri'
with testtools.ExpectedException(exceptions.InvalidCredentials):
service_clients.ServiceClients(creds, identity_uri=uri)
clients.ServiceClients(creds, identity_uri=uri)
def test___init___invalid_creds_uri(self):
creds = fake_credentials.FakeKeystoneV2Credentials()
delattr(creds, 'username')
uri = 'fake_uri'
with testtools.ExpectedException(exceptions.InvalidCredentials):
service_clients.ServiceClients(creds, identity_uri=uri)
clients.ServiceClients(creds, identity_uri=uri)
def test___init___creds_uri_none(self):
creds = fake_credentials.FakeKeystoneV2Credentials()
@ -230,7 +230,7 @@ class TestServiceClients(base.TestCase):
"non-empty")
with testtools.ExpectedException(exceptions.InvalidCredentials,
value_re=msg):
service_clients.ServiceClients(creds, None)
clients.ServiceClients(creds, None)
def test___init___creds_uri_params(self):
creds = fake_credentials.FakeKeystoneV2Credentials()
@ -238,8 +238,8 @@ class TestServiceClients(base.TestCase):
'fake_param2': 'fake_value2'}
params = {'fake_service1': expeted_params}
uri = 'fake_uri'
_manager = service_clients.ServiceClients(creds, identity_uri=uri,
client_parameters=params)
_manager = clients.ServiceClients(creds, identity_uri=uri,
client_parameters=params)
self.assertIn('fake_service1', _manager.parameters)
for _key in expeted_params:
self.assertIn(_key, _manager.parameters['fake_service1'].keys())
@ -255,14 +255,14 @@ class TestServiceClients(base.TestCase):
msg = "(?=.*{0})(?=.*{1})".format(*list(params.keys()))
with testtools.ExpectedException(
exceptions.UnknownServiceClient, value_re=msg):
service_clients.ServiceClients(creds, identity_uri=uri,
client_parameters=params)
clients.ServiceClients(creds, identity_uri=uri,
client_parameters=params)
def _get_manager(self, init_region='fake_region'):
# Get a manager to invoke _setup_parameters on
creds = fake_credentials.FakeKeystoneV2Credentials()
return service_clients.ServiceClients(creds, identity_uri='fake_uri',
region=init_region)
return clients.ServiceClients(creds, identity_uri='fake_uri',
region=init_region)
def test__setup_parameters_none_no_region(self):
kwargs = {}
@ -294,7 +294,7 @@ class TestServiceClients(base.TestCase):
_manager = self._get_manager(init_region='fake_region_default')
# Mock after the _manager is setup to preserve the call count
factory_mock = self.useFixture(fixtures.MockPatch(
'tempest.service_clients.ClientsFactory')).mock
'tempest.lib.services.clients.ClientsFactory')).mock
_manager.register_service_client_module(
name='fake_module',
service_version='fake_service',
@ -323,7 +323,7 @@ class TestServiceClients(base.TestCase):
_manager = self._get_manager(init_region='fake_region_default')
# Mock after the _manager is setup to preserve the call count
factory_mock = self.useFixture(fixtures.MockPatch(
'tempest.service_clients.ClientsFactory')).mock
'tempest.lib.services.clients.ClientsFactory')).mock
_manager.register_service_client_module(
name='fake_module',
service_version='fake_service',
@ -346,7 +346,7 @@ class TestServiceClients(base.TestCase):
def test_register_service_client_module_duplicate_name(self):
self.useFixture(fixtures.MockPatch(
'tempest.service_clients.ClientsFactory'))
'tempest.lib.services.clients.ClientsFactory')).mock
_manager = self._get_manager()
name_owner = 'this_is_a_string'
setattr(_manager, 'fake_module', name_owner)
@ -359,7 +359,7 @@ class TestServiceClients(base.TestCase):
def test_register_service_client_module_duplicate_service(self):
self.useFixture(fixtures.MockPatch(
'tempest.service_clients.ClientsFactory'))
'tempest.lib.services.clients.ClientsFactory')).mock
_manager = self._get_manager()
duplicate_service = 'fake_service1'
expected_error = '.*' + duplicate_service