955 lines
30 KiB
ReStructuredText
955 lines
30 KiB
ReStructuredText
..
|
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
|
License.
|
|
|
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
|
|
|
..
|
|
This template should be in ReSTructured text. The filename in the git
|
|
repository should match the launchpad URL, for example a URL of
|
|
https://blueprints.launchpad.net/rally/+spec/awesome-thing should be named
|
|
awesome-thing.rst . Please do not delete any of the sections in this
|
|
template. If you have nothing to say for a whole section, just write: None
|
|
For help with syntax, see http://www.sphinx-doc.org/en/stable/rest.html
|
|
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
|
|
|
|
===============================
|
|
Refactor Verification Component
|
|
===============================
|
|
|
|
Rally Verification was introduced long time ago as an easy way to launch
|
|
Tempest. It allows to manage(install, uninstall, configure and etc),
|
|
launch Tempest and process the results(store, compare, displaying in different
|
|
formats).
|
|
|
|
There is a lot of code related to Verification which can be used not only for
|
|
Tempest. Since `rally verify` was implemented to launch subunit-based
|
|
applications(Tempest is a such tool), our code is ready to launch whatever we
|
|
want subunit-frameworks by changing only one var - path to tests.
|
|
|
|
Problem description
|
|
===================
|
|
|
|
Rally is a good framework for any kind of testing (performance, functional and
|
|
etc), so it is pretty sad when we have a lot of hardcode and binding to
|
|
specific application.
|
|
|
|
* non-pluggable architecture
|
|
|
|
Most of Rally components (for example Task or Deployment) are pluggable. You
|
|
can easily extend Rally framework for such components. But we cannot say the
|
|
same about Verification.
|
|
|
|
* subunit-trace
|
|
|
|
``subunit-trace`` library is used to display the live progress and summary at
|
|
user-friendly way for each launch of Verification.
|
|
There are several issues across this library:
|
|
|
|
1. It is Tempest requirements.
|
|
|
|
It is second time when Rally Verification component uses dependency
|
|
from Tempest. ``tools/colorizer.py`` was used from Tempest repo
|
|
before ``subunit-trace``. This script was removed from Tempest which led
|
|
to breakage of whole Verification stuff.
|
|
Also, ``rally verify install`` supports ``--source`` option for installing
|
|
Tempest from non-default repos which can miss ``subunit-trace``
|
|
requirement.
|
|
|
|
2. Bad calculation(for example, skip of whole TestCase means 1 skipped test)
|
|
|
|
* Code duplication
|
|
|
|
To simplify usage of Tempest, it is required to check existence of images,
|
|
roles, networks and other resources. While implementing these checks, we
|
|
re-implemented ... "Context" class which is used in Tasks.
|
|
It was called TempestResourcesContext.
|
|
|
|
* Inner storage based on deployment
|
|
|
|
In case of several deployments and one type of verifier(one repo), Rally
|
|
creates several directories in ``~/.rally/tempest`` (``for-tempest-<uuid>``
|
|
where <uuid> is a UUID of deployment). Each of these directories will
|
|
include same files. The difference only in config files which can be stored
|
|
wherever we want.
|
|
Also, we have one more directory with the same data - cache directory
|
|
(``~/.rally/tempest/base``).
|
|
|
|
* Word "Tempest" hardcoded in logging, help messages, etc.
|
|
|
|
Proposed change
|
|
===============
|
|
|
|
Most of subunit-based frameworks can be launched in the same way, but they can
|
|
accept different arguments, different setup steps and so on.
|
|
|
|
.. note:: In further text, we will apply labels "old" for code which was
|
|
implemented before this spec and "new" for proposed change. Also, all
|
|
references for old code will be linked to `0.3.3`__ release which is latest
|
|
release at the time of writing this spec.
|
|
|
|
__ http://rally.readthedocs.org/en/0.3.3/release_notes/archive/v0.3.3.html
|
|
|
|
Declare base Verification entities
|
|
----------------------------------
|
|
|
|
Lets talk about all entities which represents Verification.
|
|
|
|
Old model
|
|
~~~~~~~~~
|
|
|
|
Old implementation uses only one entity - results of a single verification
|
|
launch.
|
|
|
|
**DB Layer**
|
|
|
|
* `Verification`__
|
|
|
|
It represents a summary of a single verification launch results. Also, it
|
|
is linked to full results (see next entity - VerificationResult).
|
|
|
|
__ https://github.com/openstack/rally/blob/0.3.3/rally/common/db/sqlalchemy/models.py#L186
|
|
|
|
* `VerificationResult`__
|
|
|
|
The full results of a single launch.
|
|
|
|
Since support of migrations was added
|
|
recently, not all places are cleared yet, so ``VerificationResults`` can
|
|
store results in two formats(old and current format). It would be nice to
|
|
fix it and support only 1 format.
|
|
|
|
__ https://github.com/openstack/rally/blob/0.3.3/rally/common/db/sqlalchemy/models.py#L217
|
|
|
|
**Object layer**
|
|
|
|
It is a bad practise to provide an access to db stuff directly and we don't do
|
|
that. ``rally.common.objects`` layer was designed to hide all db related stuff.
|
|
|
|
* `Verification`__
|
|
|
|
Just represents results.
|
|
|
|
__ https://github.com/openstack/rally/blob/0.3.3/rally/common/objects/verification.py#L28
|
|
|
|
New model
|
|
~~~~~~~~~
|
|
|
|
We want to support different verifiers and want to identify them, so let's
|
|
declare three entities:
|
|
|
|
* **Verifier type**. The name of entity is a description it self. Each type
|
|
should be represented by own plugin which implements interface for
|
|
verification tool. For example, Tempest, Gabbi should be such types.
|
|
|
|
* **Verifier**. An instance of ``verifier type``. I can be described with
|
|
following options:
|
|
|
|
* *source* - path to git repository of tool.
|
|
|
|
* *system-wide* - whether or not to use the local env instead of virtual
|
|
environment when installing verifier.
|
|
|
|
* *version* - branch, tag or hash of commit to install verifier from. By
|
|
default it is "master" branch.
|
|
|
|
* **Verification Results**. Result of a single launch.
|
|
|
|
|
|
**DB Layer**
|
|
|
|
* **Verifier**. We should add one more table to store different verifiers. New
|
|
migration should be added, which check existence verification launches and
|
|
create "default" verifier(type="Tempest", source="n/a") and map all of
|
|
launches to it.
|
|
|
|
.. code-block::
|
|
|
|
class Verifier(BASE, RallyBase):
|
|
"""Represent a unique verifier."""
|
|
|
|
__tablename__ = "verifiers"
|
|
__table_args__ = (
|
|
sa.Index("verification_uuid", "uuid", unique=True),
|
|
)
|
|
|
|
id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
|
|
uuid = sa.Column(sa.String(36), default=UUID, nullable=False)
|
|
|
|
deployment_uuid = sa.Column(
|
|
sa.String(36),
|
|
sa.ForeignKey(Deployment.uuid),
|
|
nullable=False,
|
|
)
|
|
|
|
name = sa.Column(sa.String(255), unique=True)
|
|
description = sa.Column(sa.String(1000))
|
|
|
|
status = sa.Column(sa.Enum(*list(consts.VerifierStatus),
|
|
name="enum_verifier_status"),
|
|
default=consts.VerifierStatus.INIT,
|
|
nullable=False)
|
|
started_at = sa.Column(sa.DateTime)
|
|
updated_at = sa.Column(sa.DateTime)
|
|
|
|
type = sa.Column(sa.String(255), nullable=False)
|
|
settings = info = sa.Column(
|
|
sa_types.MutableJSONEncodedDict,
|
|
default={"system-wide": False,
|
|
"source": "n/a"},
|
|
nullable=False,
|
|
)
|
|
|
|
* `Verification`__
|
|
|
|
It should be extended with a link to Verifier.
|
|
|
|
* `VerificationResult`__
|
|
|
|
We can leave it as it is.
|
|
|
|
|
|
Move storage from deployment depended logic to verifier
|
|
-------------------------------------------------------
|
|
|
|
Old structure of ``~/.rally/tempest`` dir:
|
|
|
|
.. code-block:: yaml
|
|
|
|
base:
|
|
tempest_base-<hash>:
|
|
# Cached Tempest repository
|
|
tempest:
|
|
api
|
|
api_schema
|
|
cmd
|
|
...
|
|
...
|
|
requirements.txt
|
|
setup.cfg
|
|
setup.py
|
|
...
|
|
for-deployment-<uuid>:
|
|
# copy-paste of tempest_base-<hash> + files and directories listed below
|
|
.venv # Directory for virtual environment: exists if user didn't
|
|
# specify ``--system-wide`` argument while tempest
|
|
# installation (``rally verify install`` command).
|
|
tempest.conf # Only this file is unique for each deployment. It stores
|
|
# Tempest configuration.
|
|
subunit.stream # Temporary result-file produced by ``rally verify start``.
|
|
|
|
As you can see there are a lot of copy-pasted repositories and little unique
|
|
data.
|
|
|
|
New structure(should be located in ``~/.rally/verifiers``):
|
|
|
|
.. code-block:: yaml
|
|
|
|
verifier-<uuid>:
|
|
# Storage for unique verifier. <uuid> is a uuid of verifier.
|
|
repo:
|
|
# Verifier code repository. It is same for all deployments. Also one
|
|
# virtual environment can be used across all deployment too.
|
|
...
|
|
for-deployment-<uuid>:
|
|
# Folder to store unique for deployment data. <uuid> is a deployment uuid
|
|
# here. Currently we have only configuration file to store, but lets
|
|
# reserve place to store more data.
|
|
settings.conf
|
|
...
|
|
|
|
Each registered verifier is a unique entity for Rally and can be used by all
|
|
deployments. If there is deployment specific data(for example, configuration
|
|
file) required for verifier, it should be stored separately from verifier.
|
|
|
|
Command line interface
|
|
----------------------
|
|
|
|
`rally verify` commands are not so hardcoded as other parts of Verification
|
|
component, but in the same time they are not flexible.
|
|
|
|
Old commands:
|
|
|
|
.. code-block:: none
|
|
|
|
compare Compare two verification results.
|
|
detailed Display results table of a verification with detailed errors.
|
|
discover Show a list of discovered tests.
|
|
genconfig Generate Tempest configuration file.
|
|
import Import Tempest tests results into the Rally database.
|
|
install Install Tempest.
|
|
list List verification runs.
|
|
reinstall Uninstall Tempest and install again.
|
|
results Display results of a verification.
|
|
show Display results table of a verification.
|
|
showconfig Show configuration file of Tempest.
|
|
start Start verification (run Tempest tests).
|
|
uninstall Remove the deployment's local Tempest installation.
|
|
use Set active verification.
|
|
|
|
There is another problem of old CLI. Management is split across all commands
|
|
and you can do the same things via different commands. Moreover, you can
|
|
install Tempest in virtual environment via ``rally verify install`` and use
|
|
``--system-wide`` option in ``rally verify start``.
|
|
|
|
Lets provide more strict CLI. Something like:
|
|
|
|
.. code-block:: none
|
|
|
|
list-types
|
|
|
|
create-verifier
|
|
delete-verifier
|
|
list-verifiers
|
|
update-verifier
|
|
extend-verifier
|
|
use-verifier
|
|
|
|
configure
|
|
discover
|
|
start
|
|
|
|
compare
|
|
export
|
|
import
|
|
list
|
|
show
|
|
use
|
|
|
|
list-types
|
|
~~~~~~~~~~
|
|
|
|
Verifiers types should be implemented on base Rally plugin mechanism. It allow
|
|
to not create types manually, Rally will automatically load them and user will
|
|
need only interface to list them.
|
|
|
|
create-verifier
|
|
~~~~~~~~~~~~~~~
|
|
|
|
Just creates a new verifier based on type.
|
|
|
|
Example:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ rally verify create-verifier tempest-mitaka --type tempest --source "https://git.openstack.org/openstack/tempest" --version "10.0.0" --system-wide
|
|
|
|
This command should process next steps:
|
|
|
|
1. Clone Tempest repository from "https://git.openstack.org/openstack/tempest";
|
|
2. Call ``git checkout 10.0.0``;
|
|
3. Check that all requirements from requirements.txt are satisfied;
|
|
4. Put new verifier as default one
|
|
|
|
Also, it would be nice to store verifier statuses like "Init", "Ready-to-use",
|
|
"Failed", "Updating".
|
|
|
|
delete-verifier
|
|
~~~~~~~~~~~~~~~
|
|
|
|
Deletes verifier virtual environment(if it was created), repository, deployment
|
|
specific files(configuration files).
|
|
|
|
Also, it will remove verification results produced by this verifier.
|
|
|
|
list-verifiers
|
|
~~~~~~~~~~~~~~
|
|
|
|
List all available verifiers.
|
|
|
|
update-verifier
|
|
~~~~~~~~~~~~~~~
|
|
|
|
This command gives ability to update git repository(``git pull`` or
|
|
``git checkout``) or start/stop using virtual environment.
|
|
|
|
Also, configuration file can be update via this interface.
|
|
|
|
extend-verifier
|
|
~~~~~~~~~~~~~~~
|
|
|
|
Verifier can have an interface to extend itself. For example, Tempest supports
|
|
plugins. For verifiers which do not support any extend-mechanism, lets print
|
|
user-friendly message.
|
|
|
|
use-verifier
|
|
~~~~~~~~~~~~
|
|
|
|
Choose the default verifier.
|
|
|
|
configure
|
|
~~~~~~~~~
|
|
|
|
An interface to configure verifier for an specific deployment.
|
|
|
|
Usage examples:
|
|
|
|
.. code-block:: bash
|
|
|
|
# At this step we assume that configuration file was not created yet.
|
|
# Create configuration file and show it.
|
|
$ rally verify configure
|
|
|
|
# Configuration file already exists, so just show it.
|
|
$ rally verify configure
|
|
|
|
# Recreate configuration file and show it
|
|
$ rally verify configure --renew
|
|
|
|
# Recreate configuration file using predefined configuration options and
|
|
# show it.
|
|
# via json:
|
|
$ rally verify configure --renew \
|
|
> --options '{"section_name": {"some_key": "some_var"}}'
|
|
|
|
# via config file, which can be json/yaml or ini format:
|
|
$ rally verify configure --renew --options ~/some_file.conf
|
|
|
|
# Replace configuration file by another file and show it
|
|
$ rally verify configure --replace ./some_config.conf
|
|
|
|
Also, we can provide ``--silent`` option to disable ``show`` action.
|
|
|
|
discover
|
|
~~~~~~~~
|
|
|
|
Discover and list tests.
|
|
|
|
start
|
|
~~~~~
|
|
|
|
Start verification. Basically, there is no big difference between launching
|
|
different verifiers.
|
|
|
|
Current arguments: ``--set``, ``--regex``, ``--tests-file``, ``xfails-file``,
|
|
``--failing``.
|
|
|
|
Argument ``--set`` is specific for Tempest. Each verifier can have specific
|
|
search arguments. Lets introduce new argument ``--filter-by``. In this case,
|
|
set_name for Tempest can be specified like ``--filter-by set=smoke``.
|
|
|
|
compare
|
|
~~~~~~~
|
|
|
|
Compare two verification results.
|
|
|
|
export
|
|
~~~~~~
|
|
|
|
Part of `Export task and verifications into external services`__ spec
|
|
|
|
__ https://github.com/openstack/rally/blob/0.3.2/doc/specs/in-progress/task_and_verification_export.rst
|
|
|
|
import
|
|
~~~~~~
|
|
|
|
Import outer results in Rally database.
|
|
|
|
list
|
|
~~~~
|
|
|
|
List all verifications results.
|
|
|
|
show
|
|
~~~~
|
|
|
|
Show verification results in different formats.
|
|
|
|
Refactor base classes
|
|
---------------------
|
|
|
|
Old implementation includes several classes:
|
|
|
|
* Main class **Tempest**. This class combines manage and launch logic.
|
|
|
|
.. code-block:: python
|
|
|
|
# Description of a public interface(all implementation details are skipped)
|
|
class Tempest(object):
|
|
|
|
base_repo_dir = os.path.join(os.path.expanduser("~"),
|
|
".rally/tempest/base")
|
|
|
|
def __init__(self, deployment, verification=None,
|
|
tempest_config=None, source=None, system_wide=False):
|
|
pass
|
|
|
|
@property
|
|
def venv_wrapper(self):
|
|
"""This property returns the command for activation virtual
|
|
environment. It is hardcoded on tool from Tempest repository:
|
|
|
|
https://github.com/openstack/tempest/blob/10.0.0/tools/with_venv.sh
|
|
|
|
We should remove this hardcode in new implementation."""
|
|
|
|
@property
|
|
def env(self):
|
|
"""Returns a copy of environment variables with addition of pathes
|
|
to tests"""
|
|
|
|
def path(self, *inner_path):
|
|
"""Constructs a path for inner files of
|
|
~/.rally/tempest/for-deployment-<uuid>
|
|
"""
|
|
|
|
@property
|
|
def base_repo(self):
|
|
"""The structure of ~/.rally/tempest dir was changed several times.
|
|
This method handles the difference."""
|
|
|
|
def is_configured(self):
|
|
pass
|
|
|
|
def generate_config_file(self, override=False):
|
|
"""Generate configuration file of Tempest for current deployment.
|
|
:param override: Whether or not to override existing Tempest
|
|
config file
|
|
"""
|
|
|
|
def is_installed(self):
|
|
pass
|
|
|
|
def install(self):
|
|
"""Creates local Tempest repo and virtualenv for deployment."""
|
|
|
|
def uninstall(self):
|
|
"""Removes local Tempest repo and virtualenv for deployment."""
|
|
|
|
def run(self, testr_args="", log_file=None, tempest_conf=None):
|
|
"""Run Tempest."""
|
|
|
|
def discover_tests(self, pattern=""):
|
|
"""Get a list of discovered tests.
|
|
:param pattern: Test name pattern which can be used to match
|
|
"""
|
|
|
|
def parse_results(self, log_file=None, expected_failures=None):
|
|
"""Parse subunit raw log file."""
|
|
|
|
def verify(self, set_name, regex, tests_file, expected_failures,
|
|
concur, failing):
|
|
"""Launch verification and save results in database."""
|
|
|
|
def import_results(self, set_name, log_file):
|
|
"""Import outer subunit-file to Rally database"""
|
|
|
|
def install_plugins(self, *args, **kwargs):
|
|
"""Install Tempest plugin."""
|
|
|
|
* class ``TempestConfig`` was designed to obtain all required settings from
|
|
OpenStack public API and generate configuration file. It has not-bad
|
|
interface (just ``init`` and ``generate`` public methods), but implementation
|
|
can be better(init method should not start obtaining data).
|
|
|
|
* class ``TempestResourcesContext`` looks like context which we have for Task
|
|
component.
|
|
|
|
``TempestConfig`` and ``TempestResourcesContext`` are help classes and in new
|
|
implementation they will be optional.
|
|
|
|
New implementation should looks like:
|
|
|
|
* ``VerifierManager``. It is a main class which represents a type of Verifier
|
|
and provide an interface for all management stuff(i.e. install, update,
|
|
delete). Also, it should be an entry-point for configuration and
|
|
extend-mechanism which are optional.
|
|
|
|
* ``VerifierLauncher``. It takes care about deployment's task - preparation
|
|
and launching verification and so on.
|
|
|
|
* ``VerifierContext``. The inheritor of rally.task.context.Context class with
|
|
hardcoded "hidden=True" value, since it should be inner helper class.
|
|
|
|
* ``VerifierSettings``. Obtains required data from public APIs and constructs
|
|
deployment specific configuration files for Verifiers.
|
|
|
|
Proposed implementation will be described below in `Implementation`_ section.
|
|
|
|
Remove dependency from external libraries and scripts
|
|
-----------------------------------------------------
|
|
|
|
Currently our verification code has two redundant dependencies:
|
|
|
|
* subunit-trace
|
|
* <tempest repo>/tools/with_venv.sh
|
|
|
|
subunit-trace
|
|
~~~~~~~~~~~~~
|
|
|
|
It should not be a hard task to remove this dependency. With small
|
|
modifications ``rally.common.io.subunit.SubunitV2StreamResult`` can print live
|
|
progress. Also, we an print summary info based on parsed results.
|
|
|
|
with_venv.sh script
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
It is tempest in-tree script. Its logic is too simple - just activate virtual
|
|
environment and execute transmitted cmd in it. I suppose that we can rewrite
|
|
this script in python and put it to Verification component.
|
|
|
|
Alternatives
|
|
------------
|
|
|
|
Stop development of Rally Verification.
|
|
|
|
Implementation
|
|
==============
|
|
|
|
Implementation details
|
|
----------------------
|
|
|
|
Below you can find an example of implementation. It contains some
|
|
implementation details and notes for future development.
|
|
|
|
.. note:: Proposed implementation is not ideal and not finished. It should be
|
|
reviewed without nits.
|
|
|
|
rally.common.objects.Verifier
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Basically, it will be the same design as `rally.common.objects.Verification`__.
|
|
There is no reasons to store old class. ``Verifier`` interface should be
|
|
enough.
|
|
|
|
__ https://github.com/openstack/rally/blob/0.3.3/rally/common/objects/verification.py#L28
|
|
|
|
VerifierManager
|
|
~~~~~~~~~~~~~~~
|
|
|
|
.. code-block:: python
|
|
|
|
import os
|
|
import shutil
|
|
import subprocess
|
|
|
|
from rally.common.plugin import plugin
|
|
|
|
|
|
class VerifierManager(plugin.Plugin):
|
|
|
|
def __init__(self, verifier):
|
|
"""Init manager
|
|
|
|
:param verifier: `rally.common.objects.Verifier` instance
|
|
"""
|
|
self.verifier = self.verifier
|
|
|
|
@property
|
|
def home_dir(self):
|
|
"""Home directory of verifier"""
|
|
return "~/.rally/verifier-%s" % self.verifier.id
|
|
|
|
@property
|
|
def repo_path(self):
|
|
"""Path to local repository"""
|
|
return os.path.join(self.home_dir, "repo")
|
|
|
|
def mkdirs(self):
|
|
"""Create all directories"""
|
|
if not self.home_dir:
|
|
os.mkdir(self.home_dir)
|
|
deployment_path = os.path.join(
|
|
base_path, "for-deployment-%s" % self.deployment.id))
|
|
if not deployment_path:
|
|
os.mkdir(deployment_path)
|
|
|
|
def _clone(self):
|
|
"""Clone and checkout git repo"""
|
|
self.mkdirs()
|
|
source = self.verifier.source or self._meta_get("default_repo")
|
|
subprocess.check_call(["git", "clone", source, self.repo_path])
|
|
|
|
version = self.verifier.version or self._meta_get("default_version")
|
|
if version:
|
|
subprocess.check_call(["git", "checkout", version],
|
|
cwd=self.repo_path)
|
|
|
|
def _install_virtual_env(self):
|
|
"""Install virtual environment and all requirement in it."""
|
|
if os.path.exists(os.path.join(self.repo_path, ".venv")):
|
|
# NOTE(andreykurilin): It is necessary to remove old env while
|
|
# processing update action
|
|
shutils.rmtree(os.path.join(self.repo_path, ".venv"))
|
|
|
|
# TODO(andreykurilin): make next steps silent and print output only
|
|
# on failure or debug
|
|
subprocess.check_output(["virtualenv", ".venv"], cwd=self.repo_path)
|
|
# TODO: install verifier and its requirements here.
|
|
|
|
def install(self):
|
|
if os.path.exists(self.home_dir):
|
|
# raise a proper exception
|
|
raise Exception()
|
|
self._clone()
|
|
if system_wide:
|
|
# There are several ways to check requirements. It can be done
|
|
# at least via two libraries: `pip`, `pkgutils`. The code below
|
|
# bases on `pip`, but it can be changed for better solution while
|
|
# implementation.
|
|
import pip
|
|
|
|
requirements = set(pip.req.parse_requirements(
|
|
"%s/requirements.txt" % self.repo_path,
|
|
session=False))
|
|
installed_packages = set(pip.get_installed_distributions())
|
|
missed_packages = requirements - installed_packages
|
|
if missed_packages:
|
|
# raise a proper exception
|
|
raise Exception()
|
|
else:
|
|
self._install_virtual_env()
|
|
|
|
|
|
def delete(self):
|
|
"""Remove all"""
|
|
shutils.rmtree(self.home_dir)
|
|
|
|
def update(self, update_repo=False, version=None, update_venv=False):
|
|
"""Update repository, version, virtual environment."""
|
|
pass
|
|
|
|
def extend(self, *args, **kwargs):
|
|
"""Install verifier extensions.
|
|
|
|
.. note:: It is an optional interface, so it raises UnsupportedError
|
|
by-default. If specific verifier needs this interface, it should
|
|
just implement it.
|
|
"""
|
|
raise UnsupportedAction("%s verifier is not support extensions." %
|
|
self.get_name())
|
|
|
|
For example, the implementation of verifier for Tempest will need to
|
|
implement only one method ``extend``:
|
|
|
|
.. code-block:: python
|
|
|
|
@configure("tempest_manager",
|
|
default_repo="https://github.com/openstack/tempest",
|
|
default_version="master",
|
|
launcher="tempest_launcher")
|
|
class TempestManager(VerifierManager):
|
|
|
|
def extend(self, *args, **kwargs):
|
|
"""Install tempest-plugin."""
|
|
pass
|
|
|
|
VerifierLauncher
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
.. code-block:: python
|
|
|
|
import os
|
|
import subprocess
|
|
|
|
from rally.common.io import subunit_v2
|
|
from rally.common.plugin import plugin
|
|
|
|
|
|
class EmptyContext(object):
|
|
"""Just empty default context."""
|
|
|
|
def __init__(self, verifier, deployment):
|
|
pass
|
|
|
|
def __enter__(self):
|
|
return
|
|
|
|
def __exit__(self, exc_type, exc_value, exc_traceback):
|
|
# do nothing
|
|
return
|
|
|
|
|
|
class VerifierLauncher(plugin.Plugin):
|
|
def __init__(self, deployment, verifier):
|
|
"""Init launcher
|
|
|
|
:param deployment: `rally.common.objects.Deployment` instance
|
|
:param verifier: `rally.common.objects.Verifier` instance
|
|
"""
|
|
self.deployment = deployment
|
|
self.verifier = self.verifier
|
|
|
|
@property
|
|
def environ(self):
|
|
"""Customize environment variables."""
|
|
return os.environ.copy()
|
|
|
|
@property
|
|
def _with_venv(self):
|
|
"""Returns arguments for activation virtual environment if needed"""
|
|
if self.verifier.system_wide:
|
|
return []
|
|
# FIXME(andreykurilin): Currently, we use "tools/with_venv.sh" script
|
|
# from Tempest repository. We should remove this dependency.
|
|
return ["activate-venv"]
|
|
|
|
@property
|
|
def context(self):
|
|
ctx = self._meta_get("context")
|
|
if ctx:
|
|
ctx = VerifierContext.get(ctx)
|
|
return ctx or EmptyContext
|
|
|
|
def configure(self, override=False):
|
|
# by-default, verifier doesn't support this method
|
|
raise NotImplementedError
|
|
|
|
def configure_if_necessary(self):
|
|
"""Check existence of config file and create it if necessary."""
|
|
pass
|
|
|
|
def transform_kwargs(self, **kwargs):
|
|
"""Transform kwargs into the list of testr arguments."""
|
|
args = ["--subunit", "--parallel"]
|
|
if kwargs.get("concurrency"):
|
|
args.append("--concurrency")
|
|
args.append(kwargs["concurrency"])
|
|
if kwargs.get("re_run_failed"):
|
|
args.append("--failing")
|
|
if kwargs.get("file_with_tests"):
|
|
args.append("--load-list")
|
|
args.append(os.path.abspath(kwargs["file_with_tests"]))
|
|
if kwargs.get("regexp"):
|
|
args.append(kwargs["regexp"])
|
|
return args
|
|
|
|
def run(self, regexp=None, concurrency=None, re_run_failed=False,
|
|
file_with_tests=None):
|
|
self.configure_if_necessary()
|
|
|
|
cmd = [self._with_venv, "testr", "run"]
|
|
cmd.extend(self.transform_kwargs(
|
|
regexp=regexp, concurrency=concurrency,
|
|
re_run_failed=re_run_failed, file_with_tests=file_with_tests))
|
|
|
|
with self.context(self.deployment, self.verifier):
|
|
verification = subprocess.Popen(
|
|
cmd, env=self.environ(),
|
|
cwd=self.verifier.manager.home_dir,
|
|
stdout=subprocess.PIPE,
|
|
stderr=subprocess.stdout)
|
|
results = subunit_v2.parse(verification.stdout, live=True)
|
|
verification.wait()
|
|
return results
|
|
|
|
An example of VerifierLauncher for Tempest:
|
|
|
|
.. code-block:: python
|
|
|
|
@configure("tempest_verifier")
|
|
class TempestLauncher(VerifierLauncher):
|
|
|
|
@property
|
|
def configfile(self):
|
|
return os.path.join(self.verifier.manager.home_dir,
|
|
"for-deployment-%s" % self.deployment.id,
|
|
"tempest.conf")
|
|
|
|
@property
|
|
def environ(self):
|
|
"""Customize environment variables."""
|
|
env = super(TempestLauncher, self).environ
|
|
|
|
env["TEMPEST_CONFIG_DIR"] = os.path.dirname(self.configfile)
|
|
env["TEMPEST_CONFIG"] = os.path.basename(self.configfile)
|
|
env["OS_TEST_PATH"] = os.path.join(self.verifier.manager.home_dir,
|
|
"tempest", "test_discover")
|
|
return env
|
|
|
|
def configure(self, override=False):
|
|
if os.path.exists(self.configfile):
|
|
if override:
|
|
os.remove(self.configfile)
|
|
else:
|
|
raise AlreadyConfiguredException()
|
|
# Configure Tempest.
|
|
|
|
def configure_if_necessary(self):
|
|
try:
|
|
self.configure()
|
|
except AlreadyConfiguredException:
|
|
# nothing to do. everything is ok
|
|
pass
|
|
|
|
def run(self, set_name, **kwargs):
|
|
if set_name == "full":
|
|
pass
|
|
elif set_name in consts.TempestTestsSets:
|
|
kwargs["regexp"] = set_name
|
|
elif set_name in consts.TempestTestsAPI:
|
|
kwargs["regexp"] = "tempest.api.%s" % set_name
|
|
|
|
super(TempestLauncher, self).run(**kwargs)
|
|
|
|
VerifierContext
|
|
~~~~~~~~~~~~~~~
|
|
|
|
.. code-block:: python
|
|
|
|
from rally.plugins.openstack import osclients
|
|
from rally.task import context
|
|
|
|
|
|
class VerifierContext(context.Context):
|
|
|
|
def __init__(self, **ctx):
|
|
super(VerifierContext, self).__init__(ctx)
|
|
# There are no terms "task" and "scenario" in Verification
|
|
del self.task
|
|
del self.map_for_scenario
|
|
self.clients = osclients(self.context["deployment"].credentials)
|
|
|
|
@classmethod
|
|
def _meta_get(cls, key, default=None):
|
|
# It should be always hidden
|
|
if key == "hidden":
|
|
return True
|
|
return super(VerifierContext, cls)._meta_get(key, default)
|
|
|
|
|
|
Example of context for Tempest:
|
|
|
|
.. code-block:: python
|
|
|
|
@configure("tempest_verifier_ctx")
|
|
class TempestContext(VerifierContext):
|
|
|
|
def __init__(self, **kwargs):
|
|
super(TempestContext, self).__init__(**kwargs)
|
|
self.clients = osclients(self.context["deployment"].credentials)
|
|
|
|
def setup(self):
|
|
# create required resources and save them to self.context
|
|
pass
|
|
|
|
def cleanup(self):
|
|
# remove created resources
|
|
pass
|
|
|
|
|
|
Assignee(s)
|
|
-----------
|
|
|
|
Primary assignee:
|
|
Andrey Kurilin <andr.kurilin@gmail.com>
|
|
|
|
Work Items
|
|
----------
|
|
|
|
1) CLI and API related changes.
|
|
|
|
Lets provide new interface as soon as possible, even if some APIs will not
|
|
be implemented. As soon we deprecate old interface as soon we will be able
|
|
to remove it and provide clear new one.
|
|
|
|
2) Provide base classes for Verifiers
|
|
|
|
3) Rewrite Tempest verifier based on new classes.
|
|
|
|
|
|
Dependencies
|
|
============
|
|
|
|
None
|