openstack/rally-openstack
Code Issues Proposed changes

Merge "[docs] Add changelog file"

Browse Source
This commit is contained in:
Zuul 2018-03-22 17:48:53 +00:00 committed by Gerrit Code Review
commit 6106c536cd
27 changed files with 67 additions and 4609 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

67
CHANGELOG.rst Normal file
View File

@ -0,0 +1,67 @@
=========
Changelog
=========
.. Changelogs are for humans, not machines. The end users of Rally project are
human beings who care about what's is changing, why and how it affects them.
Please leave these notes as much as possible human oriented.
.. Each release can use the next sections:
- **Added** for new features.
- **Changed** for changes in existing functionality.
- **Deprecated** for soon-to-be removed features/plugins.
- **Removed** for now removed features/plugins.
- **Fixed** for any bug fixes.
.. Release notes for existing releases are MUTABLE! If there is something that
was missed or can be improved, feel free to change it!
[Unreleased]
============
Start a fork of `rally/plugins/openstack module of original OpenStack Rally
project
<https://github.com/openstack/rally/tree/0.11.1/rally/plugins/openstack>`_
Added
-----
* GnocchiArchivePolicy.list_archive_policy
* GnocchiArchivePolicy.create_archive_policy
* GnocchiArchivePolicy.create_delete_archive_policy
* GnocchiResourceType.list_resource_type
* GnocchiResourceType.create_resource_type
* GnocchiResourceType.create_delete_resource_type
* [ci] New Zuul V3 native jobs
Changed
-------
* Methods for association and dissociation floating ips were deprecated in
novaclient a year ago and latest major release (python-novaclient 10)
`doesn't include them
<https://github.com/openstack/python-novaclient/blob/10.0.0/releasenotes/notes/remove-virt-interfaces-add-rm-fixed-floating-398c905d9c91cca8.yaml>`_.
These actions should be performed via neutronclient now. It is not as simple
as it was via Nova-API and you can find more neutron-related atomic actions
in results of scenarios.
Removed
-------
* *os-hosts* CLIs and python API bindings had been deprecated in
python-novaclient 9.0.0 and became removed in `10.0.0 release
<https://github.com/openstack/python-novaclient/blob/10.0.0/releasenotes/notes/remove-hosts-d08855550c40b9c6.yaml>`_.
This decision affected 2 scenarios `NovaHosts.list_hosts
<https://rally.readthedocs.io/en/0.11.1/plugins/plugin_reference.html#novahosts-list-hosts-scenario>`_
and `NovaHosts.list_and_get_hosts
<https://rally.readthedocs.io/en/0.11.1/plugins/plugin_reference.html#novahosts-list-and-get-hosts-scenario>`_
which become redundant and we cannot leave them (python-novaclient doesn't
have proper interfaces any more).
Fixed
-----
* The support of `kubernetes python client
<https://pypi.python.org/pypi/kubernetes>`_ (which is used by Magnum plugins)
is not limited by 3.0.0 max version. You can use more modern releases of that
library.

11
doc/release_notes/archive.rst
View File

@ -1,11 +0,0 @@
.. _release_notes/archive:
=================
All release notes
=================
.. toctree::
:glob:
:maxdepth: 1
archive/*

28
doc/release_notes/archive/v0.0.1.rst
View File

@ -1,28 +0,0 @@
============
Rally v0.0.1
============
Information
-----------
+------------------+-----------------+
| Commits | **1039** |
+------------------+-----------------+
| Bug fixes | **0** |
+------------------+-----------------+
| Dev cycle | **547 days** |
+------------------+-----------------+
| Release date | **26/Jan/2015** |
+------------------+-----------------+
Details
-------
Rally is awesome tool for generic testing of OpenStack clouds.
A lot of people started using Rally in their CI/CD so Rally team should provide
more stable product with clear strategy of deprecation and upgrades.

223
doc/release_notes/archive/v0.0.2.rst
View File

@ -1,223 +0,0 @@
============
Rally v0.0.2
============
Information
-----------
+------------------+-----------------+
| Commits | **100** |
+------------------+-----------------+
| Bug fixes | **18** |
+------------------+-----------------+
| Dev cycle | **45 days** |
+------------------+-----------------+
| Release date | **12/Mar/2015** |
+------------------+-----------------+
Details
-------
This release contains new features, new task plugins, bug fixes,
various code and API improvements.
New Features
~~~~~~~~~~~~
* rally task start **--abort-on-sla-failure**
Stopping load before things go wrong.
Load generation will be interrupted if SLA criteria stop passing.
* Rally verify command supports multiple Tempest sources now.
* python34 support
* postgres DB backend support
API changes
~~~~~~~~~~~
* [new] **rally [deployment | verify | task] use** subcommand
It should be used instead of root command **rally use**
* [new] Rally as a Lib API
To avoid code duplication between Rally as CLI tool and Rally as a Service
we decide to make Rally as a Lib as a common part between these 2 modes.
Rally as a Service will be a daemon that just maps HTTP request to Rally
as a Lib API.
* [deprecated] **rally use** CLI command
* [deprecated] Old Rally as a Lib API
Old Rally API was quite mixed up so we decide to deprecate it
Plugins
~~~~~~~
* **Task Runners**:
[improved] Improved algorithm of generation load in **constant runner**
Before we used processes to generate load, now it creates pool of
processes (amount of processes is equal to CPU count) after that in
each process use threads to generate load. So now you can easily
generate load of 1k concurrent scenarios.
[improved] Unify code of **constant** and **rps** runners
[interface] Added **abort()** to runner's plugin interface
New method **abort()** is used to immediately interrupt execution.
* **Task Scenarios**:
[new] DesignateBasic.create_and_delete_server
[new] DesignateBasic.create_and_list_servers
[new] DesignateBasic.list_servers
[new] MistralWorkbooks.list_workbooks
[new] MistralWorkbooks.create_workbook
[new] Quotas.neutron_update
[new] HeatStacks.create_update_delete_stack
[new] HeatStacks.list_stacks_and_resources
[new] HeatStacks.create_suspend_resume_delete_stac
[new] HeatStacks.create_check_delete_stack
[new] NeutronNetworks.create_and_delete_routers
[new] NovaKeypair.create_and_delete_keypair
[new] NovaKeypair.create_and_list_keypairs
[new] NovaKeypair.boot_and_delete_server_with_keypair
[new] NovaServers.boot_server_from_volume_and_live_migrate
[new] NovaServers.boot_server_attach_created_volume_and_live_migrate
[new] CinderVolumes.create_and_upload_volume_to_image
[fix] CinderVolumes.create_and_attach_volume
Pass optional \*\*kwargs only to create server command
[fix] GlanceImages.create_image_and_boot_instances
Pass optional \*\*kwargs only to create server command
[fix] TempestScenario.\* removed stress cleanup.
Major issue is that tempest stress cleanup cleans whole OpenStack.
This is very dangerous, so it's better to remove it and leave some
extra resources.
[improved] NovaSecGroup.boot_and_delete_server_with_secgroups
Add optional \*\*kwargs that are passed to boot server comment
* **Task Context**:
[new] **stacks**
Generates passed amount of heat stacks for all tenants.
[new] **custom_image**
Prepares images for internal VMs testing.
To Support generating workloads in VMs by existing tools like: IPerf,
Blogbench, HPCC and others we have to have prepared images, with
already installed and configured tools.
Rally team decide to generate such images on fly from passed to avoid
requirements of having big repository with a lot of images.
This context is abstract context that allows to automate next steps:
1) runs VM with passed image (with floating ip and other stuff)
2) execute abstract method that has access to VM
3) snapshot this image
In future we are going to use this as a base for making context that
prepares images.
[improved] **allow_ssh**
Automatically disable it if security group are disabled in neutron.
[improved] **keypair**
Key pairs are stored in "users" space it means that accessing keypair
from scenario is simpler now:
self.context["user"]["keypair"]["private"]
[fix] **users**
Pass proper EndpointType for newly created users
[fix] **sahara_edp**
The Job Binaries data should be treated as a binary content
* **Task SLA**:
[interface] SLA calculations is done in additive way now
Resolves scale issues, because now we don't need to have whole
array of iterations in memory to process SLA.
This is required to implement **--abort-on-sla-failure** feature
[all] SLA plugins were rewritten to implement new interface
Bug fixes
~~~~~~~~~
**18 bugs were fixed, the most critical are**:
* Fix **rally task detailed --iterations-data**
It didn't work in case of missing atomic actions. Such situation can occur
if scenario method raises exceptions
* Add user-friendly message if the task cannot be deleted
In case of trying to delete task that is not in "finished" status users get
traces instead of user-friendly message try to run it with --force key.
* Network context cleanups networks properly now
Documentation
~~~~~~~~~~~~~
* Image sizes are fixed
* New tutorial in "Step by Step" relate to **--abort-on-sla-failure**
* Various fixes

153
doc/release_notes/archive/v0.0.3.rst
View File

@ -1,153 +0,0 @@
============
Rally v0.0.3
============
Information
-----------
+------------------+-----------------+
| Commits | **53** |
+------------------+-----------------+
| Bug fixes | **14** |
+------------------+-----------------+
| Dev cycle | **33 days** |
+------------------+-----------------+
| Release date | **14/Apr/2015** |
+------------------+-----------------+
Details
-------
This release contains new features, new task plugins, bug fixes,
various code and API improvements.
New Features & API changes
~~~~~~~~~~~~~~~~~~~~~~~~~~
* Add the ability to specify versions for clients in scenarios
You can call self.clients("glance", "2") and get any client for
specific version.
* Add API for tempest uninstall
$ rally-manage tempest uninstall
# removes fully tempest for active deployment
* Add a --uuids-only option to rally task list
$ rally task list --uuids-only # returns list with only task uuids
* Adds endpoint to --fromenv deployment creation
$ rally deployment create --fromenv
# recognizes standard OS_ENDPOINT environment variable
* Configure SSL per deployment
Now SSL information is deployment specific not Rally specific and
rally.conf option is deprecated
Like in this sample
https://github.com/openstack/rally/blob/14d0b5ba0c75ececfdb6a6c121d9cf2810571f77/samples/deployments/existing.json#L11-L12
Specs
~~~~~
* [spec] Proposal for new task input file format
This spec describes new task input format that will allow us to generate
multi scenario load which is crucial for HA and more real life testing:
https://github.com/openstack/rally/blob/master/doc/specs/in-progress/new_rally_input_task_format.rst
Plugins
~~~~~~~
* **Task Runners**:
* Add a maximum concurrency option to rps runner
To avoid running to heavy load you can set 'concurrency' to configuration
and in case if cloud is not able to process all requests it won't start
more parallel requests then 'concurrency' value.
* **Task Scenarios**:
[new] CeilometerAlarms.create_alarm_and_get_history
[new] KeystoneBasic.get_entities
[new] EC2Servers.boot_server
[new] KeystoneBasic.create_and_delete_service
[new] MuranoEnvironments.list_environments
[new] MuranoEnvironments.create_and_delete_environment
[new] NovaServers.suspend_and_resume_server
[new] NovaServers.pause_and_unpause_server
[new] NovaServers.boot_and_rebuild_server
[new] KeystoneBasic.create_and_list_services
[new] HeatStacks.list_stacks_and_events
[improved] VMTask.boot_runcommand_delete
restore ability to use fixed IP and floating IP to connect to VM
via ssh
[fix] NovaServers.boot_server_attach_created_volume_and_live_migrate
Kwargs in nova scenario were wrongly passed
* **Task SLA**:
* [new] aborted_on_sla
This is internal SLA criteria, that is added if task was aborted
* [new] something_went_wrong
This is internal SLA criteria, that is added if something went wrong,
context failed to create or runner raised some exceptions
Bug fixes
~~~~~~~~~
**14 bugs were fixed, the most critical are**:
* Set default task uuid to running task. Before it was set only after
task was fully finished.
* The "rally task results" command showed a disorienting "task not found"
message for a task that is currently running.
* Rally didn't know how to reconnect to OpenStack in case if token
expired.
Documentation
~~~~~~~~~~~~~
* New tutorial **task templates**
https://rally.readthedocs.org/en/latest/tutorial/step_5_task_templates.html
* Various fixes

222
doc/release_notes/archive/v0.0.4.rst
View File

@ -1,222 +0,0 @@
============
Rally v0.0.4
============
Information
-----------
+------------------+-----------------+
| Commits | **87** |
+------------------+-----------------+
| Bug fixes | **21** |
+------------------+-----------------+
| Dev cycle | **30 days** |
+------------------+-----------------+
| Release date | **14/May/2015** |
+------------------+-----------------+
Details
-------
This release contains new features, new task plugins, bug fixes, various code and API improvements.
New Features & API changes
~~~~~~~~~~~~~~~~~~~~~~~~~~
* Rally now can generate load with users that already exist
Now one can use Rally for testing OpenStack clouds that are using LDAP, AD or
any other read-only keystone backend where it is not possible to create any
users. To do this, one should set up the "users" section of the deployment
configuration of the ExistingCloud type. This feature also makes it safer to
run Rally against production clouds: when run from an isolated group of
users, Rally won't affect rest of the cloud users if something goes wrong.
* New decorator *@osclients.Clients.register* can add new OpenStack clients
at runtime
It is now possible to add a new OpenStack client dynamically at runtime.
The added client will be available from osclients.Clients at the
module level and cached. Example:
.. code-block:: none
>>> from rally import osclients
>>> @osclients.Clients.register("supernova")
... def another_nova_client(self):
... from novaclient import client as nova
... return nova.Client("2", auth_token=self.keystone().auth_token,
... **self._get_auth_info(password_key="key"))
...
>>> clients = osclients.Clients.create_from_env()
>>> clients.supernova().services.list()[:2]
[<Service: nova-conductor>, <Service: nova-cert>]
* Assert methods now available for scenarios and contexts
There is now a new *FunctionalMixin* class that implements basic unittest
assert methods. The *base.Context* and *base.Scenario* classes inherit from
this mixin, so now it is possible to use *base.assertX()*
methods in scenarios and contexts.
* Improved installation script
The installation script has been almost completely rewritten. After this
change, it can be run from an unprivileged user, supports different database
types, allows to specify a custom python binary, always asks confirmation
before doing potentially dangerous actions, automatically install needed
software if run as root, and also automatically cleans up the
virtualenv and/or the downloaded repository if interrupted.
Specs & Feature requests
~~~~~~~~~~~~~~~~~~~~~~~~
* [Spec] Reorder plugins
The spec describes how to split Rally framework and plugins codebase to make
it simpler for newbies to understand how Rally code is organized and
how it works.
* [Feature request] Specify what subtasks to execute in task
This feature request proposes to add the ability to specify subtask(s)
to be executed when the user runs the *rally task start* command. A possible
solution would be to add a special flag to the *rally task start* command.
Plugins
~~~~~~~
* **Task Runners**:
* Add limits for maximum Core usage to constant and rps runners
The new 'max_cpu_usage' parameter can be used to avoid possible 100%
usage of all available CPU cores by reducing the number of CPU cores
available for processes started by the corresponding runner.
* **Task Scenarios**:
* [new] KeystoneBasic.create_update_and_delete_tenant
* [new] KeystoneBasic.create_user_update_password
* [new] NovaServers.shelve_and_unshelve_server
* [new] NovaServers.boot_and_associate_floating_ip
* [new] NovaServers.boot_lock_unlock_and_delete
* [new] NovaHypervisors.list_hypervisors
* [new] CeilometerSamples.list_samples
* [new] CeilometerResource.get_resources_on_tenant
* [new] SwiftObjects.create_container_and_object_then_delete_all
* [new] SwiftObjects.create_container_and_object_then_download_object
* [new] SwiftObjects.create_container_and_object_then_list_objects
* [new] MuranoEnvironments.create_and_deploy_environment
* [new] HttpRequests.check_random_request
* [new] HttpRequests.check_request
* [improved] NovaServers live migrate scenarios
add 'min_sleep' and 'max_sleep' parameters to simulate a pause between
VM booting and running live migration
* [improved] NovaServers.boot_and_live_migrate_server
add a usage sample to samples/tasks
* [improved] CinderVolumes scenarios
support size range to be passed to the 'size' argument as a dictionary
*{"min": <minimum_size>, "max": <maximum_size>}*
* **Task Contexts**:
* [new] MuranoPackage
This new context can upload a package to Murano from some specified path.
* [new] CeilometerSampleGenerator
Context that can be used for creating samples and collecting resources
for testing of list operations.
* **Task SLA**:
* [new] outliers
This new SLA checks that the number of outliers (calculated from the mean
and standard deviation of the iteration durations) does not exceed some
maximum value. The SLA is highly configurable: the parameters used for
outliers threshold calculation can be set by the user.
Bug fixes
~~~~~~~~~
**21 bugs were fixed, the most critical are**:
* Make it possible to use relative imports for plugins that are outside of
rally package.
* Fix heat stacks cleanup by deleting them only 1 time per tenant
(get rid of "stack not found" errors in logs).
* Fix the wrong behavior of 'rally task detailed --iterations-data'
(it lacked the iteration info before).
* Fix security groups cleanup: a security group called "default", created
automatically by Neutron, did not get deleted for each tenant.
Other changes
~~~~~~~~~~~~~~~~~~~~~~~~~~
* Streaming algorithms that scale
This release introduces the common/streaming_algorithms.py module.
This module is going to contain implementations of task data processing
algorithms that scale: these algorithms do not store exhaustive information
about every single subtask iteration duration processed. For now, the module
contains implementations of algorithms for
computation of mean & standard deviation.
* Coverage job to check that new patches come with unit tests
Rally now has a coverage job that checks that every patch submitted for
review does not decrease the number of lines covered by unit tests
(at least too much). This job allows to mark most patches with no
unit tests with '-1'.
* Splitting the plugins code (Runners & SLA) into common/openstack plugins
According to the spec "Reorder plugins" (see above), the plugins code for
runners and SLA has been moved to the *plugins/common/* directory.
Only base classes now remain in the *benchmark/* directory.
Documentation
~~~~~~~~~~~~~
* Various fixes
* Remove obsolete *.rst* files
(*deploy_engines.rst* / *server_providers.rst* / ...)
* Restructure the docs files to make them easier to navigate through
* Move the chapter on task templates to the 4th step in the tutorial
* Update the info about meetings (new release meeting & time changes)

514
doc/release_notes/archive/v0.1.0.rst
View File

@ -1,514 +0,0 @@
============
Rally v0.1.0
============
Information
-----------
+------------------+-----------------------+
| Commits | **355** |
+------------------+-----------------------+
| Bug fixes | **90** |
+------------------+-----------------------+
| Dev cycle | **132 days** |
+------------------+-----------------------+
| Release date | **25/September/2015** |
+------------------+-----------------------+
Details
-------
This release contains new features, new 42 plugins, 90 bug fixes,
various code and API improvements.
New Features & API changes
~~~~~~~~~~~~~~~~~~~~~~~~~~
* **Improved installation script**
* Add parameters:
* ``--develop`` parameter to install rally in editable (develop) mode
* ``--no-color`` to switch off output colorizing
useful for automated output parsing and terminals that don't
support colors.
* Puts rally.conf under virtualenv etc/rally/ so you can have several
rally installations in virtualenv
* Many fixes related to access of different file, like: rally.conf,
rally db file in case of sqlite
* Update pip before Rally installation
* Fix reinstallation
* **Separated Rally plugins & framework**
Now plugins are here:
https://github.com/openstack/rally/tree/master/rally/plugins
Plugins are as well separated common/* for common plugins
that can be use no matter what is tested and OpenStack related
plugins
* **New Rally Task framework**
* All plugins has the same Plugin base:
rally.common.plugin.pluing.Plugin They have the same mechanisms for:
discovering, providing information based on docstrings, and in future
they will use the same deprecation/rename mechanism.
* Some of files are moved:
* rally/benchmark -> rally/task
*This was done to unify naming of rally task command and
actually code that implements it.*
* rally/benchmark/sla/base.py -> rally/task/sla.py
* rally/benchmark/context/base.py -> rally/task/context.py
* rally/benchmark/scenarios/base.py -> rally/task/scenario.py
* rally/benchmark/runners/base.py -> rally/task/runner.py
* rally/benchmark/scenarios/utils.py -> rally/task/utils.py
This was done to:
* avoid doing rally.benchmark.scenarios import base as scenario_base
* remove one level of nesting
* simplify framework structure
* Some of classes and methods were renamed
* Plugin configuration:
* context.context() -> context.configure()
* scenario.scenario() -> scenario.configure()
* Introduced runner.configure()
* Introduced sla.configure()
This resolves 3 problems:
* Unifies configuration of different types of plugins
* Simplifies plugin interface
* Looks nice with new modules path:
>>> from rally.task import scenario
>>> @scenario.configure()
* Atomic Actions were changed:
* New rally.task.atomic module
This allow us in future to reuse atomic actions in Context plugins
* Renames:
rally.benchmark.scenarios.base.AtomicAction
-> rally.task.atomic.ActionTimer
rally.benchmark.scenarios.base.atomic_action()
-> rally.task.atomic.action_timer()
* **Context plugins decide how to map their data for scenario**
Now Context.map_for_scenario method can be override to decide
how to pass context object to each iteration of scenario.
* Samples of NEW vs OLD context, sla, scenario and runner plugins:
* Context
.. code-block:: python
# Old
from rally.benchmark.context import base
@base.context(name="users", order=100)
class YourContext(base.Context):
def setup(self):
# ...
def cleanup(self):
# ...
# New
from rally.task import context
@context.configure(name="users", order=100)
class YourContext(context.Context):
def setup(self):
# ...
def cleanup(self):
# ...
def map_for_scenario(self):
# Maps context object to the scenario context object
# like context["users"] -> context["user"] and so on.
..
* Scenario
.. code-block:: python
# Old Scenario
from rally.benchmark.scenarios import base
from rally.benchmark import validation
class ScenarioPlugin(base.Scenario):
@base.scenario()
def some(self):
self._do_some_action()
@base.atomic_action_timer("some_timer")
def _do_some_action(self):
# ...
# New Scenario
from rally.task import atomic
from rally.task import scenario
from rally.task import validation
# OpenStack scenario has different base now:
# rally.plugins.openstack.scenario.OpenStackScenario
class ScenarioPlugin(scenario.Scenario):
@scenario.configure()
def some(self):
self._do_some_action()
@atomic.action_timer("some_action")
def _do_some_action(self):
# ...
..
* Runner
.. code-block:: python
## Old
from rally.benchmark.runners import base
class SomeRunner(base.ScenarioRunner):
__execution_type__ = "some_runner"
def _run_scenario(self, cls, method_name, context, args)
# Load generation
def abort(self):
# Method that aborts load generation
## New
from rally.task import runner
@runner.configure(name="some_runner")
class SomeRunner(runner.ScenarioRunner):
def _run_scenario(self, cls, method_name, context, args)
# Load generation
def abort(self):
# Method that aborts load generation
..
* SLA
.. code-block:: python
# Old
from rally.benchmark import sla
class FailureRate(sla.SLA):
# ...
# New
from rally.task import sla
@sla.configure(name="failure_rate")
class FailureRate(sla.SLA):
# ...
..
* **Rally Task aborted command**
Finally you can gracefully shutdown running task by calling:
.. code:: bash
rally task abort <task_uuid>
..
* **Rally CLI changes**
* [add] ``rally --plugin-paths`` specify the list of directories with plugins
* [add] ``rally task report --junit`` - generate a JUnit report
This allows users to feed reports to tools such as Jenkins.
* [add] ``rally task abort`` - aborts running Rally task
when run with the ``--soft`` key, the ``rally task abort`` command is
waiting until the currently running subtask is finished, otherwise the
command interrupts subtask immediately after current scenario iterations
are finished.
* [add] ``rally plugin show`` prints detailed information about plugin
* [add] ``rally plugin list`` prints table with rally plugin names and titles
* [add] ``rally verify genconfig`` generates tempest.conf without running it.
* [add] ``rally verify install`` install tempest for specified deployment
* [add] ``rally verify reinstall`` removes tempest for specified deployment
* [add] ``rally verify uninstall`` uninstall tempest of specified deployment
* [fix] ``rally verify start --no-use`` --no-use was always turned on
* [remove] ``rally use`` now each command has subcommand ``use``
* [remove] ``rally info``
* [remove] ``rally-manage tempest`` now it is covered by ``rally verify``
* **New Rally task reports**
* New code is based on OOP style which is base step to make pluggable Reports
* Reports are now generated for only one iteration over the resulting data
which resolves scalability issues when we are working with large
amount of iterations.
* New Load profiler plot that shows amount of iterations that are working
in parallel
* Failed iterations are shown as a red areas on stacked are graphic.
Non backward compatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* [remove] ``rally use`` cli command
* [remove] ``rally info`` cli command
* [remove] ``--uuid`` parameter from ``rally deployment <any>``
* [remove ``--deploy-id`` parameter from:
``rally task <any>``, ``rally verify <any>``, ``rally show <any>``
Specs & Feature requests
~~~~~~~~~~~~~~~~~~~~~~~~
[feature request] Explicitly specify existing users for scenarios
[feature request] Improve install script and add --uninstall and --version
[feature request] Allows specific repos & packages in install-rally.sh
[feature request] Add ability to capture logs from tested services
[feature request] Check RPC queue perfdata
[spec] Refactoring Rally cleanup
[spec] Consistent resource names
Plugins
~~~~~~~
* **Scenarios**:
[new] CinderVolumes.create_volume_backup
[new] CinderVolumes.create_and_restore_volume_backup
[new] KeystoneBasic.add_and_remove_user_role
[new] KeystoneBasic.create_and_delete_role
[new] KeystoneBasic.create_add_and_list_user_roles
[new] FuelEnvironments.list_environments
[new] CinderVolumes.modify_volume_metadata
[new] NovaServers.boot_and_delete_multiple_servers
[new] NeutronLoadbalancerV1.create_and_list_pool
[new] ManilaShares.list_shares
[new] CeilometerEvents.create_user_and_get_event
[new] CeilometerEvents.create_user_and_list_event_types
[new] CeilometerEvents.create_user_and_list_events
[new] CeilometerTraits.create_user_and_list_trait_descriptions
[new] CeilometerTraits.create_user_and_list_traits
[new] NeutronLoadbalancerV1.create_and_delete_pools
[new] NeutronLoadbalancerV1.create_and_update_pools
[new] ManilaShares.create_and_delete_share
[new] ManilaShares.create_share_network_and_delete
[new] ManilaShares.create_share_network_and_list
[new] HeatStacks.create_and_delete_stack
[new] ManilaShares.list_share_servers
[new] HeatStacks.create_snapshot_restore_delete_stack
[new] KeystoneBasic.create_and_delete_ec2credential
[new] KeystoneBasic.create_and_list_ec2credentials
[new] HeatStacks.create_stack_and_scale
[new] ManilaShares.create_security_service_and_delete
[new] KeystoneBasic.create_user_set_enabled_and_delete
[new] ManilaShares.attach_security_service_to_share_network
[new] IronicNodes.create_and_delete_node
[new] IronicNodes.create_and_list_node
[new] CinderVolumes.create_and_list_volume_backups
[new] NovaNetworks.create_and_list_networks
[new] NovaNetworks.create_and_delete_network
[new] EC2Servers.list_servers
[new] VMTasks.boot_runcommand_delete_custom_imagea
[new] CinderVolumes.create_and_update_volume
* **Contexts**:
[new] ManilaQuotas
Add context for setting up Manila quotas:
shares, gigabytes, snapshots, snapshot_gigabytes, share_networks
[new] ManilaShareNetworks
Context for share networks that will be used in case of usage
deployment with existing users. Provided share networks via context
option "share_networks" will be balanced between all share creations
of scenarios.
[new] Lbaas
Context to create LBaaS-v1 resources
[new] ImageCommandCustomizerContext
Allows image customization using side effects of a command execution.
E.g. one can install an application to the image and use these image
for 'boot_runcommand_delete' scenario afterwards.
[new] EC2ServerGenerator
Context that creates servers using EC2 api
[new] ExistingNetwork
This context lets you use existing networks that have already been
created instead of creating new networks with Rally. This is useful
when, for instance, you are using Neutron with a dumb router that is
not capable of creating new networks on the fly.
* **SLA**:
[remove] max_failure_rate - use failure_rate instead
Bug fixes
~~~~~~~~~
**90 bugs were fixed, the most critical are**:
* Many fixes related that fixes access of rally.conf and DB files
* Incorrect apt-get "-yes" parameter in install_rally.sh script
* Rally bash completion doesn't exist in a virtualenv
* Rally show networks CLI command worked only with nova networks
* RPS runner was not properly generating load
* Check is dhcp_agent_scheduler support or not in network cleanup
* NetworkContext doesn't work with Nova V2.1
* Rally task input file was not able to use jinja2 include directive
* Rally in docker image was not able to
* Rally docker image didn't contain samples
* Do not update the average duration when iteration failed
Documentation
~~~~~~~~~~~~~
* **Add plugin reference page**
:ref:`Rally Plugins Reference page <plugin-reference>` page contains a
full list with
* **Add maintainers section on project info page**
:ref:`Rally Maintainers section <project_info>` contains information
about core contributors of OpenStack Rally their responsibilities and
contacts. This will help us to make our community more transparent and open
for newbies.
* **Added who is using section in docs**
* **Many small fixes**

131
doc/release_notes/archive/v0.1.1.rst
View File

@ -1,131 +0,0 @@
============
Rally v0.1.1
============
Information
-----------
+------------------+-----------------------+
| Commits | **32** |
+------------------+-----------------------+
| Bug fixes | **9** |
+------------------+-----------------------+
| Dev cycle | **11 days** |
+------------------+-----------------------+
| Release date | **6/October/2015** |
+------------------+-----------------------+
Details
-------
This release contains new features, new 6 plugins, 9 bug fixes,
various code and API improvements.
New Features
~~~~~~~~~~~~
* **Rally verify generates proper tempest.conf file now**
Improved script that generates tempest.conf, now it works out of box for
most of the clouds and most of Tempest tests will pass without hacking it.
* **Import Tempest results to Rally DB**
``rally verify import`` command allows you to import already existing Tempest
results and work with them as regular "rally verify start" results:
generate HTML/CSV reports & compare different runs.
API Changes
~~~~~~~~~~~~
**Rally CLI changes**
* [add] ``rally verify import`` imports raw Tempest results to Rally
Specs & Feature requests
~~~~~~~~~~~~~~~~~~~~~~~~
There is no new specs and feature requests.
Plugins
~~~~~~~
* **Scenarios**:
[new] NeutronNetworks.create_and_list_floating_ips
[new] NeutronNetworks.create_and_delete_floating_ips
[new] MuranoPackages.import_and_list_packages
[new] MuranoPackages.import_and_delete_package
[new] MuranoPackages.import_and_filter_applications
[new] MuranoPackages.package_lifecycle
[improved] NovaKeypair.boot_and_delete_server_with_keypair
New argument ``server_kwargs``, these kwargs are used to boot server.
[fix] NeutronLoadbalancerV1.create_and_delete_vips
Now it works in case of concurrency > 1
* **Contexts**:
[improved] network
Network context accepts two new arguments:
``subnets_per_network`` and ``network_create_args``.
[fix] network
Fix cleanup if nova-network is used. Networks should be dissociate from
project before deletion
[fix] custom_image
Nova server that is used to create custom image was not deleted if
script that prepares server failed.
Bug fixes
~~~~~~~~~
**9 bugs were fixed, the most critical are**:
* Fix install_rally.sh script
Set 777 access to /var/lib/rally/database file if system-wide method of
installation is used.
* Rally HTML reports Overview table had few mistakes
* Success rate was always 100%
* Percentiles were wrongly calculated
* Missing Ironic, Murano and Workload(vm) options in default config file
* ``rally verify start`` failed while getting network_id
* ``rally verify genconfig`` hangs forever if Horizon is not available
Documentation
~~~~~~~~~~~~~
* **Fix project maintainers page**
Update the information about Rally maintainers
* **Document rally --plugin-paths CLI argument**
* **Code blocks in documentation looks prettier now**

206
doc/release_notes/archive/v0.1.2.rst
View File

@ -1,206 +0,0 @@
============
Rally v0.1.2
============
Information
-----------
+------------------+-----------------------+
| Commits | **208** |
+------------------+-----------------------+
| Bug fixes | **37** |
+------------------+-----------------------+
| Dev cycle | **77 days** |
+------------------+-----------------------+
| Release date | **23/December/2015** |
+------------------+-----------------------+
Details
-------
This release, as well as all previous ones, includes a lot of internal and
external changes. Most important of them are listed below.
.. warning:: Release 0.1.2 is the last release with Python 2.6 support.
Deprecations
~~~~~~~~~~~~
* Class `rally.common.objects.Endpoint` was renamed to `Credentials`. Old
class is kept for backward compatibility. Please, stop using the old class
in your plugins.
.. warning:: dict key was changed too in user context from "endpoint" to "credential"
* rally.task.utils: wait_is_ready(), wait_for(), wait_for_delete() deprecated
you should use wait_for_status() instead.
Rally Verify
~~~~~~~~~~~~
* Added possibility to run Tempest tests listed in a file(--tests-file argument in ``verify start``)
* Added possibility to upload Tempest subunit stream logs into data base
* Improvements in generating Tempest config file
* Reworked subunit stream parser
* Don't install Tempest when `rally verify [gen/show]config`
* Rally team tries to simplify usage of each our component.
Now Rally verification has some kind of a context like in Tasks.
Before launching each verification, Rally checks existence of required
resources(networks, images, flavours, etc) in Tempest configuration file and
pre-creates them. Do not worry, all these resources will not be forgotten
and left, Rally will clean them after verification.
Rally Task
~~~~~~~~~~
* Add --html-static argument to ``rally task report`` which allows to
generate HTML reports that doesn't require Internet.
* Rally supports different API versions now via api_versions context:
.. code-block:: none
CinderVolumes.create_and_delete_volume:
-
args:
size: 1
runner:
type: "constant"
times: 2
concurrency: 2
context:
users:
tenants: 2
users_per_tenant: 2
api_versions:
cinder:
version: 2
service_name: cinderv2
* Move rally.osclients.Clients to plugin base
Rally OSclients is pluggable now and it is very easy to extend OSClient for
your cloud out of Rally tree.
* Add 'merge' functionality to SLA
All SLA plugins should implement merge() method now.
In future this will be used for distributed load generation.
Where SLA results from different runners will be merged together.
* New optional_action_timer decorator
Allows to make the methods that can be both atomic_action or regular
method. Method changes behavior based on value in extra key "atomic_action"
Rally Certification
~~~~~~~~~~~~~~~~~~~
* Fix Glance certification arguments
* Add Neutron Quotas only if Neutron service is available
Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~
* Spec consistent-resource-names:
Resource name is based on Task id now. It is a huge step to persistence
and disaster cleanups.
* Add a spec for distributed load generation:
https://github.com/openstack/rally/blob/master/doc/specs/in-progress/distributed_runner.rst
* Improvements for scenario output format
https://github.com/openstack/rally/blob/master/doc/specs/in-progress/improve_scenario_output_format.rst
* Task and Verify results export command
https://github.com/openstack/rally/blob/master/doc/specs/in-progress/task_and_verification_export.rst
Plugins
~~~~~~~
* **Scenarios**:
* [new] NovaServers.boot_and_get_console_output
* [new] NovaServers.boot_and_show_server
* [new] NovaServers.boot_server_attach_created_volume_and_resize
* [new] NovaServers.boot_server_from_volume_and_resize
* [new] NeutronSecurityGroup.create_and_delete_security_groups
* [new] NeutronSecurityGroup.create_and_list_security_groups
* [new] NeutronSecurityGroup.create_and_update_security_groups
* [new] NeutronLoadbalancerV1.create_and_delete_healthmonitors
* [new] NeutronLoadbalancerV1.create_and_list_healthmonitors
* [new] NeutronLoadbalancerV1.create_and_update_healthmonitors
* [new] SwiftObjects.list_and_download_objects_in_containers
* [new] SwiftObjects.list_objects_in_containers
* [new] FuelNodes.add_and_remove_node
* [new] CeilometerMeters.list_matched_meters
* [new] CeilometerResource.list_matched_resources
* [new] CeilometerSamples.list_matched_samples
* [new] CeilometerStats.get_stats
* [new] Authenticate.validate_monasca
* [new] DesignateBasic.create_and_delete_zone
* [new] DesignateBasic.create_and_list_zones
* [new] DesignateBasic.list_recordsets
* [new] DesignateBasic.list_zones
* [fix] CinderVolumes.create_nested_snapshots_and_attach_volume
Remove random nested level which produce different amount of atomic
actions and bad reports.
* Support for Designate V2 api
* A lot of improvements in Sahara scenarios
* **Context**:
* [new] api_versions
Context allows us to setup client to communicate to specific service.
* [new] swift_objects
Context pre creates swift objects for future usage in scenarios
* [update] sahara_cluster
It supports proxy server which allows to use single floating IP for
whole cluster.
* [fix] cleanup
Fix cleanup of networks remove vip before port.
Bug fixes
~~~~~~~~~
**37 bugs were fixed, the most critical are**:
* Follow symlinks in plugin discovery
* Use sed without -i option for portability (install_rally.sh)
* Fixed race in rally.common.broker
* Fixed incorrect iteration number on "Failures" Tab
* Fixing issue with create_isolated_networks = False
* Fix docker build command
Documentation
~~~~~~~~~~~~~
Fixed some minor typos and inaccuracies.
Thanks
~~~~~~
We would like to thank Andreas Jaeger for ability to provide Python 2.6 support in this release.

520
doc/release_notes/archive/v0.10.0.rst
View File

@ -1,520 +0,0 @@
=============
Rally v0.10.0
=============
Overview
--------
+------------------+-----------------------+
| Release date | **10/20/2017** |
+------------------+-----------------------+
* Ability to use OpenStack deployments without admin credentials
* Better validation process of tasks
* New task format (with an ability to set description for workloads)
* New JSON report
* ElasticSearch exporter
* `OSProfiler support
<https://rally.readthedocs.io/en/0.10.0/quick_start/tutorial/step_10_profiling_openstack_internals.html>`_
* Restructure of configuration options.
Details
-------
Command Line Interface
~~~~~~~~~~~~~~~~~~~~~~
* Introduce `rally task import
<https://rally.readthedocs.io/en/0.10.0/cli_reference.html#rally-task-import>`_
command for importing task results into database.
* Extend tags support for tasks. Now you can specify several tags for a single
task using `--tag argument
<https://rally.readthedocs.io/en/0.10.0/cli_reference.html#task-start-tag>`_.
Also filtering tasks by tags is now available.
* Move DB category from ``rally-manage db`` to `rally db
<https://rally.readthedocs.io/en/0.10.0/cli_reference.html#category-db>`_ and
introduce `rally db show
<https://rally.readthedocs.io/en/0.10.0/cli_reference.html#rally-db-show>`_
command for printing the used connection string.
Deployments
~~~~~~~~~~~
This release we started a huge work related to simplification of deployment
component of Rally. There is a good progress which includes several nice
features:
* The format.
"ExistingCloud" deployment type covers 99.99% cases and is used as a base for
all new things. Also, it will be extended to support different platforms
soon. The new format looks like (for OpenStack case):
.. code-block:: json
{
"openstack": {
"admin": {
"username": "admin",
"password": "changeme",
"tenant_name": "foo",
},
"auth_url": "https://example.com",
}
}
* admin user is optional in case of setting existing users.
From the beginning, setting admin credentials was a required section of Rally
deployment configuration. Even with introducing existing users feature, this
behaviour left.
Finally, we finished a big code refactoring and admin credential become
optional section. If a set of plugins for particular workload doesn't require
admin user, you can launch this task at deployment without setting it.
The information about the requirements of plugins you can find at
`Plugins Reference page
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html>`_ (see
``Requires platform(s):`` section at the bottom of each plugin).
* Originally, Rally project was designed to check performance of OpenStack and
we succeeded in building awesome tool. We do not plan to stop and just want
to inform about our future plans to expand a number of supported platforms.
Subscribe to our `GitHub organization
<https://github.com/xrally>`_ to not miss new plugins.
Task component
~~~~~~~~~~~~~~
* The new task format is introduced. It includes a bunch of improvements,
unification, etc. All the docs and samples will be updated soon.
As for now, you can check `a spec
<https://github.com/openstack/rally/blob/0.10.0/doc/specs/implemented/new_rally_input_task_format.rst>`_
for this big change.
* SLA failure_rate max=0 become a default if nothing else is specified.
* Totally reworked atomic actions. The atomic actions now supports nested
actions which allows to measure durations inside the scenario even more
precise. You can find them in HTML report or in our new json report
(see ``rally task report --json``).
* Generation of names for new resources takes care about particular workload
id, so it helps to provide a better cleanup and prepare for new feature -
disaster cleanup.
Plugins
~~~~~~~
We started supporting discovering plugins by entry-points, so you can easily
deliver your custom plugins as a simple python package.
To make you package after-discoverable, you need to specify the proper
entry-point at your setup.cfg file:
.. code-block::
rally_plugins =
path=package_name
**Deployment Engines**:
Remove serverproviders & rarely used deployers
Unfortunately, seems like nobody is using deployers for deploying
their clouds and mostly people would like just to execute their code.
1) Remove server provides
2) Remove engines that uses server providers
**OpenStack clients**:
* Deprecate EC2 client. It wasn't used in any of plugins and doesn't support
keystone v3
* Move ``rally.osclients`` module to ``rally.plugins.openstack.oscliens``
**Scenarios**:
The old way to describe scenario plugin via method is finally removed.
Initially Rally scenario plugins were methods of special class, like below:
.. code-block:: python
from rally.task import scenario
class SomeBasicClass(scenario.Scenario):
@scenario.configure()
def some_scenario(self, arg1):
"""An implementation of SomeBasicClass.foo scenario."""
@scenario.configure()
def another_scenario(self):
"""Implementation of another scenario, SomeBasicClass.bar."""
However to unify scenarios with other plugins we moved to model where
plugin is class. It was done long time ago.
.. code-block:: python
from rally.task import scenario
@scenario.configure(name="CustomName")
class Some(scenario.Scenario):
def run(self, arg1):
"""An implementation of the scenario."""
We had a bunch of code that was used for transition and backward compatibility
that we finally removed.
* *NEW!!*
- `CinderQos.create_and_get_qos
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cinderqos-create-and-get-qos-scenario>`_
- `CinderQos.create_and_list_qos
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cinderqos-create-and-list-qos-scenario>`_
- `CinderQos.create_and_set_qos
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cinderqos-create-and-set-qos-scenario>`_
- `CinderQos.create_qos_associate_and_disassociate_type
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cinderqos-create-qos-associate-and-disassociate-type-scenario>`_
- `CinderVolumeTypes.create_and_get_volume_type
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-and-get-volume-type-scenario>`_
- `CinderVolumeTypes.create_and_list_volume_types
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-and-list-volume-types-scenario>`_
- `CinderVolumeTypes.create_and_update_encryption_type
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-and-update-encryption-type-scenario>`_
- `CinderVolumeTypes.create_and_update_volume_type
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-and-update-volume-type-scenario>`_
- `CinderVolumeTypes.create_get_and_delete_encryption_type
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-get-and-delete-encryption-type-scenario>`_
- `CinderVolumeTypes.create_volume_type_add_and_list_type_access
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumetypes-create-volume-type-add-and-list-type-access-scenario>`_
- `Dummy.openstack
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#dummy-openstack-scenario>`_
- `GlanceImages.create_and_deactivate_image
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#glanceimages-create-and-deactivate-image-scenario>`_
- `GlanceImages.create_and_download_image
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#glanceimages-create-and-download-image-scenario>`_
- `GlanceImages.create_and_get_image
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#glanceimages-create-and-get-image-scenario>`_
- `GlanceImages.create_and_update_image
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#glanceimages-create-and-update-image-scenario>`_
- `K8sPods.create_pods
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#k8spods-create-pods-scenario>`_
- `K8sPods.create_rcs
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#k8spods-create-rcs-scenario>`_
- `K8sPods.list_pods
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#k8spods-list-pods-scenario>`_
- `ManilaShares.create_and_extend_share
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#manilashares-create-and-extend-share-scenario>`_
- `ManilaShares.create_and_shrink_share
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#manilashares-create-and-shrink-share-scenario>`_
- `ManilaShares.create_share_then_allow_and_deny_access
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#manilashares-create-share-then-allow-and-deny-access-scenario>`_
- `NeutronBGPVPN.create_and_delete_bgpvpns
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-delete-bgpvpns-scenario>`_
- `NeutronBGPVPN.create_and_list_bgpvpns
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-list-bgpvpns-scenario>`_
- `NeutronBGPVPN.create_and_list_networks_associations
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-list-networks-associations-scenario>`_
- `NeutronBGPVPN.create_and_list_routers_associations
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-list-routers-associations-scenario>`_
- `NeutronBGPVPN.create_and_update_bgpvpns
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-and-update-bgpvpns-scenario>`_
- `NeutronBGPVPN.create_bgpvpn_assoc_disassoc_networks
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-bgpvpn-assoc-disassoc-networks-scenario>`_
- `NeutronBGPVPN.create_bgpvpn_assoc_disassoc_routers
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronbgpvpn-create-bgpvpn-assoc-disassoc-routers-scenario>`_
- `NeutronNetworks.create_and_show_ports
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronnetworks-create-and-show-ports-scenario>`_
- `NeutronNetworks.create_and_show_routers
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronnetworks-create-and-show-routers-scenario>`_
- `NeutronNetworks.create_and_show_subnets
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronnetworks-create-and-show-subnets-scenario>`_
- `NeutronNetworks.set_and_clear_router_gateway
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronnetworks-set-and-clear-router-gateway-scenario>`_
- `NeutronSecurityGroup.create_and_delete_security_group_rule
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronsecuritygroup-create-and-delete-security-group-rule-scenario>`_
- `NeutronSecurityGroup.create_and_list_security_group_rules
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronsecuritygroup-create-and-list-security-group-rules-scenario>`_
- `NeutronSecurityGroup.create_and_show_security_group
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronsecuritygroup-create-and-show-security-group-scenario>`_
- `NeutronSecurityGroup.create_and_show_security_group_rule
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#neutronsecuritygroup-create-and-show-security-group-rule-scenario>`_
- `NovaServerGroups.create_and_delete_server_group
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservergroups-create-and-delete-server-group-scenario>`_
- `NovaServerGroups.create_and_get_server_group
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservergroups-create-and-get-server-group-scenario>`_
- `NovaServers.boot_and_get_console_url
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-and-get-console-url-scenario>`_
- `NovaServers.boot_server_and_attach_interface
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-server-and-attach-interface-scenario>`_
- `NovaServers.boot_server_and_list_interfaces
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-server-and-list-interfaces-scenario>`_
- `NovaServers.boot_server_attach_volume_and_list_attachments
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-server-attach-volume-and-list-attachments-scenario>`_
* *UPDATED!!*
- The new argument ``properties`` is added to scenario
`IronicNodes.create_and_list_node
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#ironicnodes-create-and-list-node-scenario>`_
* *DELETED*
Fuel and Nova-Network are not alive any more. So we removed those scenarios.
If any of those scenarios a critical for you, please contact us.
- `FuelEnvironments.create_and_delete_environment
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#fuelenvironments-create-and-delete-environment-scenario>`_
- `FuelEnvironments.create_and_list_environments
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#fuelenvironments-create-and-list-environments-scenario>`_
- `FuelNodes.add_and_remove_node
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#fuelnodes-add-and-remove-node-scenario>`_
- `NovaFloatingIpsBulk.create_and_delete_floating_ips_bulk
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novafloatingipsbulk-create-and-delete-floating-ips-bulk-scenario>`_
- `NovaFloatingIpsBulk.create_and_list_floating_ips_bulk
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novafloatingipsbulk-create-and-list-floating-ips-bulk-scenario>`_
- `NovaNetworks.create_and_delete_network
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novanetworks-create-and-delete-network-scenario>`_
- `NovaNetworks.create_and_list_networks
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novanetworks-create-and-list-networks-scenario>`_
- `NovaSecGroup.boot_and_delete_server_with_secgroups
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-boot-and-delete-server-with-secgroups-scenario>`_
- `NovaSecGroup.boot_server_and_add_secgroups
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-boot-server-and-add-secgroups-scenario>`_
- `NovaSecGroup.create_and_delete_secgroups
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-create-and-delete-secgroups-scenario>`_
- `NovaSecGroup.create_and_list_secgroups
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-create-and-list-secgroups-scenario>`_
- `NovaSecGroup.create_and_update_secgroups
<https://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novasecgroup-create-and-update-secgroups-scenario>`_
**Validators**:
The validators refactoring was a long-term task which blocked us to abandon
alignment to only OpenStack platform and requirements of setting admin
credentials. In this release, we made a great progress and fixed a lot of
issues and blockers which made possible to refactor validators.
Now validation step is equal for all types of plugins (Scenario, SLA, Context,
Hooks, Runners, etc).
The old way to add validator for scenario is deprecated. The new unified way
looks like:
.. code-block:: python
import time
from rally.common import validation
from rally.task import scenario
@validation.add("number", param_name="timeout", minval=0)
@scenario.configure(name="Foo.bar")
class FooScenario(scenario.Scenario):
def run(self, timeout):
time.sleep()
The old validators from ``rally.task.validators`` module is deprecated too, see
equivalents which can be used with ``add`` decorator:
- required_openstack --> `required_platform
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-platform-validator>`_
with setting platform argument to "openstack"
- external_network_exists ->`external_network_exists
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#external-network-exists-validator>`_
- file_exists ->`file_exists
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#file-exists-validator>`_
- flavor_exists ->`flavor_exists
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#flavor-exists-validator>`_
- image_exists ->`image_exists
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#image-exists-validator>`_
- image_valid_on_flavor ->`image_valid_on_flavor
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#image-valid-on-flavor-validator>`_
- number ->`number
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#number-validator>`_
- required_api_versions ->`required_api_versions
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-api-versions-validator>`_
- required_cinder_services ->`required_cinder_services
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-cinder-services-validator>`_
- required_clients ->`required_clients
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-clients-validator>`_
- required_contexts ->`required_contexts
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-contexts-validator>`_
- required_neutron_extensions ->`required_neutron_extensions
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-neutron-extensions-validator>`_
- required_param_or_context ->`required_param_or_context
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-param-or-context-validator>`_
- required_services ->`required_services
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#required-services-validator>`_
- restricted_parameters ->`restricted_parameters
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#restricted-parameters-validator>`_
- validate_heat_template ->`validate_heat_template
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#validate-heat-template-validator>`_
- volume_type_exists ->`volume_type_exists
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#volume-type-exists-validator>`_
- workbook_contains_workflow -> `workbook_contains_workflow
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#workbook-contains-workflow-validator>`_
- network_exists is removed, since we do not find any customers for it.
Please contact us if it was useful for you.
- validate_share_proto is removed in favor of enum validator
Fixed bugs
~~~~~~~~~~
* [plugins] JSON schema of `servers
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#servers-context>`_
context allows to transmit a list of nics in two formats. First one is a
format that novaclient expects to see (each network should be represented
like ``{"nic-id": "the id of the network"}``). The second one is more
user-friendly - just list of strings (each network should be represented
just by id of the network). Unfortunately, the second case was not covered
by our code base.
Also, the first described format works in limited cases due to bad
serialization.
`Launchpad bug-report #1695245
<https://bugs.launchpad.net/rally/+bug/1695245>`_
* [deployment] ~/rally/.openrc not working for keystone v3
`Launchpad bug-report #1683820
<https://bugs.launchpad.net/rally/+bug/1683820>`_
* [plugins] Failed to list volumes in case of missed name in the object.
* [backported into 0.9.1][deployment] Credentials is not updated as soon as
deployment is recreated. Need to call recreate request twice.
`Launchpad bug-report #1675271
<https://bugs.launchpad.net/rally/+bug/1675271>`_
* [backported into 0.9.1][plugins] Scenario `IronicNodes.create_and_list_node
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#ironicnodes-create-and-list-node-scenario>`_
had a wrong check that list of all nodes contains newly created one.
* [backported into 0.9.1][task][cleanup] Do not remove quotas in case of
existing users
* [backported into 0.9.1][task][cleanup] Various traces of neutron resources
* [backported into 0.9.1][core] Keystone v3, authentication error for Rally
users if the value of project_domain_name of admin user isn't equal "default"
`Launchpad bug-report #1680837
<https://bugs.launchpad.net/rally/+bug/1680837>`_
* [backported into 0.9.1][task] Scenario `NovaHosts.list_and_get_hosts
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novahosts-list-and-get-hosts-scenario>`_
obtains hostname for all hosts. But it fails in some environments if host is
not compute.
`Launchpad bug-report #1675254
<https://bugs.launchpad.net/rally/+bug/1675254>`_
* [backported into 0.9.1][verification] Rally fails to run on systems on which
python-virtualenv is not installed
`Launchpad bug-report #1678047
<https://bugs.launchpad.net/rally/+bug/1678047>`_
* [backported into 0.9.1][verification] CLI `rally verify rerun
<https://rally.readthedocs.io/en/0.9.1/verification/cli_reference.html#rally-verify-rerun>`_
fails with TypeError due to wrong integration with Rally API.
* [plugins] Rally fails while creating neutron router on the clouds where
ext-gw-mode extension is not installed.
* [plugins] Scenario `CinderVolumes.create_nested_snapshots_and_attach_volume
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#cindervolumes-create-nested-snapshots-and-attach-volume-scenario>`_
fails on a big load due to the fact that one server is used for several
iterations. In such case we are facing 2 issues: the maximum number of
volumes per VM is 26 (which is a number of available names for volumes);
detaching volume of one iteration can block attaching of other iterations.
`Launchpad bug-report #1708160
<https://bugs.launchpad.net/rally/+bug/1708160>`_
Thanks
~~~~~~
2 Everybody!

47
doc/release_notes/archive/v0.10.1.rst
View File

@ -1,47 +0,0 @@
=============
Rally v0.10.1
=============
Overview
--------
+------------------+-----------------------+
| Release date | **12/05/2017** |
+------------------+-----------------------+
Details
-------
This release is fully backward compatible with Rally 0.10.0 and contains just
bug fixes.
Fixed bugs
~~~~~~~~~~
* [deployment] Suppress deprecation warning about an old format in case of
using `--fromenv option of rally deployment create
<https://rally.readthedocs.io/en/0.10.1/cli_reference.html#deployment-create-fromenv>`_
* [deployment] Failure `rally deployment show
<https://rally.readthedocs.io/en/0.10.1/cli_reference.html#rally-deployment-show>`_
while displaying the information about deployment with a config in an old
format.
* [task] New json report processed the hook results in a wrong way
`Launchpad bug-report #1734336
<https://bugs.launchpad.net/rally/+bug/1734336>`_
* [task] Failure while generating trends reports in case of failures in setup
method of any context
`Launchpad bug-report #1732193
<https://bugs.launchpad.net/rally/+bug/1732193>`_
* [task] Failure to export results in ElasticSearch 5.x cluster in case of
extra ``/`` in the end of destination url.
Thanks
~~~~~~
2 Everybody!

263
doc/release_notes/archive/v0.11.0.rst
View File

@ -1,263 +0,0 @@
=============
Rally v0.11.0
=============
Overview
--------
+------------------+-----------------------+
| Release date | **02/16/2018** |
+------------------+-----------------------+
* Stabilize Rally releases (see requirements section)
* Publishing docker images is returned!
* Environment introduction (see Deployment section)
Details
-------
Requirements
~~~~~~~~~~~~
As for long time we tried to make our releases stable and workable even after
a year from release date. For this purpose, upper limits for all our
requirements were used. Such solution helped to make releases more stable, but
it did not work well and created more pain than a profit.
The main issue was related to new releases of not-direct rally
dependencies which can be incompatible which each other.
From Rally 0.11.0 we are stopping adding upper version limits for our
`requirements
<https://github.com/openstack/rally/blob/0.11.0/requirements.txt>`_ (this does
not apply to cases when we sure that some new releases of dependency broke us
for sure).
As alternative solution, the `upper-constraints file
<https://github.com/openstack/rally/blob/0.11.0/upper-constraints.txt>`_ is
selected. It is a file with a list of packages and their versions which we used
in our CI while testing. Versions from this list are suggested to use in
production.
Please note, that it also contains not direct Rally dependencies (dependencies
of rally dependencies and dependencies of dependencies of rally dependencies
as well) and packages which possibly doesn't relate to Rally at all.
The example ou usage:
.. console:: bash
$ git clone https://github.com/openstack/rally && cd rally
$ pip install . --constraint upper-constraints.txt
Logging
~~~~~~~
The default value of ``use_stderr`` option of Rally config is changed to
**True**. It is done to ensure that json output of rally commands will not be
messed with logging and integration with third-party tools and scripts should
become simpler.
Docker
~~~~~~
Unfortunately, we lost access to `rallyforge organization at DockerHub
<https://hub.docker.com/r/rallyforge/>`_, so our docker images were not
published anywhere officially for some time. Docker Support did not help us a
lot. At least, the original repo is removed now and we can start new
organization at DockerHub from the scratch.
The new docker images for Rally+OpenStack plugins with an updated HowTo you
can find at `xrally/xrally-openstack repo
<https://hub.docker.com/r/xrally/xrally-openstack>`_. It contains all Rally
releases + latest tag which maps to master branch.
Command Line Interface
~~~~~~~~~~~~~~~~~~~~~~
* Introduce `rally db ensure
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#rally-db-ensure>`_
command. It is going to create Rally DB if it doesn't exist, otherwise it
does nothing.
* Various improvements for `rally db
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#category-db>`_.
Such as printing results of operations as well as connection string to DB,
hiding credentials by default, etc.
* Various changes in returning error codes. Error codes become different for
different exceptions. Also, `rally task start
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#rally-task-start>`_
began to return 1 in case of any runtime issues and 2 if any workload doesn't
pass it's SLA.
* The new CLI for new component (see ``Environments & Deployments`` section
for more details) - `rally env
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#category-env>`_
Environments & Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~
Deployment Component is fundamental part of Rally which is used by Task and
Verification components. Whereas Task and Verification components experienced
redesigns (some parts were redesigned even several times), Deployment component
was only extended over the time.
Currently, we are at the point when further extending requires much work and
user experience become worth. It is a hard decision, as like we had done with
Verification component in Rally 0.8, the Deployment component is re-written
from the scratch. To be clear, the new version of the component has not only
the new code, but the new name as well - it is the Environment.
We are at the stage when Rally is suitable not only for OpenStack but for
various platforms and systems. The Environment is some entity against which
Rally should perform load. The specific Environment can include credentials for
OpenStack or for Kubernetes or for both. The Environment is a way to describe
the complex system/cloud which all of us have. As well it can be used for
simple systems as easy as for complex.
If you are regular Rally user which tests OpenStack clusters, nothing is
changing for you at this moment. `rally deployment
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#category-deployment>`_
CLI still works and it is not even deprecated. It will be kept for a while.
But you can start looking at our new CLI `rally env
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#category-env>`_ .
As for writing plugins for external platforms - we are working on updating our
documentary.
Task component
~~~~~~~~~~~~~~
* The json results are extended with context execution stats (i.e when and
which context was executed, how much time setup and cleanup methods took,
etc). Also, `volumes@openstack
<https://rally.readthedocs.io/en/0.11.0/plugins/plugin_reference.html#volumes-context>`_
and
`volume_types@openstack
<https://rally.readthedocs.io/en/0.11.0/plugins/plugin_reference.html#volume-types-context>`_
are ported to include detailed stats of executed actions. In further
releases, all existing contexts will be adopted to the similar behavior.
* Better OSProfiler integration
Rally environment&deployment config began accept
``profiler_conn_str`` property which is used to generate OSProfiler native
html-report and to embed it to Rally's html-report.
* HTML report is extended to include a timestamp of failures.
Plugins
~~~~~~~
As it mentioned above, the Deployment Component will be replaced soon. Such
decision led to abandoning all deployment-related plugins (engines, providers,
etc).
**Scenarios**:
* *NEW!!*
`NovaServers.boot_server_attach_volume_and_list_attachments
<https://rally.readthedocs.io/en/0.10.0/plugins/plugin_reference.html#novaservers-boot-server-attach-volume-and-list-attachments-scenario>`_
* *UPDATED!!*
- Extend several Nova&Neutron related scenarios with
``create_floating_ip_args`` parameter
`NovaServers.boot_and_associate_floating_ip
<https://rally.readthedocs.io/en/0.11.0/plugins/plugin_reference.html#novaservers-boot-and-associate-floating-ip-scenario>`_
`NovaServers.boot_server_associate_and_dissociate_floating_ip
<https://rally.readthedocs.io/en/0.11.0/plugins/plugin_reference.html#novaservers-boot-server-associate-and-dissociate-floating-ip-scenario>`_
- Modify Bgpvpn scenarios to test true bgpvpn
All Bgpvpn scenarios began to boot a server to add active port in the
network associated to the bgpvpn which allows to test not only the record in
the database, but true bgpvpn
**Contexts**:
*UPDATED!!*
`network@openstack
<https://rally.readthedocs.io/en/latest/plugins/plugin_reference.html#network-context>`_
context is extended with ability to specify external router information.
Fixed bugs
~~~~~~~~~~
* [backported into 0.10.1][deployment] Suppress deprecation warning about an
old format in case of using `--fromenv option of rally deployment create
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#deployment-create-fromenv>`_
* [backported into 0.10.1][deployment] Failure `rally deployment show
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#deployment-show>`_
while displaying the information about deployment with a config in an old
format.
* [backported into 0.10.1][task] New json report processed the hook results in
a wrong way
`Launchpad bug-report #1734336
<https://bugs.launchpad.net/rally/+bug/1734336>`_
* [backported into 0.10.1][task] Failure while generating trends reports in
case of failures in setup method of any context
`Launchpad bug-report #1732193
<https://bugs.launchpad.net/rally/+bug/1732193>`_
* [backported into 0.10.1][task] Failure to export results in ElasticSearch 5.x
cluster in case of extra ``/`` in the end of destination url.
* [deployment] OpenStack deployment creation with `--fromenv
<https://rally.readthedocs.io/en/0.11.0/cli_reference.html#deployment-create-fromenv>`_
option used old deprecated format.
* [verify] Rally did not support creating verifiers from Gerrit/SSH source.
`Launchpad bug-report #1737529
<https://bugs.launchpad.net/rally/+bug/1737529>`_
* [task][openstack] Removing default security group in users@openstack context
did not take into account that neutron can return multiple resources in some
configuration instead of one security group which relates to requested
tenant.
* [task][openstack] Existing openstack users get their roles un-assigned if
they are equal to what roles@openstack context is configured to assign.
`Launchpad bug-report #1720270
<https://bugs.launchpad.net/rally/+bug/1720270>`_
* [task][openstack] Validation step ignores roles@openstack context and
marks as "invalid" valid cases
Some actions in openstack can be performed only if the user has specific
role. Since these roles can be different in different OpenStack installations
Rally has `roles@openstack context
<https://rally.readthedocs.io/en/0.11.0/plugins/plugin_reference.html#roles-context>`_
context which can assign roles to the users.
Validation step did not check for specified roles in workload config and made
wrong assumption about accessibility of resources
`Launchpad bug-report #1539878
<https://bugs.launchpad.net/rally/+bug/1539878>`_
* [task][openstack] Wrong identifiers were used for filtering Mistral resources
while cleanup step.
* [task][openstack] `NovaServers.boot_and_live_migrate_server
<https://rally.readthedocs.io/en/0.11.0/plugins/plugin_reference.html#novaservers-boot-and-live-migrate-server-scenario>`_
does wrong target host selection
`Launchpad bug-report #1734914
<https://bugs.launchpad.net/rally/+bug/1734914>`_
Thanks
~~~~~~
2 Everybody!

165
doc/release_notes/archive/v0.2.0.rst
View File

@ -1,165 +0,0 @@
============
Rally v0.2.0
============
Information
-----------
+------------------+-----------------------+
| Commits | **48** |
+------------------+-----------------------+
| Bug fixes | *6** |
+------------------+-----------------------+
| Dev cycle | **19 days** |
+------------------+-----------------------+
| Release date | **1/11/2015** |
+------------------+-----------------------+
Details
-------
This release, as well as all previous ones, includes a lot of internal and
external changes. Most important of them are listed below.
.. warning:: Release 0.2.0 doesn't support python 26
Deprecations
~~~~~~~~~~~~
* Option --system-wide-install for ``rally verify start`` was deprecated in
favor of --system-wide
* `rally show` commands were deprecated because of 3 reasons:
- It blocks us to make Rally generic testing tool
- It complicates work on Rally as a Service
- You can always use standard OpenStack clients to do the same
Rally Verify
~~~~~~~~~~~~
* Add "xfail" mechanism for Tempest tests.
This mechanism allows us to list some tests, that are expected to fail,
in a YAML file and these tests will have "xfail" status instead of "fail".
Use new argument "--xfails-file" of rally verify start command.
Rally Task
~~~~~~~~~~
* --out argument of `rally task report` is optional now
If you don't specify --out <file> it will just print the resulting report
* Better scenario output support
As far as you know each scenario plugin are able to return data as a dict.
This dict contained set of key-values {<name>: <float>} where each name
was line on graph and each number was one of point. Each scenario run adds
a single point for each line on that graph.
This allows to add extra data to the Rally and see how some values were
changed over time. However, in case when Rally was used to execute some other
tool and collect it's data this was useless.
To address this **Scenario.add_output(additive, complete)** was introduced:
Now it is possible to generate as many as you need graphs by calling this
method multiple times.
There are two types of graph additive and complete. **Additive** is the same
as legacy concept of output data which is generated from results of all
iterations, **complete** are used when you would like to return whole chart
from each iteration.
HTML report has proper sub-tabs *Aggregated* and *Per iteration*
inside *Scenario Data* tab.
Here is a simple example how output can be added in any
scenario plugin:
.. code-block:: python
# This represents a single X point in result StackedArea.
# Values from other X points are taken from other iterations.
self.add_output(additive={"title": "How do A and B changes",
"description": ("Trend for A and B "
"during the scenario run"),
"chart_plugin": "StackedArea",
"data": [["foo", 42], ["bar", 24]]})
# This is a complete Pie chart that belongs to this concrete iteration
self.add_output(
complete={"title": "",
"description": ("Complete results for Foo and Bar "
"from this iteration"),
"chart_plugin": "Pie",
"data": [["foo", 42], ["bar", 24]]})
Rally Certification
~~~~~~~~~~~~~~~~~~~
None.
Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~
[Spec][Implemented] improve_scenario_output_format
https://github.com/openstack/rally/blob/master/doc/specs/implemented/improve_scenario_output_format.rst
Plugins
~~~~~~~
* **Scenarios**:
* [new] DesignateBasic.create_and_update_domain
* [improved] CinderVolumes.create_and_attach_volume
.. warning:: Use "create_vm_params" dict argument instead of ``**kwargs``
for instance parameters.
* **Context**:
* [improved] images
.. warning:: The min_ram and min_disk arguments in favor of image_args,
which lets the user specify any image creation keyword arguments they want.
Bug fixes
~~~~~~~~~
**6 bugs were fixed**:
* #1522935: CinderVolumes.create_and_attach_volume does not accept additional
args for create_volume
* #1530770: "rally verify" fails with error 'TempestResourcesContext' object
has no attribute 'generate_random_name'
* #1530075: cirros_img_url in rally.conf doesn't take effective in
verification tempest
* #1517839: Make CONF.set_override with parameter enforce_type=True by default
* #1489059: "db type could not be determined" running py34
* #1262123: Horizon is unreachable outside VM when we are using DevStack +
OpenStack
Documentation
~~~~~~~~~~~~~
None.
Thanks
~~~~~~
2 Everybody!

188
doc/release_notes/archive/v0.3.0.rst
View File

@ -1,188 +0,0 @@
============
Rally v0.3.0
============
Information
-----------
+------------------+-----------------------+
| Commits | **69** |
+------------------+-----------------------+
| Bug fixes | **7** |
+------------------+-----------------------+
| Dev cycle | **29 days** |
+------------------+-----------------------+
| Release date | **2/16/2016** |
+------------------+-----------------------+
Details
-------
This release, as well as all previous ones, includes a lot of internal and
external changes. Most important of them are listed below.
.. warning:: In this release Rally DB schema migration is introduced. While
upgrading Rally from previous versions it is required now to run
``rally-manade db upgrade``. Please see 'Documentation' section for details.
CLI changes
~~~~~~~~~~~
* .. warning:: [Removed] ``rally info`` in favor of ``rally plugin *``.
It was deprecated for a long time.
* [Modified] ``rally deployment check`` now prints services, which don't have
names, since such services can be used via api_versions context.
* .. warning:: [Modified] ``rally verify [re]install``
option --no-tempest-venv was deprecated in favor of --system-wide
* [Added] ``rally-manage db revision`` displays current revision of
Rally database schema
* [Added] ``rally-manage db upgrade`` upgrades pre-existing Rally
database schema to the latest revision
* [Added] ``rally-manage db downgrade`` to downgrades existing Rally
database schema to previous revision
* [Added] ``rally task export`` exports task results to external
services (only CLI command introduced, no real service support
implemented yet, however one could write own plugins)
* [Added] ``rally verify export`` exports verification results to
external services (only CLI command introduced, no real service support
implemented yet, however one could write own plugins)
Rally Deployment
~~~~~~~~~~~~~~~~
* .. warning:: ``fuel`` deployment engine is removed since it was outdated and
lacked both usage and support
Rally Task
~~~~~~~~~~
Add custom labels for "Scenario Output" charts
* X-axis label can be specified to add_output() by
"axis_label" key of chart options dict.
The key is named "axis_label" but not "x_label"
because chart can be displayed as table, so we explicitly
mention "axis" in option name to make this parameter
useless for tables
* Y-axis label can be specified to add_output() by
"label" key of chart options dict
In some cases this parameter can be used for rendering
tables - it becomes column name in case if chart with
single iteration is transformed into table
* As mentioned above, if we have output chart
with single iteration, then it is transformed to table,
because chart with single value is useless
* OutputLinesChart is added, it is displayed by
NVD3 lineChart()
* Chart "description" is optional now. Description is
not shown if it is not specified explicitly
* Scenario Dummy.add_output is improved to display labels
and OutputLinesChart
* Fix: If Y-values are too long and overlaps chart box,
then JavaScript updates chart width in runtime to fit
width of chart graphs + Y values to their DOM container
Rally Certification
~~~~~~~~~~~~~~~~~~~
None.
Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~
* [Spec][Introduced] Export task and verification results to external services
https://github.com/openstack/rally/blob/master/doc/specs/in-progress/task_and_verification_export.rst
* [Spec][Implemented] Consistent resource names
https://github.com/openstack/rally/blob/master/doc/specs/implemented/consistent_resource_names.rst
* [Feature request][Implemented] Tempest concurrency
https://github.com/openstack/rally/blob/master/doc/feature_request/implemented/add_possibility_to_specify_concurrency_for_tempest.rst
Plugins
~~~~~~~
* **Scenarios**:
- [added] VMTasks.workload_heat
- [added] NovaFlavors.list_flavors
- [updated] Flavors for Master and Worker node groups are now
configured separately for SaharaCluster.* scenarios
* **Context**:
- .. warning:: [deprecated] rally.plugins.openstack.context.cleanup
in favor of rally.plugins.openstack.cleanup
- [improved] sahara_cluster
Flavors for Master and Worker node groups are now
configured separately in ``sahara_cluster`` context
Miscellaneous
~~~~~~~~~~~~~
* Cinder version 2 is used by default
* Keystone API v3 compatibility improved
- Auth URL in both formats http://foo.rally:5000/v3
and http://foo.rally:5000 is supported for Keystone API v3
- Tempest configuration file is created properly according
to Keystone API version used
* ``install_rally.sh --branch`` now accepts all git tree-ish,
not just branches or tags
* VM console logs are now printed when Rally fails to connect to VM
* Add support for Rally database schema migration (see 'Documentation' section)
Bug fixes
~~~~~~~~~
**7 bugs were fixed**:
* #1540563: Rally is incompatible with liberty Neutron client
The root cause is that in Neutron Liberty client,
the _fx function doesn't take any explicit keyword parameter
but Rally is passing one (tenant_id).
* #1543414: The `rally verify start` command fails when running
a verification against Kilo OpenStack
* #1538341: Error in logic to retrieve image details in image_valid_on_flavor
Documentation
~~~~~~~~~~~~~
* Add documentation for DB migration
https://github.com/openstack/rally/blob/master/rally/common/db/sqlalchemy/migrations/README.rst
Thanks
~~~~~~
2 Everybody!

47
doc/release_notes/archive/v0.3.1.rst
View File

@ -1,47 +0,0 @@
============
Rally v0.3.1
============
Information
-----------
+------------------+-----------------------+
| Commits | **9** |
+------------------+-----------------------+
| Bug fixes | **6** |
+------------------+-----------------------+
| Dev cycle | **2 days** |
+------------------+-----------------------+
| Release date | **2/18/2016** |
+------------------+-----------------------+
Details
-------
This release is more about bug-fixes than features.
.. warning:: Please, update 0.3.0 to latest one.
Features
~~~~~~~~
* Pass api_versions info to glance images context
* [Verify] Don't create new flavor when flavor already exists
Bug fixes
~~~~~~~~~
**6 bugs were fixed, the most critical are**:
* #1545889: Existing deployment with given endpoint doesn't work anymore
* #1547092: Insecure doesn't work with Rally 0.3.0
* #1547083: Rally Cleanup failed with api_versions context in 0.3.0 release
* #1544839: Job gate-rally-dsvm-zaqar-zaqar fails since the recent Rally patch
* #1544522: Non-existing "called_once_with" method of Mock library is used

216
doc/release_notes/archive/v0.3.2.rst
View File

@ -1,216 +0,0 @@
============
Rally v0.3.2
============
Information
-----------
+------------------+-----------------------+
| Commits | **55** |
+------------------+-----------------------+
| Dev cycle | **25 days** |
+------------------+-----------------------+
| Release date | **3/14/2016** |
+------------------+-----------------------+
Details
-------
This release, as well as all previous ones, includes a lot of internal and
external changes. Most important of them are listed below.
CLI changes
~~~~~~~~~~~
* .. warning:: [Modified] Option '--tempest-config' for 'rally verify
reinstall' command was deprecated for removal.
* .. warning:: [Removed] Option `--system-wide-install` was removed from
`rally verify` commands in favor of `--system-wide` option.
* .. warning:: [Modified] Step of installation of Tempest during execution of
the `rally verify start` command was deprecated and will be removed in the
future. Please use `rally verify install` instead.
* Rework commands.task.TaskCommands.detailed. Now output of the command
contains the same results as in HTML report.
Rally Verify
~~~~~~~~~~~~
* Re-run failed Tempest tests
Add the ability to re-run the tempest tests that failed in the last test
execution. Sometimes Tempest tests fail due to a special temporary condition
in the environment, in such cases it is very useful to be able to re-execute
those tests.
Running the following command will re-run all the test that failed during
the last test execution regardless of what test suite was run.
.. code-block:: bash
rally verify start --failing
Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~
* `[Spec][Introduced] Refactoring scenario utils`__
__ https://github.com/openstack/rally/blob/master/doc/specs/in-progress/refactor_scenario_utils.rst
* `[Spec] Deployment unification`__
__ https://github.com/openstack/rally/blob/master/doc/specs/in-progress/deployment_type.rst
Plugins
~~~~~~~
* **Scenarios**:
* [updated] Fix flavor for cloudera manager
Cloudera manager need master-node flavor
* [added] Add more Nova API scenarios
Add support for listing nova hosts, agents, availability-zones
and aggregates.
* [updated] Make sure VolumeGenerator uses the api version info while cleanup
* Designate V2 - Add recordset scenarios
Add create_and_(list|delete)_recordset scenarios
Remove the test also that checks the allowed methods, this is in order for
us to be able to have a private method _walk_pages that will do fetching of
pages for us vs attempting to fetch 1 giant list at once.
* unify \*_kwargs name in scenarios
When running a scenario, `kwargs` is used as default key-word arguments.
But in some scenarios, there are more and one services being called, and
we use xxx_kwargs for this case.
However, some xxx_kwargs are not unified for same usage[0]. Unifying
these could avoid misleading for end users. Another improvement is to
add xxx_kwargs with empty settings for scenario config files.
[0] http://paste.openstack.org/show/489505/
* .. warning:: Deprecated arguments 'script' and 'interpreter' were removed
in favor of 'command' argument.
VM task scenarios executes a script with a interpreter provided through a
formatted argument called 'command' which expects the remote_path or
local_path of the script and optionally an interpreter with which the
script has to be executed.
Miscellaneous
~~~~~~~~~~~~~
* Avoid using `len(x)` to check if x is empty
This cases are using `len()` to check if collection has items. As
collections have a boolean representation too, directly check for true /
false. And fix the wrong mock in its unit test.
* Fix install_rally.sh to get it to work on MacOSX
On MacOSX, `mktemp` requires being passed a template. This change modifies
the calls to `mktemp` to explicitly pass a template so that the code works
on both MacOSX and linux.
* Use new-style Python classes
There are some classes in the code that didn't inherited from
nothing and this is an old-style classes. A "New Class" is the
recommended way to create a class in modern Python.A "New Class"
should always inherit from `object` or another new-style class.
Hacking rule added as well.
* Make Rally cope with unversioned keystone URL
With the change, the client version that's returned is now determined by
the keystoneclient library itself based on whether you supply a URL with a
version in it or not.
* Fix rally-mos job to work with mos-8.0
Also remove hardcoded values for some other jobs.
* Add name() to ResourceManager
This will allow us to perform cleanup based on the name.
* Add task_id argument to name_matches_object
This will be used to ensure that we are only deleting resources for a
particular Rally task.
* Extend api.Task.get_detailed
Extend api.Task.get_detailed with ability to return task data as dict with
extended results.
Bug fixes
~~~~~~~~~
**The most critical fixed bugs are**:
* #1547624: Wrong configuration for baremetal(ironic) tempest tests
* #1536800: openrc values are not quoted
The openrc file created after rally deployment --fromenv did not quote the
values for environment variables that will be exported.
* #1509027: Heat delete_stack never exits if status is DELETE_FAILED
* #1540545: Refactored atomic action in authenticate scenario
* #1469897: Incompatible with Keystone v3 argument in service create scenario
* #1550262: Different results in ``rally task detailed``, ``rally task report``
and ``rally task status`` commands.
* #1553024: Backward incompatible change in neutronclient(release 4.1.0) broke
Tempest config generation to support latest neutronclient.
Documentation
~~~~~~~~~~~~~
* Add documentation for DB migration
* Make documentation for output plugins
* Add descriptive docstrings for plugins based on OutputChart
* Register these plugins in `Rally Plugins Reference`__
__ https://docs.openstack.org/rally/latest/plugins/plugin_reference.html
* Documentation tox fix
Added information about debugging unit test with tox. Replace 3 references
to py26 with py34 to reflect current rally tox configuration.
* Change structure of rally plugin and plugin references page
* Update the scenario development, runner and context sections
* The design of `Rally Plugins Reference`__ page was improved
__ https://docs.openstack.org/rally/latest/plugins/plugin_reference.html
* New page was added - `CLI references`__
__ https://docs.openstack.org/rally/latest/cli_reference.html
Thanks
~~~~~~
To Everybody!

112
doc/release_notes/archive/v0.3.3.rst
View File

@ -1,112 +0,0 @@
============
Rally v0.3.3
============
Information
-----------
+------------------+-----------------------+
| Commits | **20** |
+------------------+-----------------------+
| Dev cycle | **10 days** |
+------------------+-----------------------+
| Release date | **3/24/2016** |
+------------------+-----------------------+
Details
-------
A half of patches relate to Cleanup. We have once again proved that ideal
stuff can be improved. :)
Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~
* `[Spec][Introduced] Improve atomic actions format`__
__ https://github.com/openstack/rally/blob/master/doc/specs/in-progress/improve_atomic_actions_format.rst
Plugins
~~~~~~~
* **Cleanups**:
* Use proper attribute to get heat stack name
* Always assign a name to created images.
This is necessary for name-based cleanup. If a name is not specified, one
will be generated automatically.
* Improve filtering glance images in case of V2 API
* Delete only images created by images context
Since the images context allows creating images with arbitrary names,
name-based cleanup won't work for it, so we have to delete the exact list
of images that it created instead.
* New config option to set cleanup threads
Allow the user to change the number of cleanup threads via the rally
config. When scaling out to thousands of instances, the cleanup can take
forever with the static 20 threads.
* Add inexact matching to name_matches_object
This will support places where we create resources with names that start
with a given name pattern, but include some additional identifier
afterwards. For instance, when bulk creating instances, Nova appends a UUID
to each instance name.
* **Scenarios**:
* Add sample of template for testing for testing heat caching.
* Introduced new scenario `Dummy.dummy_random_action`__. It is suitable for
demonstration of upcoming trends report.
__ http://rally.readthedocs.org/en/latest/plugin/plugin_reference.html#dummy-dummy-random-action-scenario
* **Contexts**:
`api_versions`__ context was extended to support switch between Keystone V2
and V3 API versions. Now it is possible to use one Rally deployment to check
both Keystone APIs.
__ http://rally.readthedocs.org/en/latest/plugin/plugin_reference.html#api-versions-context
* **Newcomer in the family**:
All ResourceType classes are pluggable now and it is much easier to use and
extend them.
.. warning:: Decorator ``rally.task.types.set`` is deprecated now in favor of
``rally.task.types.convert``.
Bug fixes
~~~~~~~~~
* #1536172: rally deployment destroy failed with traceback for failed
deployments. At current moment it is impossible to delete deployment if for
some reason deployment engine plugin cannot be found, because exception will
be thrown.
Documentation
~~~~~~~~~~~~~
* Remove extra link in `All release notes`
Previously, two links for latest release were presented.
* Update release notes for 0.3.2
* Fixed indents for warning messages
* Fixed all references
Thanks
~~~~~~
To Everybody!

286
doc/release_notes/archive/v0.4.0.rst
View File

@ -1,286 +0,0 @@
============
Rally v0.4.0
============
Information
-----------
+------------------+-----------------------+
| Commits | **76** |
+------------------+-----------------------+
| Bug fixes | **12** |
+------------------+-----------------------+
| Dev cycle | **28 days** |
+------------------+-----------------------+
| Release date | **4/18/2016** |
+------------------+-----------------------+
Details
-------
.. warning:: Rally DB schema was changed since previous release.
See `HOWTO <http://rally.readthedocs.org/en/latest/db_migrations.html>`_
about updating your database.
CLI changes
~~~~~~~~~~~
* Add status messages of db migration process
* Display task errors in human-friendly form
* Support OS_PROJECT_NAME as well as OS_TENANT_NAME
Messages
~~~~~~~~
* Removed deprecation warning in case of transmitted "name" attribute while
creation neutron resources.
.. warning:: Deprecated code was deleted.
* Suppress warning insecure URL messages
Do not spam end users by insecure URL messages because it is
quite valid case in testing process
Database
~~~~~~~~
While preparing for deployment refactoring:
* db schema was changed;
* migration with new column `credentials` to deployment model was added;
* columns `users` and `admin` were dropped.
Rally Task
~~~~~~~~~~
* Remove deprecated scenario output mechanism via returing value
.. warning:: Deprecated code was deleted.
* Friendlier error message with empty task file
This is particularly useful when a Jinja2 template results in an empty
task. The current error message isn't very helpful:
Task config is invalid: `'NoneType' object has no attribute 'get'`
* Add Heat template validator
Plugins
~~~~~~~
**Scenarios**:
* Extend VM bind actions with "pause_unpause", "suspend_resume", "lock_unlock",
"shelve_unshelve".
* Add exact error message into `VMTasks.runcommand_heat scenario`__
__ http://rally.readthedocs.org/en/0.4.0/plugin/plugin_reference.html#vmtasks-runcommand-heat-scenario
* Add heat scenarios: output-show, output-list
Current patch contains 4 scenarios from heat repo:
- `output-show for old algorithm
<http://rally.readthedocs.org/en/0.4.0/plugin/plugin_reference.html#heatstacks-create-stack-and-list-output-via-api-scenario>`_
- `output-show for new algorithm
<http://rally.readthedocs.org/en/0.4.0/plugin/plugin_reference.html#heatstacks-create-stack-and-show-output-scenario>`_
- `output-list for old algorithm
<http://rally.readthedocs.org/en/0.4.0/plugin/plugin_reference.html#heatstacks-create-stack-and-list-output-via-api-scenario>`_
- `output-list for new algorithm
<http://rally.readthedocs.org/en/0.4.0/plugin/plugin_reference.html#heatstacks-create-stack-and-list-output-scenario>`_
**Contexts**:
* Reduce default speed of users creation in users context from 30 to 20
by default.
**SLAs**:
* *NEW!!* MaxAverageDurationPerAtomic : Maximum average duration of one
iterations atomic actions in seconds.
`Plugin Reference <http://rally.readthedocs.org/en/0.4.0/plugin/plugin_reference.html#max-avg-duration-per-atomic-sla>`_
**Reports**:
* Improve results calculation in charts.Table
* Use int instead of float for Y axis. It's number of parallel iterations and
it can't be float.
* Remove accuracy that makes no sense, and creates a lot of noise on this graph
* Include failed iterations as well, otherwise we will calculate load
incorrectly
* Graph should start from 0 (begging of experiment)
* Add 2 points at the end of graph to get at the end of graph 0 iterations
in parallel
**Task Exporter**:
In previous release we introduced new mechanism to export results in various
external systems and various formats.
In this release, we added first plugin for this stuff - `file_exporter`
**Services**:
Remove hardcoded timeout from heat service
**Utils**:
Make glance web uploads streamable
Without this change entire file get's downloaded into memory and can cause
issues.
Rally Verify
~~~~~~~~~~~~
* Set time precision to 3 digits (instead of 5) after dot.
* Don't use "--parallel" flag when concurrency == 1
If concurrency equals to 1, it means that we use only one thread to run
Tempest tests and the "--parallel" flag is not needed.
Plugin for DevStack
~~~~~~~~~~~~~~~~~~~
* Support to be enabled with different plugin name
Allow rally to be installed by devstack through a different plugin
name, e.g:
.. code-block:: bash
enable_plugin test-rally http://github.com/rally/rally.git master
* Removed uncalled code
Devstack won't "source plugin.sh source" any more.
Bug fixes
~~~~~~~~~
**12 bugs were fixed**:
* X-Fail mechanism did not work for TestCase which failed on setUp step
If Tempest fails in a test's setUpClass(), there is only one subunit event
for each TestCase. In this case, Rally did not check partial test with x-fail
list and marked test as "fail" insted of "x-fail".
`Launchpad bug-report #1568133
<https://bugs.launchpad.net/rally/+bug/1568133>`_
* Weak isolation of scenario arguments between iterations
Input arguments for sub-task were shared between all iterations. Rally team
found one scenario which modified mutable input variable.
Affected scenario: NeutronNetworks.create_and_update_ports
* Incompatible filters between V1 and V2 for Glance images listing
Glance V1 and V2 have different filters. For example, "owner" is a separate
kwarg in V1, not a generic filter. Also, visibility has different labels in
different APIs. We modified our Glance wrapper to support Glance V2 format
of filters for both V1 and V2
* Wrong way to store validation errors
Results of failed task validations saved in incorrect format. It broke and
made un-userfriendly `rally task detailed` command.
`Launchpad bug-report #1562713
<https://bugs.launchpad.net/rally/+bug/1562713>`_
* Hardcoded task's status in `rally task results`
If there are no results for task, `rally task results` printed message that
task has failed status, but it can be not true(tasks in running state do not
have results).
`Launchpad bug-report #1539096
<https://bugs.launchpad.net/rally/+bug/1539096>`_
* Tempest context failed to create network resources
While we merged improvement for keystoneclient, we used wrong way to obtain
tenant id in TempestContext.
`Launchpad bug-report #1550848
<https://bugs.launchpad.net/rally/+bug/1550848>`_
* Tasks based on Tempest failed to parse execution time.
There is an ability in Rally to launch tasks based on Tempest. Since launch
of Tempest is just subprocess, it is needed to parse subunit to set correct
atomic actions.
There was an issue while converting task execution time.
`Launchpad bug-report #1566712
<https://bugs.launchpad.net/rally/+bug/1566712>`_
* JSONSchema huge impact on task performance
Before runner sent data to engine we were checking jsonschema. This operation
is very expensive and in some cases it can take a lot of time.
Here are test results, with Dummy.dummy_output scenario, sleep 0.5s
(added manually), 8000 iterations, 400 in parallel:
* on master branch before the fix:
Load duration: 117.659588099
Full duration: 227.451056004
* on master before the fix but remove jsonschema validation in scenario:
Load duration: 12.5437350273
Full duration: 128.942219973
* on this patch before the fix (pure python validation):
Load duration: 11.5991640091
Full duration: 22.7199981213
* Wrong Calculation of running iterations in parallel
Load profile chart was calculated wrongly.
It showed more running iterations in parallel than actually are running.
* Rally did not show "missing argument" error raised by argparse while parsing
cli args
`Launchpad bug-report #1562916
<https://bugs.launchpad.net/rally/+bug/1562916>`_
* Issue while checking required arguments in CLI
There was a possible issue in case of several required arguments
`Launchpad bug-report #1555764
<https://bugs.launchpad.net/rally/+bug/1555764>`_
* Prepare step of verification did not check visibility of obtained image
When we request a list of images to choose one of them for tests, we should
make sure all images are active and they are PUBLIC. If images are not
public, we will have failures of Tempest tests as described in the bug.
`Launchpad bug-report #1564431
<https://bugs.launchpad.net/rally/+bug/1564431>`_
Thanks
~~~~~~
2 Everybody!

410
doc/release_notes/archive/v0.5.0.rst
View File

@ -1,410 +0,0 @@
============
Rally v0.5.0
============
Information
-----------
+------------------+-----------------------+
| Commits | **175** |
+------------------+-----------------------+
| Bug fixes | **19** |
+------------------+-----------------------+
| Dev cycle | **93 days** |
+------------------+-----------------------+
| Release date | **7/20/2016** |
+------------------+-----------------------+
Details
-------
This release took much more time than we expected, but we have a lot of
reasons for such delay and if you look at our change-log, you will understand
them.:)
Here is a quick introduction:
* To make our releases as much as possible stable, we added upper limits for
each of our requirements;
* A lot of deprecated lines of code were removed, so be careful;
* Statistics trends for given tasks were introduced;
* Support for tempest plugins was added;
* Several new pages at docs.
Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~
* `[Introduced && implemented] Introduce class-based scenario implementation <https://github.com/openstack/rally/blob/0.5.0/doc/specs/implemented/class-based-scenarios.rst>`_
* `[Introduced] Rally Task Validation refactoring <https://github.com/openstack/rally/blob/0.5.0/doc/specs/in-progress/pluggable_validators.rst>`_
* `[Introduced] Scaling & Refactoring Rally DB <https://github.com/openstack/rally/blob/0.5.0/doc/specs/in-progress/db_refactoring.rst>`_
* `[Introduced] SLA Performance degradation plugin <https://github.com/openstack/rally/blob/0.5.0/doc/specs/in-progress/sla_pd_plugin.rst>`_
Logging
~~~~~~~
* disable urllib3 warnings only if the library provide them
Database
~~~~~~~~
[doesn't require migration]
Transform DB layer to return dicts, not SQLAlchemy models
Rally Deployment
~~~~~~~~~~~~~~~~
* Support single-AZ deployment
This supports the case where OpenStack is deployed with a single AZ for both
controller(s) and compute(s), and not all hosts in the AZ that contains an
instance are guaranteed to have the nova-compute service.
* Extend creation from environment with several new vars
- OS_ENDPOINT_TYPE/OS_INTERFACE
- OS_USER_DOMAIN_NAME
- OS_PROJECT_DOMAIN_NAME
* Improve devstack plugin for Keystone V3
Rally Task
~~~~~~~~~~
*NEW!!* Statistics trends for given tasks.
Rally Verify
~~~~~~~~~~~~
* Remove '--tempest-config' arg from 'reinstall' command
.. warning:: Using `--tempest-config` is became an error from this release.
Use `rally verify genconfig` cmd for all config related stuff.
* Don't install Tempest when `rally verify start`
.. warning:: Use should use `rally verify install` cmd to install tempest now
* Add ability to setup version of Tempest to install
`CLI argument to setup version <http://rally.readthedocs.io/en/0.5.0/cli/cli_reference.html#verify-install-version>`_
* Configure 'aodh' service in 'service_available' section
* Check existence of Tempest-tree in `rally verify discover` cmd
* Make Tempest work with auth url which doesn't include keystone version
Tempest needs /v2.0 and /v3 at the end of URLs. Actually, we can't fix
Tempest, so we extend our configuration module with workaround which allow
to specify auth_url without version in rally deployment config.
* Use default list of plugins for sahara
* Move tempest related options of rally configuration to separate section.
* *NEW!!* Support for tempest plugins.
`CLI argument to install them <http://rally.readthedocs.io/en/0.5.0/cli/cli_reference.html#verify-installplugin>`_
Plugins
~~~~~~~
In this release we are happy to introduce new entity - plugins Base classes
We have a lot of base plugin entities: Context, Scenario, SLA and etc.
Sometimes plugins of different bases can have equal names(i.e ceilometer
OSClient and ceilometer Context). It is normal and we should allow such
conflicts. To support such cases we introduced new entity - plugin base.
Statements of plugin bases:
- Each plugin base is unique entity;
- Names of plugin bases can't conflict with each other;
- Names of two or more plugins in one plugin base can't conflict with each
other(in case of same platform).
- Names of two or more plugins in different plugin base can conflict
Current list of plugin bases:
- rally.task.context.Context
- rally.task.scenario.Scenario
- rally.task.types.ResourceType
- rally.task.exporter.TaskExporter
- rally.task.processing.charts.Chart
- rally.task.runner.ScenarioRunner
- rally.task.sla.SLA
- rally.deployment.serverprovider.provider.ProviderFactory
- rally.deployment.engine.Engine
- rally.osclients.OSClient
**OSClients**
* *NEW!!* Support for Senlin client
* *NEW!!* Support for Gnocchi client
* *NEW!!* Support for Magnum client
* *NEW!!* Support for Watcher client
* Transmit endpoint_type to saharaclient
**Scenarios**:
* *NEW!!*:
- `Authenticate.validate_ceilometer <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#authenticate-validate-ceilometer-scenario>`_
- `CinderVolumes.create_volume_from_snapshot <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#cindervolumes-create-volume-from-snapshot>`_
- `CinderVolumes.create_volume_and_clone <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#cindervolumes-create-volume-and-clone>`_
- `NovaFlavors.create_and_list_flavor_access <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#novaflavors-create-and-list-flavor-access-scenario>`_
- `NovaFlavors.create_flavor <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#novaflavors-create-flavor-scenario>`_
- `NovaServers.boot_and_update_server <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#novaservers-boot-and-update-server>`_
- `NovaServers.boot_server_from_volume_snapshot <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#novaservers-boot-server-from-volume-snapshot>`_
* [Sahara] Add configs to MapR plugin
* Extend CinderVolumes.create_and_upload_volume_to_image with "image" argument
`Plugin Reference <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#cindervolumes-create-and-upload-volume-to-image-scenario>`_
* Deprecate Dummy.dummy_with_scenario_output scenario in favor of Dummy.dummy_output
.. warning:: Dummy.dummy_with_scenario_output scenario will be removed after
several releases
`Deprecated Plugin Reference <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#dummy-dummy-with-scenario-output-scenario>`_
`New Plugin Reference <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#dummy-dummy-output-scenario>`_
* Extend CinderVolumes.create_volume_and_clone with nested_level
Add nested_level argument for nested cloning volume to new volume
* Extend `CinderVolumes.create_nested_snapshots_and_attach_volume
<http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#cindervolumes-create-nested-snapshots-and-attach-volume-scenario>`_
Two new arguments were added: create_volume_kwargs and create_snapshot_kwargs
.. warning:: All arguments related to snapshot creation should be transmitted
only via create_snapshot_kwargs.
* Introduce new style of scenarios - class based.
`Spec Reference <https://github.com/openstack/rally/blob/0.5.0/doc/specs/implemented/class-based-scenarios.rst>`_
* Improve report for VMTasks.boot_runcommand_delete
* [Sahara] Added 5.5.0 version for cdh-plugin and 1.6.0 version for spark
* Extend boot_server_from_volume_and_delete, boot_server_from_volume,
boot_server_from_volume_and_live_migrate, boot_server_from_volume_snapshot
scenarios of NovaServers class with "volume_type" parameter.
**Contexts**:
* *NEW!!*:
- `Cinder volume_types <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#volume-types-context>`_
- `Murano environments <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#murano-environments-context>`_
- `Heat dataplane <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#heat-dataplane-context>`_
* Use Broker Pattern in Keystone roles context
* Use immutable types for locking context configuration
Since context configuration passed to Context.__init__() was a mutable type
(dict or list), sometimes we had unexpected changes done by unpredictable
code (for example, in wrappers).
* Add possibility to balance usage of users
For the moment all users for tasks were taken randomly and there was no way
to balance them between tasks. It may be very useful when we have difference
between first usage of tenant/user and all consecutive. In this case we get
different load results.
Therefore, "users" context was extended with new config option
'user_choice_method' that defines approach for picking up users.
Two values are available:
- random
- round_robin
Default one is compatible with old approach - "random".
* Make sahara_image and custom_image contexts glance v2 compatible
* Extend servers context with "nics" parameter
* Extend network context with "dns_nameservers" parameter
* Extend volume context with "volume_type" parameter
**Cleanup**:
* Mark several cleanup resources as tenant_resource
Nova servers and security groups are tenant related resources, but resource
decorator missed that fact which makes cleanup tries to delete one resources
several times.
* Turn off redundant nova servers cleanup for NovaFlavors.list_flavors scenario
* Add neutron cleanup for NeutronSecurityGroup.create_and_delete_security_groups
**Exporter**:
Rename task-exporter "file-exporter" to "file".
.. warning:: "file-exporter" is deprecated and will be removed in further
releases.
**Types**:
Remove deprecated types.
.. warning:: you should use rally.task.types.convert instead of
rally.task.types.set decorator
**Validators**
* Add a required_api_version validator
* Add validators for scenario arguments
**Utils**:
Use glance wrapper where appropriate to support compatibility between V1 and V2
Bug fixes
~~~~~~~~~
**19 bugs were fixed**:
* Wrong arguments order of Keystone wrapper in case of V2 and V3
* AttributeError while disabling urllib3 warnings on old installations
`Launchpad bug-report #1573650
<https://bugs.launchpad.net/rally/+bug/1573650>`_
* install_rally.sh script is failed while obtaining setuptools
* "-inf" load duration in case of wrong runner plugin and failed start of
contexts
* Strange input task in the report
`Launchpad bug-report #1570328
<https://bugs.launchpad.net/rally/+bug/1570328>`_
* Wrong behaviour of boot_server_from_volume scenarios in case of booting
server from image.
The arg of image must be None, when booting server from volume. Otherwise
still boot server from image.
Affected scenarios:
NovaServers.boot_server_from_volume
NovaServers.boot_server_from_volume_and_delete
NovaServers.boot_server_from_volume_and_resize
NovaServers.boot_server_from_volume_and_live_migrate
`Launchpad bug-report #1578556
<https://bugs.launchpad.net/rally/+bug/1578556>`_
* Weak validation of json schema of RPS runner
JSON Schema of RPS runner doesn't have "required" field. It means that
users are able to pass wrong configs and we will have runtime error while
running task.
* Rally doesn't take cacert setting while creating keystone session
`Launchpad bug-report #1577360
<https://bugs.launchpad.net/rally/+bug/1577360>`_
* Heat scenarios fail when API uses TLS
`Launchpad bug-report #1585456
<https://bugs.launchpad.net/rally/+bug/1585456>`_
* Example in comment of context manila_share_networks wrong
`Launchpad bug-report #1587164
<https://bugs.launchpad.net/rally/+bug/1587164>`_
* There is no way to get UUID of a verification after it is created by
"rally verify start" or "rally verify import_results" when --no-use is set
`Launchpad bug-report #1587034
<https://bugs.launchpad.net/rally/+bug/1587034>`_
* Exposed ssh timeout and interval in vm scenario
`Launchpad bug-report #1587728
<https://bugs.launchpad.net/rally/+bug/1587728>`_
* Ceilometer scenario doesn't require "ceilometer" ctx
`Launchpad bug-report #1557642
<https://bugs.launchpad.net/rally/+bug/1557642>`_
* "servers" context requires setting network id for multiple possible networks
found.
`Launchpad bug-report #1592292
<https://bugs.launchpad.net/rally/+bug/1592292>`_
* nested_level data type incorrect in create_nested_snapshots_and_attach_volume
`Launchpad bug-report #1594656
<https://bugs.launchpad.net/rally/+bug/1594656>`_
* Rally cleanup servers raises exception
`Launchpad bug-report #1584104
<https://bugs.launchpad.net/rally/+bug/1584104>`_
* Stopping server is redundant before cold-migrating server
`Launchpad bug-report #1594730
<https://bugs.launchpad.net/rally/+bug/1594730>`_
* existing_users context doesn't work in case of Keystone v3
* Whether validates flavor's disk or not depends on booting type of the instance
`Launchpad bug-report #1596756
<https://bugs.launchpad.net/rally/+bug/1596756>`_
Documentation
~~~~~~~~~~~~~
* Re-use openstack theme for building docs outside rtd.
`Rally Docs at docs.openstack.org
<https://docs.openstack.org/rally/latest/>`_
* Add page for Verification component
`RTD page for Verification component
<http://rally.readthedocs.io/en/0.5.0/tutorial/step_10_verifying_cloud_via_tempest.html>`_
* Add glossary page
`RTD page for Glossary
<http://rally.readthedocs.io/en/0.5.0/tutorial/glossary.html>`_
* Adjust docs reference to "KeystoneBasic.authenticate" scenario
`Step 6. Aborting load generation on success criteria failure
<http://rally.readthedocs.io/en/0.5.0/tutorial/step_6_aborting_load_generation_on_sla_failure.html>`_
Thanks
~~~~~~
2 Everybody!

198
doc/release_notes/archive/v0.6.0.rst
View File

@ -1,198 +0,0 @@
============
Rally v0.6.0
============
Overview
--------
+------------------+-----------------------+
| Release date | **9/05/2016** |
+------------------+-----------------------+
Details
-------
Common
~~~~~~
* Added Python 3.5 support
* Sync requirements with OpenStack global-requirements
* Start using latest way of authentication - keystoneauth library
* Start porting all scenario plugins to class-based view.
Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~
* `[Implemented] SLA Performance degradation plugin <https://github.com/openstack/rally/blob/0.6.0/doc/specs/implemented/sla_pd_plugin.rst>`_
* `[Proposed] New Tasks Configuration section - hook <https://github.com/openstack/rally/blob/0.6.0/doc/specs/in-progress/hook_section.rst>`_
Database
~~~~~~~~
* disable db downgrade api
* [require migration] upgrade deployment config
Docker image
~~~~~~~~~~~~
* Add sudo rights to rally user
Rally is a pluggable framework. External plugins can require installation of
additional python or system packages, so we decided to add sudo rights.
* Move from ubuntu:14.04 base image to ubuntu:16.04 .
Ubuntu 16.04 is current/latest LTS release. Let's use it.
* pre-install vim
Since there are a lot of users who like to experiment and modify samples
inside container, rally team decided to pre-install vim
* configure/pre-install bash-completion
Rally provides bash-completion script, but it doesn't work without installed
`bash-completion` package and now it is included in our image.
Rally Deployment
~~~~~~~~~~~~~~~~
* Add strict jsonschema validation for ExistingCloud deployments. All incorrect
and unexpected properties will not be ignored anymore. If you need to store
some extra parameters, you can use new "extra" property.
* Fix an issue with endpoint_type.
Previously, endpoint type was not transmitted to keystone client. In this
case, keystoneclient used default endpoint type (for different API calls it
can differ). Behaviour after the fix:
- None endpoint type -> Rally will initialize all clients without setting
endpoint type. It means that clients will choose what default values for
endpoint type use by itself. Most of clients have "public" as default
values. Keystone use "admin" or "internal" by default.
- Not none endpoint type -> Rally will initialize all clients with this
endpoint. Be careful, by default most of keystone v2 api calls do not work
with public endpoint type.
Rally Task
~~~~~~~~~~
* [core] Iterations numbers in logging and reports must be synchronized. Now
they start from 1 .
* [config] users_context.keystone_default_role is a new config option
(Defaults to "member") for setting default user role for new users in case
of Keystone V3.
* [Reports] Embed Rally version into HTML reports
This adds Rally version via meta tag into HTML reports:
<meta name="generator" content="Rally version {{ version }}">
* [Reports] Expand menu if there is only one menu group
* [logging] Remove deprecated rally.common.log module
* [Trends][Reports] Add success rate chart to trends report
* [Reports] Hide menu list if there is no data at all
Rally Verify
~~~~~~~~~~~~
* Updating Tempest config file
- Some tests (for boto, horizon, etc.) were removed from Tempest and now there
is no need to keep the corresponding options in Tempest config file.
- Some options in Tempest were moved from one section to another and we should
to do the corresponding changes in Rally to be up to date with the latest
Tempest version.
* Adding '--skip-list' arg to `rally verify start` cmd
`CLI argument for --skip-list <http://rally.readthedocs.io/en/0.6.0/cli/cli_reference.html#verify-start-skiplist>`_
* *NEW!!*:
- `Command for plugin listing <http://rally.readthedocs.io/en/0.6.0/cli/cli_reference.html#rally-verify-listplugins>`_
- `Command to uninstall plugins <http://rally.readthedocs.io/en/0.6.0/cli/cli_reference.html#rally-verify-uninstallplugin>`_
* Rename and deprecated several arguments for `rally verify start` cmd:
- tests-file -> load-list
- xfails-file -> xfail-list
Plugins
~~~~~~~
**Scenarios**:
* Extend Sahara scenarios with autoconfig param
Affected plugins:
- `SaharaClusters.create_and_delete_cluster <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#saharaclusters-create-and-delete-cluster-scenario>`_
- `SaharaClusters.create_scale_delete_cluster <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#saharaclusters-create-scale-delete-cluster-scenario>`_
- `SaharaNodeGroupTemplates.create_and_list_node_group_templates <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#saharanodegrouptemplates-create-and-list-node-group-templates-scenario>`_
- `SaharaNodeGroupTemplates.create_delete_node_group_templates <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#saharanodegrouptemplates-create-delete-node-group-templates-scenario>`_
* *NEW!!*:
- `MonascaMetrics.list_metrics <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#monascametrics-list-metrics-scenario>`_
- `SenlinClusters.create_and_delete_cluster <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#senlinclusters-create-and-delete-cluster-scenario>`_
- `Watcher.create_audit_template_and_delete <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#watcher-create-audit-template-and-delete-scenario>`_
- `Watcher.create_audit_and_delete <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#watcher-create-audit-and-delete-scenario>`_
- `Watcher.list_audit_templates <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#watcher-list-audit-templates-scenario>`_
* Rename **murano.create_service** to **murano.create_services** atomic action
**SLA**:
*NEW!!*: `performance degradation plugin <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#performance-degradation-sla>`_
**Contexts**:
* *NEW!!*:
- `Monasca monasca_metrics <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#monasca-metrics-context>`_
- `Senlin profiles <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#profiles-context>`_
- `Watcher audit_templates <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#audit-templates-context>`_
* Extend `manila_share_networks <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#manila-share-networks-context>`_
context with share-network autocreation support.
* Extend `volumes <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#volumes-context>`_
context to allow volume_type to be None to allow using default value
Bug fixes
~~~~~~~~~
* [existing users] Quota context does not restore original settings on exit
`Launchpad bug-report #1595578 <https://bugs.launchpad.net/rally/+bug/1595578>`_
* [keystone v3] Rally task's test user role setting failed
`Launchpad bug-report #1595081 <https://bugs.launchpad.net/rally/+bug/1595081>`_
* [existing users] context cannot fetch 'tenant' and 'user' details from cloud
deployment
`Launchpad bug-report #1602157 <https://bugs.launchpad.net/rally/+bug/1602157>`_
* UnboundLocalError: local variable 'cmd' referenced before assignment
`Launchpad bug-report #1587941 <https://bugs.launchpad.net/rally/+bug/1587941>`_
* [Reports] Fix trends report generation if there are n/a results
Documentation
~~~~~~~~~~~~~
* Add page about task reports
`RTD page for reports <http://rally.readthedocs.io/en/0.6.0/reports.html>`_
Thanks
~~~~~~
2 Everybody!

135
doc/release_notes/archive/v0.7.0.rst
View File

@ -1,135 +0,0 @@
============
Rally v0.7.0
============
Overview
--------
+------------------+-----------------------+
| Release date | **10/11/2016** |
+------------------+-----------------------+
Details
-------
Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~
* [Used] Ported all rally scenarios to class base
`Spec reference <https://github.com/openstack/rally/blob/0.7.0/doc/specs/implemented/class-based-scenarios.rst>`_
* `[Implemented] New Plugins Type - Hook <https://github.com/openstack/rally/blob/0.7.0/doc/specs/implemented/hook_plugins.rst>`_
Database
~~~~~~~~
.. warning:: Database schema is changed, you must run
`rally-manage db upgrade <http://rally.readthedocs.io/en/0.7.0/cli/cli_reference.html#rally-manage-db-upgrade>`_
to be able to use old Rally installation with latest release.
* [require migration] fix for wrong format of "verification_log" of tasks
* [require migration] remove admin_domain_name from OpenStack deployments
Rally Deployment
~~~~~~~~~~~~~~~~
* Remove admin_domain_name from openstack deployment
Reason: admin_domain_name parameter is absent in Keystone Credentials.
Rally Task
~~~~~~~~~~
* [Trends][Reports] Use timestamps on X axis in trends report
* [Reports] Add new OutputTextArea chart plugin
New chart plugin can show arbitrary textual data on
"Scenario Stata -> Per iteration" tab.
This finally allows to show non-numeric data like IP addresses, notes and
even long comments.
Plugin `Dummy.dummy_output <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#dummy-dummy-output-scenario>`_
is also updated to provide demonstration.
* [cli] Add version info to *rally task start* output
* [api] Allow to delete stopped tasks without force=True
It is reasonable to protect deletion of running tasks (statuses INIT,
VERIFYING, RUNNING, ABORTING and so on...) but it is strange to protect
deletion for stopped tasks (statuses FAILED and ABORTED). Also this is
annoying in CLI usage.
* Added hooks and triggers.
Hook is a new entity which can be launched on specific events. Trigger is
another new entity which processes events and launches hooks.
For example, hook can launch specific destructive action - just execute cli
command(we have sys_call hook for this task) and it can be launched by
simple trigger on specific iteration(s) or time (there is event trigger).
Rally Verify
~~~~~~~~~~~~
Scenario tests in Tempest require an image file. Logic of obtaining this image
is changed:
* If CONF.tempest.img_name_regex is set, Rally tries to find an image matching
to the regex in Glance and download it for the tests.
* If CONF.tempest.img_name_regex is not set (or Rally didn't find the image
matching to CONF.tempest.img_name_regex), Rally downloads the image by the
link specified in CONF.tempest.img_url.
Plugins
~~~~~~~
**Scenarios**:
* *Removed*: `Dummy.dummy_with_scenario_output <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#dummy-dummy-with-scenario-output-scenario>`_
It was deprecated in 0.5.0
.. warning:: This plugin is not available anymore in 0.7.0
* *NEW!!*:
- `MagnumClusterTemplates.list_cluster_templates <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#magnumclustertemplates-list-cluster-templates-scenario>`_
- `MagnumClusters.list_clusters <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#magnumclusters-list-clusters-scenario>`_
- `MagnumClusters.create_and_list_clusters <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#magnumclusters-create-and-list-clusters-scenario>`_
- `NovaAggregates.create_aggregate_add_and_remove_host <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaaggregates-create-aggregate-add-and-remove-host-scenario>`_
- `NovaAggregates.create_and_list_aggregates <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaaggregates-create-and-list-aggregates-scenario>`_
- `NovaAggregates.create_and_delete_aggregate <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaaggregates-create-and-delete-aggregate-scenario>`_
- `NovaAggregates.create_and_update_aggregate <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaaggregates-create-and-update-aggregate-scenario>`_
- `NovaFlavors.create_and_get_flavor <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaflavors-create-and-get-flavor-scenario>`_
- `NovaFlavors.create_flavor_and_set_keys <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaflavors-create-flavor-and-set-keys-scenario>`_
- `NovaHypervisors.list_and_get_hypervisors <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novahypervisors-list-and-get-hypervisors-scenario>`_
- `NovaServers.boot_server_associate_and_dissociate_floating_ip <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaservers-boot-server-associate-and-dissociate-floating-ip-scenario>`_
- `KeystoneBasic.authenticate_user_and_validate_token <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#keystonebasic-authenticate-user-and-validate-token-scenario>`_
**Contexts**:
* *NEW!!*:
- `Manila manila_security_services <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#manila-security-services-context>`_
- `Magnum cluster_templates <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#cluster-templates-context>`_
- `Magnum clusters <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#clusters-context>`_
**OSClients**:
Port all openstack clients to use keystone session.
Bug fixes
~~~~~~~~~
* [tasks] rally task detailed incorrect / inconsistent output
`Launchpad bug-report #1562713 <https://bugs.launchpad.net/rally/+bug/1562713>`_
Thanks
~~~~~~
2 Everybody!

232
doc/release_notes/archive/v0.8.0.rst
View File

@ -1,232 +0,0 @@
============
Rally v0.8.0
============
Overview
--------
+------------------+-----------------------+
| Release date | **1/25/2017** |
+------------------+-----------------------+
Details
-------
Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~
* `[Implemented] Refactor Verification Component
<https://github.com/openstack/rally/blob/0.8.0/doc/specs/implemented/verification_refactoring.rst>`_
* `[Implemented] Scaling & Refactoring Rally DB
<https://github.com/openstack/rally/blob/0.8.0/doc/specs/implemented/db_refactoring.rst>`_
Installation
~~~~~~~~~~~~
We switched to use bindep library for checking required system packages.
All our dependencies moved to separate file (like requirements.txt for python
packages) `bindep.txt
<https://github.com/openstack/rally/blob/0.8.0/bindep.txt>`_.
Database
~~~~~~~~
.. warning:: Database schema is changed, you must run
`rally-manage db upgrade <http://rally.readthedocs.io/en/0.8.0/cli/cli_reference.html#rally-manage-db-upgrade>`_
to be able to use old Rally installation with latest release.
* change structure of database to be more flexible
* save raw task results in chunks (see raw_result_chunk_size option of
[DEFAULT] rally configuration section)
* add db revision check in rally API, so it is impossible to use rally with
wrong db now.
Rally API
~~~~~~~~~
Single entry point for Rally API is added - rally.api.API . Old API classes
(``rally.api.Task``, ``rally.api.Verification``, ``rally.api.Deployment``) are
deprecated now.
Rally CLI
~~~~~~~~~
* ``rally task sla_check`` is deprecated now in favor of
``rally task sla-check``
* Deprecated category ``rally show`` was removed.
* `rally plugin list` is extended with plugin base column
Task Component
~~~~~~~~~~~~~~
- [Random names] scenario for checking performance of generate_random_name
method is added to our CI with proper SLA. Be sure, whatever number of random
names you need, it will not affect performance of Rally at all, we checked.
- [atomic actions] scenario for checking performance of calculating atomic
actions is added to our CI with proper SLA. Be sure, whatever number atomics
you have in scenarios, it will not affect performance of Rally at all, we
checked.
- [services] new entity is introduced for helping to provide compatibility
layer between different API versions of one service.
Verification component
~~~~~~~~~~~~~~~~~~~~~~
We completely redesign the whole Verification component. For more details see
`our new docs for that component
<http://rally.readthedocs.io/en/0.8.0/verification/index.html>`_
Unfortunately, such big change could not be done in backward compatible way,
so old code is not compatible with new one. See `HowTo migrate from
Verification component 0.7.0 to 0.8.0
<http://rally.readthedocs.io/en/0.8.0/verification/howto/migrate_from_old_design.html>`_
Plugins
~~~~~~~
**Services**:
* Glance:
Switched from V1 to V2 API by default.
* Keystone:
- Transmit endpoint_type to keystoneclient
- Full keystone V3 support
**Scenarios**:
* *Updated*:
- The meaning of the volume_type argument is changes in
`CinderVolumes.create_snapshot_and_attach_volume
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#cindervolumes-create-snapshot-and-attach-volume-scenario>`_
scenario. It should contain actual volume type instead of boolean value to
choose random volume type.
- Extend `GlanceImages.create_image_and_boot_instances
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#glanceimages-create-image-and-boot-instances-scenario>`_
with create_image_kwargs and boot_server_kwargs arguments.
* *NEW!!*:
- `CeilometerAlarms.create_and_get_alarm
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#ceilometeralarms-create-and-get-alarm-scenario>`_
- `CinderVolumeBackups.create_incremental_volume_backup
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#cindervolumebackups-create-incremental-volume-backup-scenario>`_
- `CinderVolumeTypes.create_and_delete_volume_type
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#cindervolumetypes-create-and-delete-volume-type-scenario>`_
- `CinderVolumeTypes.create_volume_type_and_encryption_type
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#cindervolumetypes-create-volume-type-and-encryption-type-scenario>`_
- `CinderVolumes.create_and_accept_transfer
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#cindervolumes-create-and-accept-transfer-scenario>`_
- `CinderVolumes.create_and_get_volume
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#cindervolumes-create-and-get-volume-scenario>`_
- `CinderVolumes.create_volume_and_update_readonly_flag
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#cindervolumes-create-volume-and-update-readonly-flag-scenario>`_
- `CinderVolumes.list_transfers
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#cindervolumes-list-transfers-scenario>`_
- `CinderVolumes.list_types
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#cindervolumes-list-types-scenario>`_
- `KeystoneBasic.create_and_get_role
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#keystonebasic-create-and-get-role-scenario>`_
- `ManilaShares.create_and_list_share
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#manilashares-create-and-list-share-scenario>`_
- `ManilaShares.set_and_delete_metadata
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#manilashares-set-and-delete-metadata-scenario>`_
- `MistralExecutions.create_execution_from_workbook
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#mistralexecutions-create-execution-from-workbook-scenario>`_
- `MistralExecutions.list_executions
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#mistralexecutions-list-executions-scenario>`_
- `NeutronLoadbalancerV2.create_and_list_loadbalancers
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#neutronloadbalancerv2-create-and-list-loadbalancers-scenario>`_
- `NeutronNetworks.create_and_show_network
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#neutronnetworks-create-and-show-network-scenario>`_
- `NeutronNetworks.list_agents
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#neutronnetworks-list-agents-scenario>`_
- `NovaAggregates.create_aggregate_add_host_and_boot_server
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#novaaggregates-create-aggregate-add-host-and-boot-server-scenario>`_
- `NovaAggregates.create_and_get_aggregate_details
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#novaaggregates-create-and-get-aggregate-details-scenario>`_
- `NovaFlavors.create_and_delete_flavor
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#novaflavors-create-and-delete-flavor-scenario>`_
- `NovaFlavors.create_flavor_and_add_tenant_access
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#novaflavors-create-flavor-and-add-tenant-access-scenario>`_
- `NovaHosts.list_and_get_hosts
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#novahosts-list-and-get-hosts-scenario>`_
- `NovaHypervisors.list_and_get_uptime_hypervisors
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#novahypervisors-list-and-get-uptime-hypervisors-scenario>`_
- `NovaHypervisors.list_and_search_hypervisors
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#novahypervisors-list-and-search-hypervisors-scenario>`_
- `NovaHypervisors.statistics_hypervisors
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#novahypervisors-statistics-hypervisors-scenario>`_
- `NovaSecGroup.boot_server_and_add_secgroups
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#novasecgroup-boot-server-and-add-secgroups-scenario>`_
- `NovaServerGroups.create_and_list_server_groups
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#novaservergroups-create-and-list-server-groups-scenario>`_
- `Quotas.nova_get
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#quotas-nova-get-scenario>`_
**Hooks**:
* *NEW!!*:
- `fault_injection
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#fault-injection-hook>`_
**Runners**
* *Updated*:
- `RPS runner
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#rps-scenario-runner>`_
is extended with ability to increase 'rps' value by arithmetic progression
across certain duration. Now it can be also a dict specifying progression
parameters:
.. code-block:: json
rps": {
"start": 1,
"end": 10,
"step": 1,
"duration": 2
}
This will generate rps value: ``start, start + step, start + 2 * step, ..,
end`` across certain 'duration' seconds each step. If iteration count not
ended at the last step of progression, then rps will continue to generate
with "end" value. Note that the last rps could be generated smaller.
Fixed bugs
~~~~~~~~~~
* [hooks] incorrect encoding of stdout/stderr streams opened by sys_call hook
for py3
* [hooks] sorting Hook column at HTML report doesn't work
* [tasks][scenarios][neutron] L3 HA: Unable to complete operation on subnet
`Launchpad bug-report #1562878 <https://bugs.launchpad.net/rally/+bug/1562878>`_
* [tasks] JSON report doesn't save order of atomics
* [tasks][cleanup][nova] Failed to remove aggregate which has hosts in it
* [tasks] `--abort-on-sla-failure
<http://rally.readthedocs.io/en/0.8.0/cli_reference.html#task-start-abortonslafailure>`_
mechanism works only for current workload, but does not stop the next ones.
* [hooks] hooks section isn't displayed in HTML report
Thanks
~~~~~~
2 Everybody!

40
doc/release_notes/archive/v0.8.1.rst
View File

@ -1,40 +0,0 @@
============
Rally v0.8.1
============
Overview
--------
+------------------+-----------------------+
| Release date | **1/27/2017** |
+------------------+-----------------------+
Details
-------
Fix for python requirements list.
Plugins
~~~~~~~
**Scenarios**:
* *Updated*:
- Use new network for each subnet at
`NeutronNetworks.create_and_list_subnets
<http://rally.readthedocs.io/en/0.8.1/plugins/plugin_reference.html#neutronnetworks-create-and-list-subnets-scenario>`_
scenario.
* *NEW!!*:
- `CinderVolumeTypes.create_and_list_encryption_type
<http://rally.readthedocs.io/en/0.8.1/plugins/plugin_reference.html#cindervolumetypes-create-and-list-encryption-type-scenario>`_
- `Quotas.cinder_get
<http://rally.readthedocs.io/en/0.8.1/plugins/plugin_reference.html#quotas-cinder-get-scenario>`_
Thanks
~~~~~~
2 Everybody!

163
doc/release_notes/archive/v0.9.0.rst
View File

@ -1,163 +0,0 @@
============
Rally v0.9.0
============
Overview
--------
+------------------+-----------------------+
| Release date | **3/20/2017** |
+------------------+-----------------------+
Details
-------
Command Line Interface
~~~~~~~~~~~~~~~~~~~~~~
* `rally plugin list` now does not contain hidden plugins.
Task component
~~~~~~~~~~~~~~
* Added check for duplicated keys in task files.
* The order of subtasks (scenarios/workloads) is not ignored any more. You can
generate whatever you want load or use that feature for up the cloud (put
small scenario to the start of task to wake up the cloud before the real
load).
* Information about workload creation is added to HTML-reports.
* Task statuses is changed to be more clear and cover more cases:
- ``verifying`` is renamed to ``validating``.
- ``failed`` is divided for 2 statuses - ``validation_failed``, which means
that task did not pass validation step, and ``crashed``, which means that
something went wrong in rally engine.
* Our awesome cleanup become more awesome! The filter mechanism is improved to
discover resources in projects created only by Rally (it works for most of
resources, except several network-related ). It makes possible to run Rally
with existing users in real tenants without fear to remove something
important.
Verification component
~~~~~~~~~~~~~~~~~~~~~~
* Fixed an issue with missed tests while listing all supported tests of
specified verifier.
* Fixed an issue with displaying the wrong version of verifier in case of
cloning from the local directory.
* Extend `rally verify rerun
<http://rally.readthedocs.io/en/0.9.0/verification/cli_reference.html#rally-verify-rerun>`_
with ``--detailed``, ``--no-use``, ``--tag`` and ``--concurrency`` arguments.
* Add output examples for `JSON
<http://rally.readthedocs.io/en/0.9.0/verification/reports.html#json>`_ and
`JUnit-XML
<http://rally.readthedocs.io/en/0.9.0/verification/reports.html#junit-xml>`_
reporters.
Plugins
~~~~~~~
**Contexts**
* Extend cinder quotas to support ``backups`` and ``backup_gigabytes``.
**Deployment Engines**:
*Updated* Extend `DevstackEngine
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#devstackengine-engine>`_
with ``enable_plugin`` option.
**OpenStack clients**:
* Extend support for auth urls like ``https://example.com:35357/foo/bar/v3``
* Pass endpoint type to heatclient
**Scenarios**:
* *NEW!!*
- `CinderVolumeTypes.create_and_delete_encryption_type
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#cindervolumetypes-create-and-delete-encryption-type-scenario>`_
- `CinderVolumeTypes.create_and_set_volume_type_keys
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#cindervolumetypes-create-and-set-volume-type-keys-scenario>`_
- `KeystoneBasic.create_and_list_roles
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#keystonebasic-create-and-list-roles-scenario>`_
- `KeystoneBasic.create_and_update_user
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#keystonebasic-create-and-update-user-scenario>`_
- `NovaKeypair.create_and_get_keypair
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novakeypair-create-and-get-keypair-scenario>`_
- `NovaServers.resize_shutoff_server
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novaservers-resize-shutoff-server-scenario>`_
- `VMTasks.dd_load_test
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#vmtasks-dd-load-test-scenario>`_
* *UPDATED!!*
- Extend `VMTasks.boot_runcommand_delete
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#vmtasks-boot-runcommand-delete-scenario>`_
to display just raw text output of executed command.
* *DELETED*
Scenario `VMTasks.boot_runcommand_delete_custom_image
<http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#vmtasks-boot-runcommand-delete-custom-image-scenario>`_
is removed since `VMTasks.boot_runcommand_delete
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#vmtasks-boot-runcommand-delete-scenario>`_
covers the case of that particular scenario without adding any complexity.
**Validators**:
* Extend ``required_contexts`` validator to support ``at least one of the``
logic.
* Fix a bunch of JSON schemas which are used for validation of all plugins.
Documentation
~~~~~~~~~~~~~
We totally reworked `Plugins Reference
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html>`_ page.
Now it looks more like `Command Line Interface
<http://rally.readthedocs.io/en/0.9.0/cli_reference.html>`_, which means that
you can get links for particular parameter of particular plugin.
Also, you can find expected parameters and their types of all contexts, hooks,
SLAs and so on! Most of them still miss descriptions, but we are working on
adding them.
Fixed bugs
~~~~~~~~~~
* [osclients] Custom auth mechanism was used for zaqarclient instead of unified
keystonesession, which led to auth errors at some envs.
* [plugins] During running
`CinderVolumes.create_and_restore_volume_backup
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#cindervolumes-create-and-restore-volume-backup-scenario>`_
scenario we had a race problem with backup deleting due to wrong check of
backup status.
* [plugins][verifications] Jenkins expexts "classname" JUnitXML attribute
instead of "class_name".
Thanks
~~~~~~
2 Everybody!

63
doc/release_notes/archive/v0.9.1.rst
View File

@ -1,63 +0,0 @@
============
Rally v0.9.1
============
Overview
--------
+------------------+-----------------------+
| Release date | **4/12/2017** |
+------------------+-----------------------+
Details
-------
Unfortunately, Rally 0.9.0 contains various bugs. We work hard to fix them,
improve our CI to avoid such issues in future and ready to present a new Rally
release which includes only bug-fixes.
Fixed bugs
~~~~~~~~~~
* [deployment] Credentials is not updated as soon as deployment is recreated.
Need to call recreate request twice.
`Launchpad bug-report #1675271
<https://bugs.launchpad.net/rally/+bug/1675271>`_
* [task] Scenario `IronicNodes.create_and_list_node
<https://rally.readthedocs.io/en/0.9.1/plugins/plugin_reference.html#ironicnodes-create-and-list-node-scenario>`_
had a wrong check that list of all nodes contains newly created one.
* [task][cleanup] Do not remove quotas in case of existing users
* [task][cleanup] Various traces of neutron resources
* [core] Keystone v3, authentication error for Rally users if the value of
project_domain_name of admin user isn't equal "default"
`Launchpad bug-report #1680837
<https://bugs.launchpad.net/rally/+bug/1680837>`_
* [task] Scenario `NovaHosts.list_and_get_hosts
<https://rally.readthedocs.io/en/0.9.1/plugins/plugin_reference.html#novahosts-list-and-get-hosts-scenario>`_
obtains hostname for all hosts. But it fails in some environments if host is
not compute.
`Launchpad bug-report #1675254
<https://bugs.launchpad.net/rally/+bug/1675254>`_
* [verification] Rally fails to run on systems on which python-virtualenv is
not installed
`Launchpad bug-report #1678047
<https://bugs.launchpad.net/rally/+bug/1678047>`_
* [verification] CLI `rally verify rerun
<https://rally.readthedocs.io/en/0.9.1/verification/cli_reference.html#rally-verify-rerun>`_
fails with TypeError due to wring integration with Rally API.
Thanks
~~~~~~
2 Everybody!

35
doc/release_notes/archive/v0.9.2.rst
View File

@ -1,35 +0,0 @@
============
Rally v0.9.2
============
Overview
--------
+------------------+-----------------------+
| Release date | **10/20/2017** |
+------------------+-----------------------+
Details
-------
The latest OpenStack merged a bunch of incompatible changes. This release
is an attempt to fix compatibitily issuses in Rally 0.9.
Now it works well for old and new OpenStack releases.
.. note:: OpenStack Nova abandoned networking and image API. It is impossible
to do anything with it, so we suggest you to use Neutron and Glance instead.
Fixed bugs
~~~~~~~~~~
* [broken dependency] One of hook plugins required ansible which released
incompatible version. To fix this, the proper version of os-faults lib should
be used
* [keystone] The format of keystone URL's completely change which resulted in a
wrong processing it from our side.
Thanks
~~~~~~
2 Everybody!

1
doc/release_notes/latest.rst
View File

@ -1 +0,0 @@
archive/v0.11.0.rst