Benchmark engine spawns a consumer thread, which asynchronously handles results
for each scenario run. Current consumer implementation just collects all
results and stores them as 'raw' to save integrity.
Scenario runners have to call _send_result method for every single result
instead of collecting a list of results.
Change-Id: Icbff8b2470af3b3ced6bb55573b5842726177b70
The json file of the raw results is useful if you want to pull
the results into an external system, e.g. to plot the performance
of the gate over time.
Change-Id: I03d6b6f6ab5c5aee3a0f5de0c87eaf393f041d14
In this patch:
* validation function benchmark.validation.required_services()
* a piece of refactoring in scenarios.base and runners.base modules
* new decorator benchmark.validation.validator(),
which transforms validation function into scenario validator
* constants for OpenStack services names and types
* refactoring in rally.osclients
* method osclients.Clients.services() to obtain available services
* services validation is added to scenarios
blueprint check-service-availability-before-start-scenario
Change-Id: Id6126efb1bda73dd9ff92346ddf690329765f43f
Added missing tempest test cases
as part of unit test coverage.
test_single_test_negative
test_set_smoke
test_set_full
test_set_from_list
test_set_selective
test_specific_regex
Change-Id: I89d96ce4142dd72c79fdd21b100addd56ae03188
Implements: blueprint improve-unit-test-coverage-rally
Change the default value of the use_public_urls configuration option of
the ExistingCloud deployment engine to True.
Change-Id: I8ccb19fdef7911c2e27145758014e942086bf954
This patch allows VM runcommand scenario attach a floating ip
to the instance to perform ssh connection.
Blueprint: benchmark-scenarios-vm-floating-ip
Change-Id: I2e77d6be34a64d5feb9d3ecc5e35c8a712b10bed
* The results_dir is created to save log files during tempest run
* Ensure it is created correctly
Change-Id: I440e42ca6f8a8858773a09b0a756a0d05235ba3e
Closes-Bug: #1335122
Change the default value of the scenario_output variable of the
_run_scenario_once function of the rally/benchmark/runners/base.py
module because there was no errors key in this variable when a scenario
failed with timeout.
Closes-Bug: 1336762
Change-Id: Ieebc2064bcfee3f2bd23ee40394f8117e9096f01
Update ICMP allow rule to use the value -1 for to_port and
from_port parameters as ICMP protocol does not have port.
Change-Id: Idc76e156f07b99574b6b77dba803d5a4f823380b
Closes-bug: #1336704
SLA (Service-level agreement) is set of details for determining
compliance with contracted values such as maximum error rate or
minimum response time.
Add two criteria:
maximum time per iteration
maximum error rate
Change-Id: I1212bd684831461a7d7e33636d45a6d346e3b574
Blueprint: task-success-criteria
Implemented a unit test that takes all the scenarios from rally-scenarios and
tests their correctness after loading plugins from rally-scenarios/plugins
Change-Id: I2af0b67216bdee65a413c8037130cab8e65ac826
Some linux distributions have buggy curl versions, which
incorrect hande redirects.
Change-Id: Ice79958096240446955eb92517dfd41ce9432a5f
Closes-Bug: #1335117
Spread the content of doc/samples/tasks directory to separate folders
for context, scenarios and runners.
Add documentation and samples for runners.
Change-Id: I4394e73060c6c97fae7bcf7039674041d43917e8
Implemented a unit test that takes all the config pairs from doc/samples/tasks/
with identical names (e.g. nova/boot-and-delete.json and
nova/boot-and-delete.yaml) and checks that their contents are identical as
well. Also it tests that every config should have a pair (if there is *.json,
there also has to be *.yaml, and vice versa)
Change-Id: I4c0d185439e2a2c48a801fa8c62c7ed2e80a6031
This Context Class implements Image Context Subclass
to Benchmarking Mechanism. This also adds a simple list_image benchmark
scenario which tests glance image-list command
bp benchmark-context-image
Change-Id: I272a36cebb568e8d599ebd65beb53b6a2c628b6e
Here we enhance the rst docs (used for readthedocs):
* Simplify its index structure by reducing it to only 5 points:
1. Deploy (engines & server providers)
2. Verify
3. Benchmark (scenarios, engine, contexts, plugins)
4. Command Line Interface
5. Implementation (all source docs moved here)
* Extend the 'Benchmark' page with info from the "Main concepts" page from Wiki
* Add some introductory text on Deploy & Verify parts based on that from Wiki
* Simplify headings by removing "The" and "Layer"
Change-Id: I04bb553aba04009634272a88a757baa8f2c8c359
- rename a base class from FunctionalTest to PecanControllerTest which
makes this naming more accurate
- remove some print statement for debug uses
- use _request_json in delete method
- use requests module instead of six.moves.urllib
Change-Id: I4463466bab91ad538295668b9e8133a1a1f99caa
Scenario NeutronNetworks.create_and_list_ports is added, which does the following:
- creates network
- creates a given number of ports
- lists ports
blueprint benchmark-scenarios-for-neutron
Change-Id: I029f220dd70beb0c248877bd4bd718dfa63aba40
With PKI, tokens can be validated at the service endpoint. There might
be some caching policies which might affect the validation process.
This benchmark can be used to check effect of caching related to tokens.
Change-Id: I6277d8514bbdb1e5dca5032f44ac3d5ff9618cc2
Sync modelus: service, periodic_task and related dependencies
from oslo-incubator. Using the following command in oslo-incubator:
python update.py --base rally --dest-dir ../rally --modules \
periodic_task,service
Change-Id: I0b8278404d53a91add7a8cf3876121aad1e497c2
* add unit tests for rest.types
* add unit tests for rest.controllers.root
* add unit tests for rest.controllers.v1.root
Change-Id: Icb7d243643a6a0a57d2b1f01879774008491ea0a
Use `requests` library instead of `urllib2` & `httplib`
Main profit from `requests` library:
- `requests` is HTTP for Humans;
- `requests` is well documented[1];
- `requests` can save downloaded files by parts, so usage of memory
is less(useful when downloading big data, for example - some images)
- `requests` has same name as in Python 2x and Python 3x
Use `inspect.getmembers`[2][3] for discovery of each function for section
generator. Profit : decrease lines of code
Add checks for existence of subnet in 'network' section.
Move function `_write_config`[4] from class `Tempest`[5] to `TempestConf`.
Reason: this function not related to verifier for Tempest.
Fix message for exception when user doesn't have admin role.
Move name of section into kwargs to each section function.
Profit: less string objects with section names
[1] - http://docs.python-requests.org/en/latest/
[2] - https://docs.python.org/2/library/inspect.html#types-and-members
[3] - https://docs.python.org/3.3/library/inspect.html#types-and-members
[4] - http://git.io/E1zPOQ
[5] - http://git.io/8Ru-Yw
Change-Id: I8307ec33dd93ef055450f58d7cd55bf6b200f249
* Remove subunit2junitxml dependency and code from rally
* Convert raw subunit stream from tempest tests to json and store only
json in DB
* Run tempest scenario (smoke) in rally gate
Change-Id: I6628375d83da9af3984819dfc5815ae0c06ec1aa
Implements: blueprint tempest-subunit-to-json
