Patch I7fc109ee5785f426211fbff7eb9b4553a3671ba7 splitted functional and samples check into 2 separate modules. It was done due to some conflicts while launching tests in parallel. Despite the fact it was a good patch, it had a bug - missed `set -e` called in the launch script. Which results in ignoring results of launching functional tests for more then 2 months. Now we have 1/4 tests fail. This patch starts refactoring CLI job. First of all, we need to stop ignoring the results of functional tests. The broken tests are marked as skipped, so we can enable a part of tests right away. Also, this patch moves test_certification_task.py to the proper dir with samples check. The new tox environment is introduced for launching only functional tests(which should not depend on OpenStack): tox -efunctional Change-Id: I222c13f724e8e70d10d58ca546094c076d73d737
|2 years ago|
|devstack||2 years ago|
|doc||2 years ago|
|etc||2 years ago|
|rally||2 years ago|
|rally-jobs||2 years ago|
|samples||2 years ago|
|tasks||2 years ago|
|tests||2 years ago|
|.coveragerc||4 years ago|
|.dockerignore||4 years ago|
|.gitignore||2 years ago|
|.gitreview||5 years ago|
|CONTRIBUTING.rst||3 years ago|
|Dockerfile||2 years ago|
|LICENSE||7 years ago|
|README.rst||3 years ago|
|bindep.txt||3 years ago|
|requirements.txt||2 years ago|
|setup.cfg||2 years ago|
|setup.py||4 years ago|
|test-requirements.txt||2 years ago|
|tox.ini||2 years ago|
Rally is a Benchmark-as-a-Service project for OpenStack.
Rally is intended to provide the community with a benchmarking tool that is capable of performing specific, complicated and reproducible test cases on real deployment scenarios.
If you are here, you are probably familiar with OpenStack and you also know that it's a really huge ecosystem of cooperative services. When something fails, performs slowly or doesn't scale, it's really hard to answer different questions on "what", "why" and "where" has happened. Another reason why you could be here is that you would like to build an OpenStack CI/CD system that will allow you to improve SLA, performance and stability of OpenStack continuously.
The OpenStack QA team mostly works on CI/CD that ensures that new patches don't break some specific single node installation of OpenStack. On the other hand it's clear that such CI/CD is only an indication and does not cover all cases (e.g. if a cloud works well on a single node installation it doesn't mean that it will continue to do so on a 1k servers installation under high load as well). Rally aims to fix this and help us to answer the question "How does OpenStack work at scale?". To make it possible, we are going to automate and unify all steps that are required for benchmarking OpenStack at scale: multi-node OS deployment, verification, benchmarking & profiling.
Rally workflow can be visualized by the following diagram:
Rally documentation on ReadTheDocs is a perfect place to start learning about Rally. It provides you with an easy and illustrative guidance through this benchmarking tool. For example, check out the Rally step-by-step tutorial that explains, in a series of lessons, how to explore the power of Rally in benchmarking your OpenStack clouds.
In terms of software architecture, Rally is built of 4 main components:
There are 3 major high level Rally Use Cases:
Typical cases where Rally aims to help are: