diff --git a/doc/source/index.rst b/doc/source/index.rst index f6c2f60..ae715c7 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -17,6 +17,7 @@ Contributions to this documentation are warmly encouraged; please see use-cases specs eris/index + testing/tools-list Indices and tables diff --git a/doc/source/testing/tools-list.rst b/doc/source/testing/tools-list.rst new file mode 100644 index 0000000..a5e1847 --- /dev/null +++ b/doc/source/testing/tools-list.rst @@ -0,0 +1,246 @@ +*********************** +OpenStack testing tools +*********************** + +With the rapid expansion of OpenStack, a variety of tools have emerged +for testing different aspects of OpenStack. However there is +currently no unified solution that can test availability, reliability, +performance, and scalability of OpenStack infrastructure. + +The aim of this SIG is to create self-healing infrastructure, and in +order to move in to that direction there is a need for an automated +destructive/extreme testing framework for OpenStack infrastructure. +As there have already been several projects tackling various parts of +the challenge, one goal is to avoid duplication of effort and see how +we can take advantage of these already developed tools and come up +with a unified solution. + +Here is list of tools and the testing features that are provided by +those (as of August 2019): + +=================================== ======= ======= =========== =========== =========== ============ ========== ========== ============================================================================================================== + Tools/Features Control Data Performance Scalability Concurrency Disruptions/ OpenStack Report Active community [#]_ + plane plane testing testing testing Failure Deployment generation + testing testing injection +=================================== ======= ======= =========== =========== =========== ============ ========== ========== ============================================================================================================== +Rally_ (Uses Tempest) X X X X X X X `Medium `__ +Yardstick_ X X X X X X X `NA since May 2019 `__ +Tempest_ X X `High `__ +Shaker_ X X X X `Low `__ +OS-Faults_ X `NA since February 2019 `__ +KloudBuster_ X X X X `NA since June 2019 `__ +HA-Failure-Test_ (Uses Tempest) X X `NA since November 2016 `__ +Cloud99_ X X X X `NA since May 2017 `__ +Browbeat_ (Uses Rally and Shaker) X X X X X `High `__ +Harbinger_ X `Low `__ +Enos_ (Uses Rally and Shaker) X X X X X `NA since May 2019 `__ +Chaostoolkit-openstack_ X X `NA since April 2019 `__ +Jepsen_ (not specific to OpenStack) X X `High `__ +OPNFV-Doctor_ X `NA since April 2019 `__ +Stepler_ (Uses OS-faults) X `NA since February 2019 `__ +NFVbench_ X X X `Medium `__ +Eris_ (In development) X X X X X X X `NA since November 2017 `__ +=================================== ======= ======= =========== =========== =========== ============ ========== ========== ============================================================================================================== + +.. [#] Commits per month: Low<5, Medium<15, High=>15, NA=No Activity + +.. _Rally: + +`Rally `__ +=================================================================================== + +- Deployment of OpenStack +- Verification of deployment +- Benchmarking: Simulate real user load +- Profiler for scaling and performance issues +- Report generation: Processing and aggregation of results + +De-facto control plane performance testing, allows to create different +kind of tests including functional, concurrency, regression, load, scale +capacity and chaos testing. + +.. _Yardstick: + +`Yardstick `__ +================================================================== + +- Deployment +- Run compute, storage, network test scenarios using a runner +- Report generation with the following metrics: + + - Performance/Speed + - Capacity/Scale + - Availability/Reliability + +Inspired by Rally, was developed mainly from the prespective of +VNF(Virtual Network Function), but is a generic framework for testing +scenarios in OpenStack for performance, scalability and +availability/reliability. + +.. _Tempest: + +`Tempest `__ +================================================================================================================== + +- Integration testing for live OpenStack +- Test OpenStack publicly available API and scenarios that demonstrate + a working cloud +- Load testing possible as well + +Mainly used for integration testing but non-functional testing +possible as well. + +.. _Shaker: + +`Shaker `__ +================================================================= + +- Deploy instances and network in different topologies or network + configurations using Heat +- Run network related test scenarios +- Report generation + +Network load generation, testing and benchmarking. Based on 'iperf(3)' +and 'netperf' -> network testing topologies + +.. _OS-Faults: + +`OS-Faults `__ +====================================================== + +- Library for initiating or terminating service/node/container/network + as a way for introducing fault injections. +- Allows for both hardware and software level faults + +Used by Rally to introduce faults. + +.. _KloudBuster: + +`KloudBuster `__ +========================================================== + +- Benchmarking of data plane and storage services in OpenStack + +A fully integrated framework which is easy to use with basic knowledge +of OpenStack but isn't flexible or programmable to do everything. + +.. _HA-Failure-Test: + +`HA-Failure-Test `__ +================================================================== + +- High Availability testing of OpenStack components +- Set of Tempest scripts to test HA of components in HA cluster. +- Stop service(s), perform sanity and verify + +.. _Cloud99: + +`Cloud99 `__ +====================================================== + +- Monintoring: Nodes and services +- Disruptions: in services, node, containers can be parallel, + sequential or round robin +- Runner: Perform actual tests with disruptions +- All 3 runs as seperate threads and parallel + +Makes use of Rally runner + +.. _Browbeat: + +`Browbeat `__ +========================================================== + +- Performance and scalability testing +- Create Rally work loads +- Run Shaker for network configuration, benchmarking and testing +- CI and deployment: Can be used for CI or deploying data analysis + tools. + +.. _Harbinger: + +`Harbinger `__ +============================================================================== + +- Manager for data plane testing frameworks in OpenStack +- Can similtaneously execute multiple testing frameworks (Shaker, + Yardstick) + +.. _Enos: + +`Enos `__ +================================================== + +- Deployment of OpenStack on VMs or bare metal +- Setup with different configurations +- Benchmarking: + + - Rally scenario for control plane + - Shaker scenario of data plane + +- Report generation: Get metrics/reports in Grafana + +.. _Chaostoolkit-openstack: + +`chaostoolkit-openstack `__ +============================================================================================= + +- Driver to perform set of actions and probes. + +A minimal chaostoolkit driver for OpenStack compute resources. + +.. _Jepsen: + +`Jepsen `__ +================================================ + +- Introduction of disruptions/faults +- Testing +- Report generation +- Checkers for analyzing history + +Not specific to OpenStack but general library for distributed systems, +written in Clojure + +.. _OPNFV-Doctor: + +`OPNFV Doctor `__ +==================================================================== + +- Fault management and maintenance in VIM(Virtualized Infrastructure + Manager) e.g OpenStack +- Fault monitoring (Hardware, Hypervisor, Host OS faults) +- Fault notification +- Failure recovery (Switch to STBY config.) + +.. _Stepler: + +`Stepler `__ +============================================= + +- Destructive testing +- Test the availability of OpenStack cluster incase of node/service + failure(s) + +Make use of os-faults + +.. _NFVbench: + +`NFVbench `__ +================================================================================================ + +- Measure network performance in OpenStack +- Data plane performance testing + + +.. _Eris: + +`Eris `__ +============================================================================= + +- Control and data plane performance testing +- High Availability testing +- Scalability Testing +- Concurrency Testing +- Benchmarking and results generation +