From 4421b11d2800cd448a274ef986e0eccbf10062c6 Mon Sep 17 00:00:00 2001 From: Ihtisham ul Haq Date: Wed, 7 Aug 2019 17:43:56 +0200 Subject: [PATCH] Add list of tools for Openstack testing Story: 2002129 Task: 35628 Change-Id: I51df92d6ffabf12e96aafb8f6275075a697c82bb --- doc/source/index.rst | 1 + doc/source/testing/tools-list.rst | 234 ++++++++++++++++++++++++++++++ 2 files changed, 235 insertions(+) create mode 100644 doc/source/testing/tools-list.rst diff --git a/doc/source/index.rst b/doc/source/index.rst index f6c2f60..ae715c7 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -17,6 +17,7 @@ Contributions to this documentation are warmly encouraged; please see use-cases specs eris/index + testing/tools-list Indices and tables diff --git a/doc/source/testing/tools-list.rst b/doc/source/testing/tools-list.rst new file mode 100644 index 0000000..e6ec02c --- /dev/null +++ b/doc/source/testing/tools-list.rst @@ -0,0 +1,234 @@ +*********************** +Openstack testing tools +*********************** + +With the rapid expansion of OpenStack, a variety of testing tools have emerged for testing different aspects of OpenStack. However there is currently no unified solution that can test availability, reliability, performance, and scalability of OpenStack infrastructure. + +The aim of this SIG is to create self-healing infrastructure, and in order to move in to that direction there is a need for an automated destructive/extreme testing framework for OpenStack infrastructure. As there have already been several projects tackling various parts of the challenge, one goal is to avoid duplication of effort and see how we can take advantage of these already developed tools and come up with a unified solution. + +Here is list of tools and the testing features that are provided by those (as of August 2019): + +=================================== ===================== ================== =================== =================== =================== ============================= ==================== ================= ============================================================================================================== + Tools/Features Control plane testing Data plane testing Performance testing Scalability testing Concurrency testing Disruptions/Failure injection Openstack Deployment Report generation Active community [#]_ +=================================== ===================== ================== =================== =================== =================== ============================= ==================== ================= ============================================================================================================== +Rally_ (Uses Tempest) X X X X X X X `Medium `__ +Yardstick_ X X X X X X X `NA since May 2019 `__ +Tempest_ X X `High `__ +Shaker_ X X X X `Low `__ +OS-Faults_ X `NA since February 2019 `__ +KloudBuster_ X X X X `NA since June 2019 `__ +HA-Failure-Test_ (Uses Tempest) X X `NA since November 2016 `__ +Cloud99_ X X X X `NA since May 2017 `__ +Browbeat_ (Uses Rally and Shaker) X X X X X `High `__ +Harbinger_ X `Low `__ +Enos_ (Uses Rally and Shaker) X X X X X `NA since May 2019 `__ +Chaostoolkit-openstack_ X X `NA since April 2019 `__ +Jepsen_ (not specific to OpenStack) X X `High `__ +OPNFV-Doctor_ X `NA since April 2019 `__ +Stepler_ (Uses OS-faults) X `NA since February 2019 `__ +NFVbench_ X X X `Medium `__ +Eris_ (In development) X X X X X X X `NA since November 2017 `__ +=================================== ===================== ================== =================== =================== =================== ============================= ==================== ================= ============================================================================================================== + +.. [#] Commits per month: Low<5, Medium<15, High=>15, NA=No Activity + +.. _Rally: + +`Rally `__ +=================================================================================== + +- Deployment of Openstack +- Verification of deployment +- Benchmarking: Simulate real user load +- Profiler for scaling and performance issues +- Report generation: Processing and aggregation of results + +De-facto control plane performance testing, allows to create different +kind of tests including functional, concurrency, regression, load, scale +capacity and chaos testing. + +.. _Yardstick: + +`Yardstick `__ +================================================================== + +- Deployment +- Run compute, storage, network test scenarios using a runner +- Report generation with the following metrics: + + - Performance/Speed + - Capacity/Scale + - Availability/Reliability + +Inspired by Rally, was developed mainly from the prespective of +VNF(Virtual Network Function), but is a generic framework for testing +scenarios in Openstack for performance, scalability and +availability/reliability. + +.. _Tempest: + +`Tempest `__ +================================================================================================================== + +- Integration testing for live OpenStack +- Test OpenStack publicly available API and scenarios that demonstrate + a working cloud +- Load testing possible as well + +Mainly used for integration testing but non-funtional testing possible as +well. + +.. _Shaker: + +`Shaker `__ +================================================================= + +- Deploy instances and network in different topologies or network + configurations using Heat +- Run network related test scenarios +- Report generation + +Network load generation, testing and benchmarking. Based on 'iperf(3)' +and 'netperf' -> network testing topologies + +.. _OS-Faults: + +`OS-Faults `__ +====================================================== + +- Library for initiating or terminating service/node/container/network + as a way for introducing fault injections. +- Allows for both hardware and software level faults + +Used by Rally to introduce faults. + +.. _KloudBuster: + +`KloudBuster `__ +========================================================== + +- Benchmarking of data plane and storage services in Openstack + +A fully integrated framework which is easy to use with basic knowledge +of Openstack but isn't flexible or programmable to do everything. + +.. _HA-Failure-Test: + +`HA-Failure-Test `__ +================================================================== + +- High Availability testing of Openstack components +- Set of Tempest scripts to test HA of components in HA cluster. +- Stop service(s), perform sanity and verify + +.. _Cloud99: + +`Cloud99 `__ +====================================================== + +- Monintoring: Nodes and services +- Disruptions: in services, node, containers can be parallel, + sequential or round robin +- Runner: Perform actual tests with disruptions +- All 3 runs as seperate threads and parallel + +Makes use of Rally runner + +.. _Browbeat: + +`Browbeat `__ +========================================================== + +- Performance and scalability testing +- Create Rally work loads +- Run Shaker for network configuration, benchmarking and testing +- CI and deployment: Can be used for CI or deploying data analysis + tools. + +.. _Harbinger: + +`Harbinger `__ +============================================================================== + +- Manager for data plane testing frameworks in Openstack +- Can similtaneously execute multiple testing frameworks (Shaker, + Yardstick) + +.. _Enos: + +`Enos `__ +================================================== + +- Deployment of Openstack on VMs or bare metal +- Setup with different configurations +- Benchmarking: + + - Rally scenario for control plane + - Shaker scenario of data plane + +- Report generation: Get metrics/reports in Grafana + +.. _Chaostoolkit-openstack: + +`chaostoolkit-openstack `__ +============================================================================================= + +- Driver to perform set of actions and probes. + +A minimal chaostoolkit driver for Openstack compute resources. + +.. _Jepsen: + +`Jepsen `__ +================================================ + +- Introduction of disruptions/faults +- Testing +- Report generation +- Checkers for analyzing history + +Not specific to Openstack but general library for distributed systems, +written in Clojure + +.. _OPNFV-Doctor: + +`OPNFV Doctor `__ +==================================================================== + +- Fault management and maintenance in VIM(Virtualized Infrastructure + Manager) e.g OpenStack +- Fault monitoring (Hardware, Hypervisor, Host OS faults) +- Fault notification +- Failure recovery (Switch to STBY config.) + +.. _Stepler: + +`Stepler `__ +============================================= + +- Destructive testing +- Test the availability of Openstack cluster incase of node/service + failure(s) + +Make use of os-faults + +.. _NFVbench: + +`NFVbench `__ +================================================================================================ + +- Measure network performance in Openstack +- Data plane performance testing + + +.. _Eris: + +`Eris `__ +================================================================================================ + +- Control and data plane performance testing +- High Availability testing +- Scalability Testing +- Concurrency Testing +- Benchmarking and results generation +