Add list of tools for Openstack testing

Story: 2002129
Task: 35628

Change-Id: I51df92d6ffabf12e96aafb8f6275075a697c82bb
This commit is contained in:
Ihtisham ul Haq 2019-08-07 17:43:56 +02:00
parent d313c13496
commit 7002d67d4b
2 changed files with 215 additions and 0 deletions

View File

@ -17,6 +17,7 @@ Contributions to this documentation are warmly encouraged; please see
use-cases
specs
eris/index
testing/tools-list
Indices and tables

View File

@ -0,0 +1,214 @@
***********************
Openstack testing tools
***********************
================================= ===================== ================== =================== =================== ============================= ==================== =================
Tools/Features Control plane testing Data plane testing Scalability testing Concurrency testing Disruptions/Failure injection Openstack Deployment Report generation
================================= ===================== ================== =================== =================== ============================= ==================== =================
Rally_ X X X X X X
Yardstick_ X X X X X X
Tempest_ X
Shaker_ X X X
OS-Faults_ X
KloudBuster_ X X X
HA-Failure-Test_ (Uses Tempest) X X
Cloud99_ X X X X
Browbeat_ (Uses Rally and Shaker) X X X X
Harbinger_ X
Enos_ (Uses Rally and Shaker) X X X X
Chaostoolkit-openstack_ X X
Jepsen_ X X
OPNFV-Doctor_ X
Stepler_ (Uses OS-faults) X
NFVbench_ X X
================================= ===================== ================== =================== =================== ============================= ==================== =================
.. _Rally:
`Rally <https://rally.readthedocs.io/en/latest/overview/overview.html#use-cases>`__
===================================================================================
- Deployment of Openstack
- Verification of deployment
- Benchmarking: Simulate real user load
- Profiler for scaling and performance issues
- Report generation: Processing and aggregation of results
De-facto control plane performance testing, allows to create different
kind of tests including functional, concurrency, regression, load, scale
capacity and chaos testing.
.. _Yardstick:
`Yardstick <https://wiki.opnfv.org/display/yardstick/Yardstick>`__
==================================================================
- Deployment
- Run compute, storage, network test scenarios using a runner
- Report generation with the following metrics:
- Performance/Speed
- Capacity/Scale
- Availability/Reliability
Inspired by Rally, was developed mainly from the prespective of
VNF(Virtual Network Function), but is a generic framework for testing
scenarios in Openstack for performance, scalability and
availability/reliability.
.. _Tempest:
`Tempest <https://docs.openstack.org/tempest/latest/overview.html#tempest-the-openstack-integration-test-suite>`__
==================================================================================================================
- Integration testing for live OpenStack
- Test OpenStack publicly available API and scenarios that demonstrate
a working cloud
- Load testing possible as well
Mainly used for functional testing but non-funtional testing possible as
well.
.. _Shaker:
`Shaker <https://pyshaker.readthedocs.io/en/latest/index.html>`__
=================================================================
- Deploy instances and network in different topologies or network
configurations using Heat
- Run network related test scenarios
- Report generation
Network load generation, testing and benchmarking. Based on 'iperf(3)'
and 'netperf' -> network testing topologies
.. _OS-Faults:
`OS-Faults <https://github.com/openstack/os-faults>`__
======================================================
- Library for initiating or terminating service/node/container/network
as a way for introducing fault injections.
- Allows for both hardware and software level faults
Used by Rally to introduce faults.
.. _KloudBuster:
`KloudBuster <https://github.com/openstack/kloudbuster>`__
==========================================================
- Benchmarking of data plane and storage services in Openstack
A fully integrated framework which is easy to use with basic knowledge
of Openstack but isn't flexible or programmable to do everything.
.. _HA-Failure-Test:
`HA-Failure-Test <https://github.com/avdhoot07/HA-Failure-TEST>`__
==================================================================
- High Availability testing of Openstack components
- Set of Tempest scripts to test HA of components in HA cluster.
- Stop service(s), perform sanity and verify
.. _Cloud99:
`Cloud99 <https://github.com/cisco-oss-eng/Cloud99>`__
======================================================
- Monintoring: Nodes and services
- Disruptions: in services, node, containers can be parallel,
sequential or round robin
- Runner: Perform actual tests with disruptions
- All 3 runs as seperate threads and parallel
Makes use of Rally
.. _Browbeat:
`Browbeat <https://github.com/cloud-bulldozer/browbeat>`__
==========================================================
- Create Rally work loads
- Run Shaker for network configuration, benchmarking and testing
- CI and deployment: Can be used for CI or deploying data analysis
tools.
.. _Harbinger:
`Harbinger <https://harbinger-dpm.readthedocs.io/en/latest/what/index.html>`__
==============================================================================
- Manager for data plane testing frameworks in Openstack
- Can similtaneously execute multiple testing frameworks (Shaker,
Yardstick)
.. _Enos:
`Enos <https://github.com/BeyondTheClouds/enos>`__
==================================================
- Deployment Openstack on VMs or bare metal
- Setup with different configurations
- Benchmarking:
- Rally scenario for control plane
- Shaker scenario of data plane
- Report generation: Get metrics/reports in Grafana
.. _Chaostoolkit-openstack:
`chaostoolkit-openstack <https://pypi.org/project/chaostoolkit-openstack/>`__
=============================================================================
- Driver to perform set of actions and probes.
A minimal chaostoolkit driver for Openstack compute resources.
.. _Jepsen:
`Jepsen <https://github.com/jepsen-io/jepsen>`__
================================================
- Introduction of disruptions/faults
- Testing
- Report generation
- Checkers for analyzing history
Not specific to Openstack but general library for distributed systems,
written in Clojure
.. _OPNFV-Doctor:
`OPNFV Doctor <https://wiki.opnfv.org/display/doctor/Doctor+Home>`__
====================================================================
- Fault management and maintenance in VIM(Virtualized Infrastructure
Manager) e.g OpenStack
- Fault monitoring (Hardware, Hypervisor, Host OS faults)
- Fault notification
- Failure recovery (Switch to STBY config.)
.. _Stepler:
`Stepler <https://stepler.readthedocs.io/>`__
=============================================
- Destructive testing
- Test the availability of Openstack cluster incase of node/service
failure(s)
Make use of os-faults
.. _NFVbench:
`NFVbench <https://opnfv-nfvbench.readthedocs.io/en/latest/testing/user/userguide/index.html>`__
================================================================================================
- Measure network performance in Openstack
- Data plane performance testing