Merge "Define performance test suite framework"

Zuul 3 years ago committed by Gerrit Code Review
commit 00ed48e9bc

@ -0,0 +1,268 @@
This work is licensed under a Creative Commons Attribution 3.0 Unported
StarlingX: Performance measurement framework
The measurement of performance metrics in edge cloud systems is a key factor in
quality assurance. Having a common understanding and a practical framework to
run the performance tests are important and necessary for users and developers
in the community. This specification will describe the scope of the performance
framework, use cases and how it can scale by new tests developed by the
community or imported from existing test performance frameworks.
Problem description
The deployment of StarlingX could happen in a wide variability of hardware and
network environments, these differences could result in a difference in
performance metrics. As of today, the community does not have a consolidated
performance framework to measure the performance metrics with specific and
well-defined test cases.
Use Cases
Developers who made some fairly significant changes in codebase want to ensure
no performance regression brought by the changes. Developers who did some
performance improvements need to measure the performance result with a
consolidating framework valid across the community. Some of the community
members want to measure out the performance items on StarlingX and promote
StarlingX advantages in terms of performance which are relevant to an edge
solution. Potential users want to evaluate StarlingX as one of their Edge
solution candidates, so offering such a performance framework will be a
positive factor of contributing to the evaluation process.
Proposed change
The proposed change is to create a set of scripts and documentation under the
starlingx testing repository ( that anyone
can use to measure the metric they need under their hardware and software
The community can use existing performance test frameworks, Some of those are:
* Rally:
OpenStack project, dedicated to the performance analysis and benchmarking
system of individual OpenStack components
* Kubernetes perf-tests:
An open source project dedicated to Kubernetes-related performance test tools
* Yardstick by OPNFV:
The Yardstick concept decomposes typical virtual network function work-load
performance metrics into several characteristics/performance vectors, which
each of them can be represented by distinct test-cases.
The disadvantage of all these frameworks is that they coexist in separate
projects. Users need to go and find the test cases anywhere on the internet, is
not a centralized and scalable solution. The StarlingX Performance measurement
framework proposed will allow the importing of existing test cases to be
re-used in a centralized system easy to use for the community.
Data model impact
REST API impact
Security impact
Other end-user impact
End users will interact with this framework by command line on the controller
system by the script, which is the entry point to
select and configure the test case. An example of to launch a test case might
./ --test failed_vm_detection --compute=0
This will generate results of multiple runs in CSV format easy to post-process.
Performance Impact
Running this performance framework or test cases, there shouldnt be a
performance impact or obvious overhead on the system.
Other deployer impact
None, there will be no impact on how to deploy and configure StarlingX
Developer impact
There will be a good impact on the developers. Since they will be able to have
a consolidated and well aligned framework to measure the performance impact of
their code/configuration changes.
Developers can't improve something if they dont know it needs improvement.
Thats where this performance testing framework comes in. It gives the
developers tools to detect how fast, stable, and scalable their StarlingX, is
so they can decide if it needs to improve or not.
Upgrade impact
The test framework will have the next components on the implementation
* Common scripts:
This directory will contain scripts to capture key timestamps or collect other
key data along with some use cases so that developers can measure out the time
(latency) or other performance data (such as throughput, system CPU utilization
and so on). Any common script that any other test case could use should be in
this directory.
Despite the existence of a directory for common scripts, each one of the test
cases is responsible for its own metrics. One example is the time measurement
of test cases that run in more than one node. For these kinds of test cases
will be necessary to use synchronization protocols like the Network Time
Protocol (NTP). The Network Time Protocol (NTP) is a networking protocol for
clock synchronization between computer systems. NTP is intended to synchronize
all participating computers to within a few milliseconds of Coordinated
Universal Time (UTC).In some other cases, it might not be necessary to have a
protocol for clock synchronization.
Most of the details for wether to re-use a common script form the common
directory or handle the measurement by the test itself will be left to the test
The main command-line interface to execute the test cases on the system under
* Test cases directory:
This directory can contain wrapper scripts to existing upstream performance
test cases as well as new test cases specific for Starlting X.
A view of the directory layout will look like:
├── common
│ ├──
│ ├──
│ ├──
│ └──
└── tests_cases
│ ├──
│ ├──
│ ├──
│ ├──
│ ├──
│ ├──
│ ├──
│ ├──
│ ├──
The goal is that anyone on the StarlingX community can either define
performance tests cases or re use existing on other projects and create
scripts to measure in a scalable and repeatable framework.
In the future, it might be possible to evaluate how to connect some basic
performance tests cases to zuul to detect regressions on commits to be merged.
This will require to set up a robust infrastructure to run the sanity
performance test cases for each commit to be merged.
Wang Hai Tao
Elio Martinez
Juan Carlos Alonzo
Victor Rodriguez
Repos Impacted
Work Items
* Develop automated scripts for core test cases, such as:
* Detection of failed VM
* Detection failed compute node
* Controller/Node fail detect/recovery
* Detection of network link fail
* DPDK Live Migrate Latency
* Avg/Max host/guest Latency (Cyclictest)
* Swact Time
* Develop that call each test case script
* Integrate with pytest framework
* Implement call clone of opnfv test cases
* Implement execution of opnfv test case
* Implement monitoring pipeline to automatically detect changes in upstream
* Automate weekly mail with list of upstream performance test cases that change
* Implement automatic test cases documentation based on pydoc
* Document framework on StarlingX wiki
* Send regular status to ML and demos on community call for feedback
Documentation Impact
The performance test cases will be documented with pydoc
( The pydoc module
automatically generates documentation from Python modules. The documentation
can be presented as pages of text on the console, served to a Web browser, or
saved to HTML files.
The goal is to generate automatically the documentation for the end-user
testing guide. This methodology will catch when the community adds or modify a
new performance test case. At the same time if a standard config option changes
or is deprecated the test framework documentation will be updated to reflect
the change.
.. list-table:: Revisions
:header-rows: 1
* - Release Name
- Description
* - Stein
- Introduced