163 lines
5.5 KiB
ReStructuredText
163 lines
5.5 KiB
ReStructuredText
..
|
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
|
License.
|
|
|
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
|
|
|
..
|
|
This template should be in ReSTructured text. The filename in the git
|
|
repository should match the launchpad URL, for example a URL of
|
|
https://blueprints.launchpad.net/rally/+spec/awesome-thing should be named
|
|
awesome-thing.rst . Please do not delete any of the sections in this
|
|
template. If you have nothing to say for a whole section, just write: None
|
|
For help with syntax, see http://sphinx-doc.org/rest.html
|
|
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
|
|
|
|
=================================
|
|
OSProfiler integration into Rally
|
|
=================================
|
|
|
|
The OSProfiler is a distributed trace toolkit library. It provides pythonic
|
|
helpers to do trace generation to avoid repeated code to trace WSGI, RPC, DB,
|
|
and other important places...
|
|
|
|
Integration OSProfiler into Rally can help to dig into concurrency problems of
|
|
OpenStack which is a huge ecosystem of cooperative services.
|
|
|
|
Problem description
|
|
===================
|
|
|
|
Rally Framework provides a powerful interface to generate real, big, load for
|
|
the deployment. Such load can kill the cloud, specifically OpenStack. There is
|
|
no way to identify reasons and bottlenecks without parsing timestamps and logs.
|
|
To fix that issue embedding profiling into each of workload iteration can help
|
|
to display wide picture of where we were in that particular moment when
|
|
something went wrong.
|
|
|
|
Proposed change
|
|
===============
|
|
|
|
Two facts about OSProfiler which are the start point for the proposed changes:
|
|
|
|
* HMAC key is used as a secret identifier while profiling
|
|
* Initialization of profiling is made in thread safe mode. Profiling of one
|
|
iteration should not influence profiling another one
|
|
|
|
Storing secret key
|
|
------------------
|
|
|
|
The HMAC key is not something that will be changed from one task to another.
|
|
It is specific thing for the deployment, like authentication url or other
|
|
credentials. That is why Rally deployment config is the best place to store
|
|
such information.
|
|
|
|
Since OSProfiler is OpenStack specific tool, we need to extend OpenStack
|
|
credentials model in Rally to support new argument. It should be done in two
|
|
places: validation (by modifying jsonschema [0]_) and the place where
|
|
credentials are store (specific class [1]_ [2]_).
|
|
|
|
Initialization profiling
|
|
------------------------
|
|
|
|
As it was mentioned before, we need to initialize OSProfiler per iteration.
|
|
OSProfiler is made in thread safe mode [3]_, so we should not have problem from
|
|
that side.
|
|
|
|
Initialization of OSProfiler is quite simple.
|
|
|
|
.. code-block:: python
|
|
|
|
from osprofiler import profiler
|
|
|
|
profiler.init(HMAC_KEY)
|
|
|
|
|
|
As for the place where to initialize OSProfiler in Rally, constructor of
|
|
scenario is a good choice. First of all, we have a separate class for OpenStack
|
|
scenarios [4]_ which means that integration with OSProfiler there will not
|
|
affect all other platforms. Another reason for using constructor is that we
|
|
initialize new instance of scenario class for each iterations.
|
|
|
|
Storing profiling results
|
|
-------------------------
|
|
|
|
OSProfiler sends to collector a message at every trace point. We should not
|
|
care about supported OSProfiler backends and use only OSProfiler as
|
|
entry-point.
|
|
|
|
The full trace can be obtained via trace-id after profiling is initialized.
|
|
|
|
.. code-block:: python
|
|
|
|
from osprofiler import profiler
|
|
|
|
trace_id = profiler.get().get_base_id()
|
|
|
|
At the first step of integration OSProfiler in Rally, let's store that trace-id
|
|
just like simple text. It will allow to show trace-id in Rally HTML and JSON
|
|
reports.
|
|
|
|
.. code-block:: python
|
|
|
|
self.add_output(complete={"title": "OSProfiler Trace-ID",
|
|
"chart_plugin": "TextArea",
|
|
"data": [trace_id]})
|
|
|
|
We can execute these lines in the same place where we initialize OSProfiler.
|
|
|
|
In future, we should develop a separate chart that will embed OSProfiler html
|
|
report as a separate tab in the Rally report.
|
|
|
|
Enabling profiling
|
|
------------------
|
|
|
|
Enabling/disabling profiling should be done via rally configuration file:
|
|
|
|
* It is common place for storing different kinds of options.
|
|
* There is planned feature that will able to re-set config options via
|
|
deployment config or task file.
|
|
|
|
The default value of that options should be True. In case of missing HMAC key
|
|
in credentials, attempt to initialize OSProfiler should not be started.
|
|
|
|
Alternatives
|
|
------------
|
|
|
|
Here [5]_ you can find the answer to that section.
|
|
|
|
|
|
Implementation
|
|
==============
|
|
|
|
Assignee(s)
|
|
-----------
|
|
|
|
Primary assignee:
|
|
Andrey Kurilin <andr.kurilin@gmail.com>
|
|
|
|
|
|
Work Items
|
|
----------
|
|
|
|
* Extend OpenStack credentials
|
|
* Add new configuration option to Rally
|
|
* Extend OpenStack scenario base class to initialize OSProfiler and store
|
|
trace id
|
|
|
|
|
|
Dependencies
|
|
============
|
|
|
|
None
|
|
|
|
|
|
References
|
|
==========
|
|
|
|
.. [0] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/credential.py#L154
|
|
.. [1] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/credential.py#L26
|
|
.. [2] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/credential.py#L161
|
|
.. [3] https://github.com/openstack/osprofiler/blob/1.8.0/osprofiler/profiler.py#L29-L30
|
|
.. [4] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/scenario.py#L28-L55
|
|
.. [5] https://docs.openstack.org/osprofiler/latest/user/background.html#why-not-cprofile-and-etc
|