[spec] OSProfiler integration in Rally
Change-Id: I0b9fefd368b0fb53f0c136c90430b1325c593b36
This commit is contained in:
Andrey Kurilin
parent
e00c391213
commit
a28cf9af27
@ -10,5 +10,6 @@ This directory contains 2 subdirectories:
|
||||
- in-progress - These specs are approved, but they are not implemented yet
|
||||
- implemented - Implemented specs archive
|
||||
|
||||
If you are looking for full rally road map overview go `here <https://docs.google.com/a/mirantis.com/spreadsheets/d/16DXpfbqvlzMFaqaXAcJsBzzpowb_XpymaK2aFY2gA2g/edit#gid=0>`_.
|
||||
If you are looking for full rally road map overview go
|
||||
`here <https://docs.google.com/a/mirantis.com/spreadsheets/d/16DXpfbqvlzMFaqaXAcJsBzzpowb_XpymaK2aFY2gA2g/edit#gid=0>`_.
|
||||
|
||||
|
||||
162
doc/specs/in-progress/osprofiler.rst
Normal file
162
doc/specs/in-progress/osprofiler.rst
Normal file
@ -0,0 +1,162 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
..
|
||||
This template should be in ReSTructured text. The filename in the git
|
||||
repository should match the launchpad URL, for example a URL of
|
||||
https://blueprints.launchpad.net/rally/+spec/awesome-thing should be named
|
||||
awesome-thing.rst . Please do not delete any of the sections in this
|
||||
template. If you have nothing to say for a whole section, just write: None
|
||||
For help with syntax, see http://sphinx-doc.org/rest.html
|
||||
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
|
||||
|
||||
=================================
|
||||
OSProfiler integration into Rally
|
||||
=================================
|
||||
|
||||
The OSProfiler is a distributed trace toolkit library. It provides pythonic
|
||||
helpers to do trace generation to avoid repeated code to trace WSGI, RPC, DB,
|
||||
and other important places...
|
||||
|
||||
Integration OSProfiler into Rally can help to dig into concurrency problems of
|
||||
OpenStack which is a huge ecosystem of cooperative services.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Rally Framework provides a powerful interface to generate real, big, load for
|
||||
the deployment. Such load can kill the cloud, specifically OpenStack. There is
|
||||
no way to identify reasons and bottlenecks without parsing timestamps and logs.
|
||||
To fix that issue embedding profiling into each of workload iteration can help
|
||||
to display wide picture of where we were in that particular moment when
|
||||
something went wrong.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
Two facts about OSProfiler which are the start point for the proposed changes:
|
||||
|
||||
* HMAC key is used as a secret identifier while profiling
|
||||
* Initialization of profiling is made in thread safe mode. Profiling of one
|
||||
iteration should not influence profiling another one
|
||||
|
||||
Storing secret key
|
||||
------------------
|
||||
|
||||
The HMAC key is not something that will be changed from one task to another.
|
||||
It is specific thing for the deployment, like authentication url or other
|
||||
credentials. That is why Rally deployment config is the best place to store
|
||||
such information.
|
||||
|
||||
Since OSProfiler is OpenStack specific tool, we need to extend OpenStack
|
||||
credentials model in Rally to support new argument. It should be done in two
|
||||
places: validation (by modifying jsonschema [0]_) and the place where
|
||||
credentials are store (specific class [1]_ [2]_).
|
||||
|
||||
Initialization profiling
|
||||
------------------------
|
||||
|
||||
As it was mentioned before, we need to initialize OSProfiler per iteration.
|
||||
OSProfiler is made in thread safe mode [3]_, so we should not have problem from
|
||||
that side.
|
||||
|
||||
Initialization of OSProfiler is quite simple.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from osprofiler import profiler
|
||||
|
||||
profiler.init(HMAC_KEY)
|
||||
|
||||
|
||||
As for the place where to initialize OSProfiler in Rally, constructor of
|
||||
scenario is a good choice. First of all, we have a separate class for OpenStack
|
||||
scenarios[4] which means that integration with OSProfiler there will not affect
|
||||
all other platforms. Another reason for using constructor is that we initialize
|
||||
new instance of scenario class for each iterations.
|
||||
|
||||
Storing profiling results
|
||||
-------------------------
|
||||
|
||||
OSProfiler sends to collector a message at every trace point. We should not
|
||||
care about supported OSProfiler backends and use only OSProfiler as
|
||||
entry-point.
|
||||
|
||||
The full trace can be obtained via trace-id after profiling is initialized.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from osprofiler import profiler
|
||||
|
||||
trace_id = profiler.get().get_base_id()
|
||||
|
||||
At the first step of integration OSProfiler in Rally, let's store that trace-id
|
||||
just like simple text. It will allow to show trace-id in Rally HTML and JSON
|
||||
reports.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
self.add_output(complete={"title": "OSProfiler Trace-ID",
|
||||
"chart_plugin": "TextArea",
|
||||
"data": [trace_id]})
|
||||
|
||||
We can execute these lines in the same place where we initialize OSProfiler.
|
||||
|
||||
In future, we should develop a separate chart that will embed OSProfiler html
|
||||
report as a separate tab in the Rally report.
|
||||
|
||||
Enabling profiling
|
||||
------------------
|
||||
|
||||
Enabling/disabling profiling should be done via rally configuration file:
|
||||
|
||||
* It is common place for storing different kinds of options.
|
||||
* There is planned feature that will able to re-set config options via
|
||||
deployment config or task file.
|
||||
|
||||
The default value of that options should be True. In case of missing HMAC key
|
||||
in credentials, attempt to initialize OSProfiler should not be started.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Here [5]_ you can find the answer to that section.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
Andrey Kurilin <andr.kurilin@gmail.com>
|
||||
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Extend OpenStack credentials
|
||||
* Add new configuration option to Rally
|
||||
* Extend OpenStack scenario base class to initialize OSProfiler and store
|
||||
trace id
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [0] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/credential.py#L154
|
||||
.. [1] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/credential.py#L26
|
||||
.. [2] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/credential.py#L161
|
||||
.. [3] https://github.com/openstack/osprofiler/blob/1.8.0/osprofiler/profiler.py#L29-L30
|
||||
.. [4] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/scenario.py#L28-L55
|
||||
.. [5] https://docs.openstack.org/developer/osprofiler/background.html#why-not-cprofile-and-etc
|
||||
@ -7,15 +7,15 @@
|
||||
..
|
||||
This template should be in ReSTructured text. The filename in the git
|
||||
repository should match the launchpad URL, for example a URL of
|
||||
https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named
|
||||
https://blueprints.launchpad.net/rally/+spec/awesome-thing should be named
|
||||
awesome-thing.rst . Please do not delete any of the sections in this
|
||||
template. If you have nothing to say for a whole section, just write: None
|
||||
For help with syntax, see http://sphinx-doc.org/rest.html
|
||||
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
|
||||
|
||||
=======================
|
||||
The title of your Spec
|
||||
=======================
|
||||
======================
|
||||
The title of your Spec
|
||||
======================
|
||||
|
||||
Rally Road map:
|
||||
|
||||
@ -81,3 +81,8 @@ Dependencies
|
||||
- Does this feature require any new library dependencies or code otherwise not
|
||||
included in OpenStack? Or does it depend on a specific version of library?
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
Links to some external resources.
|
||||
|
||||
@ -16,15 +16,16 @@ import re
|
||||
|
||||
import docutils.core
|
||||
|
||||
import rally
|
||||
from tests.unit import test
|
||||
|
||||
|
||||
class TitlesTestCase(test.TestCase):
|
||||
|
||||
OPTIONAL_SECTIONS = ["References"]
|
||||
|
||||
specs_path = os.path.join(
|
||||
os.path.dirname(__file__),
|
||||
os.pardir, os.pardir, os.pardir,
|
||||
"doc", "specs")
|
||||
os.path.dirname(rally.__file__), os.pardir, "doc", "specs")
|
||||
|
||||
def _get_title(self, section_tree):
|
||||
section = {"subtitles": []}
|
||||
@ -46,7 +47,9 @@ class TitlesTestCase(test.TestCase):
|
||||
return titles
|
||||
|
||||
def _check_titles(self, filename, expect, actual):
|
||||
missing_sections = [x for x in expect.keys() if x not in actual.keys()]
|
||||
missing_sections = [
|
||||
x for x in expect.keys()
|
||||
if x not in actual.keys() and x not in self.OPTIONAL_SECTIONS]
|
||||
extra_sections = [x for x in actual.keys() if x not in expect.keys()]
|
||||
|
||||
msgs = []
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user