3625f307f9
This change adds a section to the developer guidelines about using the key manager within the sahara codebase. It also adds some minor cleanup to the sahara_key_manager docstrings. * add a section to the development guidelines about the key manager * add an anchor in the advanced configuration guide for the key manager * improve the docstrings in the sahara.service.castellan.utils module * cleanup the docstrings in the sahara.service.castellan.sahara_key_manager module Change-Id: Ied9278113deab5af0bbd80d229810e0636bb2b72 Partial-Implements: blueprint improved-secret-storage
241 lines
8.5 KiB
ReStructuredText
241 lines
8.5 KiB
ReStructuredText
Development Guidelines
|
|
======================
|
|
|
|
Coding Guidelines
|
|
-----------------
|
|
|
|
For all the Python code in Sahara we have a rule - it should pass `PEP 8`_.
|
|
All Bash code should pass `bashate`_.
|
|
|
|
To check your code against PEP 8 and bashate run:
|
|
|
|
.. sourcecode:: console
|
|
|
|
$ tox -e pep8
|
|
|
|
.. note::
|
|
For more details on coding guidelines see file ``HACKING.rst`` in the root
|
|
of Sahara repo.
|
|
|
|
Static analysis
|
|
---------------
|
|
|
|
The static analysis checks are optional in Sahara. but they are still very
|
|
useful. The gate job will inform you if the number of static analysis warnings
|
|
has increased after your change. We recommend to always check the static
|
|
warnings.
|
|
|
|
To run the check commit yor change first and execute the following command:
|
|
|
|
.. sourcecode:: console
|
|
|
|
$ tox -e pylint
|
|
|
|
Modification of Upstream Files
|
|
------------------------------
|
|
|
|
We never modify upstream files in Sahara. Any changes in upstream files should
|
|
be made in the upstream project and then merged back in to Sahara. This
|
|
includes whitespace changes, comments, and typos. Any change requests
|
|
containing upstream file modifications are almost certain to receive lots of
|
|
negative reviews. Be warned.
|
|
|
|
Examples of upstream files are default xml configuration files used to
|
|
configure Hadoop, or code imported from the OpenStack Oslo project. The xml
|
|
files will usually be found in ``resource`` directories with an accompanying
|
|
``README`` file that identifies where the files came from. For example:
|
|
|
|
.. sourcecode:: console
|
|
|
|
$ pwd
|
|
/home/me/sahara/sahara/plugins/vanilla/v2_6_0/resources
|
|
|
|
$ ls
|
|
core-default.xml hdfs-default.xml oozie-default.xml README.rst
|
|
create_oozie_db.sql mapred-default.xml post_conf.template yarn-default.xml
|
|
..
|
|
|
|
Testing Guidelines
|
|
------------------
|
|
|
|
Sahara has a suite of tests that are run on all submitted code,
|
|
and it is recommended that developers execute the tests themselves to
|
|
catch regressions early. Developers are also expected to keep the
|
|
test suite up-to-date with any submitted code changes.
|
|
|
|
Unit tests are located at ``sahara/tests/unit``.
|
|
|
|
Sahara's suite of unit tests can be executed in an isolated environment
|
|
with `Tox`_. To execute the unit tests run the following from the root of
|
|
Sahara repo:
|
|
|
|
.. sourcecode:: console
|
|
|
|
$ tox -e py27
|
|
|
|
|
|
Documentation Guidelines
|
|
------------------------
|
|
|
|
All Sahara docs are written using Sphinx / RST and located in the main repo
|
|
in ``doc`` directory. You can add/edit pages here to update
|
|
http://docs.openstack.org/developer/sahara site.
|
|
|
|
The documentation in docstrings should follow the `PEP 257`_ conventions
|
|
(as mentioned in the `PEP 8`_ guidelines).
|
|
|
|
More specifically:
|
|
|
|
1. Triple quotes should be used for all docstrings.
|
|
2. If the docstring is simple and fits on one line, then just use
|
|
one line.
|
|
3. For docstrings that take multiple lines, there should be a newline
|
|
after the opening quotes, and before the closing quotes.
|
|
4. `Sphinx`_ is used to build documentation, so use the restructured text
|
|
markup to designate parameters, return values, etc. Documentation on
|
|
the sphinx specific markup can be found here:
|
|
|
|
|
|
|
|
Run the following command to build docs locally.
|
|
|
|
.. sourcecode:: console
|
|
|
|
$ tox -e docs
|
|
|
|
After it you can access generated docs in ``doc/build/`` directory, for
|
|
example, main page - ``doc/build/html/index.html``.
|
|
|
|
To make docs generation process faster you can use:
|
|
|
|
.. sourcecode:: console
|
|
|
|
$ SPHINX_DEBUG=1 tox -e docs
|
|
|
|
or to avoid sahara reinstallation to virtual env each time you want to rebuild
|
|
docs you can use the following command (it could be executed only after
|
|
running ``tox -e docs`` first time):
|
|
|
|
.. sourcecode:: console
|
|
|
|
$ SPHINX_DEBUG=1 .tox/docs/bin/python setup.py build_sphinx
|
|
|
|
|
|
|
|
.. note::
|
|
For more details on documentation guidelines see file HACKING.rst in the root
|
|
of Sahara repo.
|
|
|
|
|
|
.. _PEP 8: http://www.python.org/dev/peps/pep-0008/
|
|
.. _bashate: https://github.com/openstack-dev/bashate
|
|
.. _PEP 257: http://www.python.org/dev/peps/pep-0257/
|
|
.. _Tox: http://tox.testrun.org/
|
|
.. _Sphinx: http://sphinx.pocoo.org/markup/index.html
|
|
|
|
Event log Guidelines
|
|
--------------------
|
|
|
|
Currently Sahara keep with cluster useful information about provisioning.
|
|
Cluster provisioning can be represented as a linear series of provisioning
|
|
steps, which are executed one after another. Also each step would consist of
|
|
several events. The amount of events depends on the step and the amount of
|
|
instances in the cluster. Also each event can contain information about
|
|
cluster, instance, and node group. In case of errors, this event would contain
|
|
information about reasons of errors. Each exception in sahara contains a
|
|
unique identifier that will allow the user to find extra information about
|
|
the reasons for errors in the sahara logs. Here
|
|
http://developer.openstack.org/api-ref-data-processing-v1.1.html#v1.1eventlog
|
|
you can see an example of provisioning progress information.
|
|
|
|
This means that if you add some important phase for cluster provisioning to
|
|
sahara code, it's recommended to add new provisioning step for this phase.
|
|
It would allow users to use event log for handling errors during this phase.
|
|
|
|
Sahara already have special utils for operating provisioning steps and events
|
|
in module ``sahara/utils/cluster_progress_ops.py``.
|
|
|
|
.. note::
|
|
It's strictly recommended not use ``conductor`` event log ops directly
|
|
to assign events and operate provisioning steps.
|
|
|
|
.. note::
|
|
You should not add a new provisioning step until the previous step
|
|
successfully completed.
|
|
|
|
.. note::
|
|
It's strictly recommended to use ``event_wrapper`` for events handling
|
|
|
|
OpenStack client usage guidelines
|
|
---------------------------------
|
|
|
|
The sahara project uses several OpenStack clients internally. These clients
|
|
are all wrapped by utility functions which make using them more convenient.
|
|
When developing sahara, if you need to use a OpenStack client you should
|
|
check the ``sahara.utils.openstack`` package for the appropriate one.
|
|
|
|
When developing new OpenStack client interactions in sahara, it is important
|
|
to understand the ``sahara.service.sessions`` package and the usage of
|
|
keystone ``Session`` and auth plugin objects(for example, ``Token`` or
|
|
``Password``). Sahara is migrating all clients to use this authentication
|
|
methodology, where available. For more information on using sessions with
|
|
keystone, please see
|
|
http://docs.openstack.org/developer/python-keystoneclient/using-sessions.html
|
|
|
|
Storing sensitive information
|
|
-----------------------------
|
|
|
|
During the course of development, there is often cause to store sensitive
|
|
information (for example, login credentials) in the records for a cluster,
|
|
job, or some other record. Storing secret information this way is **not**
|
|
safe. To mitigate the risk of storing this information, sahara provides
|
|
access to the OpenStack Key Manager service (implemented by the
|
|
`barbican project <http://docs.openstack.org/developer/barbican/>`_) through
|
|
the `castellan library <http://docs.openstack.org/developer/castellan/>`_.
|
|
|
|
To utilize the external key manager, the functions in
|
|
``sahara.service.castellan.utils`` are provided as wrappers around the
|
|
castellan library. These functions allow a developer to store, retrieve, and
|
|
delete secrets from the manager. Secrets that are managed through the key
|
|
manager have an identifier associated with them. These identifiers are
|
|
considered safe to store in the database.
|
|
|
|
The following are some examples of working with secrets in the sahara
|
|
codebase. These examples are considered basic, any developer wishing to
|
|
learn more about the advanced features of storing secrets should look to
|
|
the code and docstrings contained in the ``sahara.service.castellan`` module.
|
|
|
|
**Storing a secret**
|
|
|
|
.. sourcecode:: python
|
|
|
|
from sahara.service.castellan import utils as key_manager
|
|
|
|
password = 'SooperSecretPassword'
|
|
identifier = key_manager.store_secret(password)
|
|
|
|
**Retrieving a secret**
|
|
|
|
.. sourcecode:: python
|
|
|
|
from sahara.service.castellan import utils as key_manager
|
|
|
|
password = key_manager.get_secret(identifier)
|
|
|
|
**Deleting a secret**
|
|
|
|
.. sourcecode:: python
|
|
|
|
from sahara.service.castellan import utils as key_manager
|
|
|
|
key_manager.delete_secret(identifier)
|
|
|
|
When storing secrets through this interface it is important to remember that
|
|
if an external key manager is being used, each stored secret creates an
|
|
entry in an external service. When you are finished using the secret it is
|
|
good practice to delete it, as not doing so may leave artifacts in those
|
|
external services.
|
|
|
|
For more information on configuring sahara to use the OpenStack Key
|
|
Manager service, see :ref:`external_key_manager_usage`.
|