diff --git a/doc/source/devref/api_change_tutorial.rst b/doc/source/devref/api_change_tutorial.rst index 967420c20..acfb7d536 100644 --- a/doc/source/devref/api_change_tutorial.rst +++ b/doc/source/devref/api_change_tutorial.rst @@ -22,8 +22,8 @@ Prerequisites ------------- In order to follow this tutorial, it is assumed that you have read our -:doc:`development_best_practices` and :doc:`../getting-started/architecture` -documents. +:doc:`index` and +:doc:`../getting-started/architecture` documents. Proposing a change ------------------ diff --git a/doc/source/devref/build-docs.rst b/doc/source/devref/build-docs.rst new file mode 100644 index 000000000..86c4e7c7c --- /dev/null +++ b/doc/source/devref/build-docs.rst @@ -0,0 +1,28 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Building the Documentation +========================== + +The documentation is generated with Sphinx using the tox command. To create HTML +docs and man pages: + +.. code-block:: bash + + $ tox -e docs + +The results are in the ``doc/build/html`` and ``doc/build/man`` directories +respectively. diff --git a/doc/source/devref/caching-layer.rst b/doc/source/devref/caching-layer.rst new file mode 100644 index 000000000..961612432 --- /dev/null +++ b/doc/source/devref/caching-layer.rst @@ -0,0 +1,153 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Caching Layer +============= + +The caching layer is designed to be applied to any ``manager`` object within Keystone +via the use of the ``on_arguments`` decorator provided in the ``keystone.common.cache`` +module. This decorator leverages `dogpile.cache`_ caching system to provide a flexible +caching backend. + +It is recommended that each of the managers have an independent toggle within the config +file to enable caching. The easiest method to utilize the toggle within the +configuration file is to define a ``caching`` boolean option within that manager's +configuration section (e.g. ``identity``). Once that option is defined you can +pass function to the ``on_arguments`` decorator with the named argument ``should_cache_fn``. +In the ``keystone.common.cache`` module, there is a function called ``should_cache_fn``, +which will provide a reference, to a function, that will consult the global cache +``enabled`` option as well as the specific manager's caching enable toggle. + + .. NOTE:: + If a section-specific boolean option is not defined in the config section specified when + calling ``should_cache_fn``, the returned function reference will default to enabling + caching for that ``manager``. + +Example use of cache and ``should_cache_fn`` (in this example, ``token`` is the manager): + +.. code-block:: python + + from keystone.common import cache + SHOULD_CACHE = cache.should_cache_fn('token') + + @cache.on_arguments(should_cache_fn=SHOULD_CACHE) + def cacheable_function(arg1, arg2, arg3): + ... + return some_value + +With the above example, each call to the ``cacheable_function`` would check to see if +the arguments passed to it matched a currently valid cached item. If the return value +was cached, the caching layer would return the cached value; if the return value was +not cached, the caching layer would call the function, pass the value to the ``SHOULD_CACHE`` +function reference, which would then determine if caching was globally enabled and enabled +for the ``token`` manager. If either caching toggle is disabled, the value is returned but +not cached. + +It is recommended that each of the managers have an independent configurable time-to-live (TTL). +If a configurable TTL has been defined for the manager configuration section, it is possible to +pass it to the ``cache.on_arguments`` decorator with the named-argument ``expiration_time``. For +consistency, it is recommended that this option be called ``cache_time`` and default to ``None``. +If the ``expiration_time`` argument passed to the decorator is set to ``None``, the expiration +time will be set to the global default (``expiration_time`` option in the ``[cache]`` +configuration section. + +Example of using a section specific ``cache_time`` (in this example, ``identity`` is the manager): + +.. code-block:: python + + from keystone.common import cache + SHOULD_CACHE = cache.should_cache_fn('identity') + + @cache.on_arguments(should_cache_fn=SHOULD_CACHE, + expiration_time=CONF.identity.cache_time) + def cachable_function(arg1, arg2, arg3): + ... + return some_value + +For cache invalidation, the ``on_arguments`` decorator will add an ``invalidate`` method +(attribute) to your decorated function. To invalidate the cache, you pass the same arguments +to the ``invalidate`` method as you would the normal function. + +Example (using the above cacheable_function): + +.. code-block:: python + + def invalidate_cache(arg1, arg2, arg3): + cacheable_function.invalidate(arg1, arg2, arg3) + +.. WARNING:: + The ``on_arguments`` decorator does not accept keyword-arguments/named arguments. An + exception will be raised if keyword arguments are passed to a caching-decorated function. + +.. NOTE:: + In all cases methods work the same as functions except if you are attempting to invalidate + the cache on a decorated bound-method, you need to pass ``self`` to the ``invalidate`` + method as the first argument before the arguments. + +.. _`dogpile.cache`: http://dogpilecache.readthedocs.org/ + + +dogpile.cache based MongoDB (NoSQL) backend +------------------------------------------- + +The ``dogpile.cache`` based MongoDB backend implementation allows for various MongoDB +configurations, e.g., standalone, a replica set, sharded replicas, with or without SSL, +use of TTL type collections, etc. + +Example of typical configuration for MongoDB backend: + +.. code-block:: python + + from dogpile.cache import region + + arguments = { + 'db_hosts': 'localhost:27017', + 'db_name': 'ks_cache', + 'cache_collection': 'cache', + 'username': 'test_user', + 'password': 'test_password', + + # optional arguments + 'son_manipulator': 'my_son_manipulator_impl' + } + + region.make_region().configure('keystone.cache.mongo', + arguments=arguments) + +The optional `son_manipulator` is used to manipulate custom data type while its saved in +or retrieved from MongoDB. If the dogpile cached values contain built-in data types and no +custom classes, then the provided implementation class is sufficient. For further details, refer +http://api.mongodb.org/python/current/examples/custom_type.html#automatic-encoding-and-decoding + +Similar to other backends, this backend can be added via Keystone configuration in +``keystone.conf``:: + + [cache] + # Global cache functionality toggle. + enabled = True + + # Referring to specific cache backend + backend = keystone.cache.mongo + + # Backend specific configuration arguments + backend_argument = db_hosts:localhost:27017 + backend_argument = db_name:ks_cache + backend_argument = cache_collection:cache + backend_argument = username:test_user + backend_argument = password:test_password + +This backend is registered in ``keystone.common.cache.core`` module. So, its usage +is similar to other dogpile caching backends as it implements the same dogpile APIs. diff --git a/doc/source/devref/conf.rst b/doc/source/devref/conf.rst new file mode 100644 index 000000000..0d5ef4d7b --- /dev/null +++ b/doc/source/devref/conf.rst @@ -0,0 +1,40 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Configuring Keystone with a sample file +======================================= + +Keystone requires a configuration file. Keystone's sample configuration file +``etc/keystone.conf.sample`` is automatically generated based upon all of the +options available within Keystone. These options are sourced from the many +files around Keystone as well as some external libraries. + +The sample configuration file will be updated as the end of the development +cycle approaches. Developers should *NOT* generate the config file and propose +it as part of their patches, this will cause unnecessary conflicts. +You can generate one locally using the following command: + +.. code-block:: bash + + $ tox -e genconfig + +The tox command will place an updated sample config in ``etc/keystone.conf.sample``. +The defaults are enough to get you going, but you can make any changes if +needed. + +If there is a new external library (e.g. ``oslo.messaging``) that utilizes the +``oslo.config`` package for configuration, it can be added to the list of libraries +found in ``config-generator/keystone.conf``. diff --git a/doc/source/devref/database-migrations.rst b/doc/source/devref/database-migrations.rst new file mode 100644 index 000000000..b7651f000 --- /dev/null +++ b/doc/source/devref/database-migrations.rst @@ -0,0 +1,92 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Database Migrations +=================== + +Starting with Newton, keystone supports upgrading both with and without +downtime. In order to support this, there are three separate migration +repositories (all under ``keystone/common/sql/``) that match the three phases +of an upgrade (schema expansion, data migration, and schema contraction): + +``expand_repo`` + For additive schema modifications and triggers to ensure data is kept in + sync between the old and new schema until the point when there are no + keystone instances running old code. + +``data_migration_repo`` + To ensure new tables/columns are fully populated with data from the old + schema. + +``contract_repo`` + Run after all old code versions have been upgraded to running the new code, + so remove any old schema columns/tables that are not used by the new + version of the code. Drop any triggers added in the expand phase. + +All migrations are required to have a migration script in each of these repos, +each with the same version number (which is indicated by the first three digits +of the name of the script, e.g. ``003_add_X_table.py``). If there is no work to +do in a specific phase, then include a no-op migration to simply ``pass`` (in +fact the ``001`` migration in each of these repositories is a no-op migration, +so that can be used as a template). + +.. NOTE:: + + Since rolling upgrade support was added part way through the Newton cycle, + some migrations had already been added to the legacy repository + (``keystone/common/sql/migrate_repo``). This repository is now closed and + no new migrations should be added (except for backporting of previous + placeholders). + +In order to support rolling upgrades, where two releases of keystone briefly +operate side-by-side using the same database without downtime, each phase of +the migration must adhere to following constraints: + +These triggers should be removed in the contract phase. There are further +restrictions as to what can and cannot be included in migration scripts in each +phase: + +Expand phase: + Only additive schema changes are allowed, such as new columns, tables, + indices, and triggers. + + Data insertion, modification, and removal is not allowed. + + Triggers must be created to keep data in sync between the previous release + and the next release. Data written by the previous release must be readable + by both the previous release and the next release. Data written by the next + release must be readable by both the next release and the previous release. + + In cases it is not possible for triggers to maintain data integrity across + multiple schemas, writing data should be forbidden using triggers. + +Data Migration phase: + Data is allowed to be inserted, updated, and deleted. + + No schema changes are allowed. + +Contract phase: + Only contractive schema changes are allowed, such as dropping or altering + columns, tables, indices, and triggers. + + Data insertion, modification, and removal is not allowed. + + Triggers created during the expand phase must be dropped. + +For more information on writing individual migration scripts refer to +`SQLAlchemy-migrate`_. + +.. _SQLAlchemy-migrate: https://git.openstack.org/cgit/openstack/sqlalchemy-migrate diff --git a/doc/source/devref/development_best_practices.rst b/doc/source/devref/development_best_practices.rst deleted file mode 100644 index f01674eb5..000000000 --- a/doc/source/devref/development_best_practices.rst +++ /dev/null @@ -1,1002 +0,0 @@ -.. - Copyright 2011-2012 OpenStack Foundation - All Rights Reserved. - - Licensed under the Apache License, Version 2.0 (the "License"); you may - not use this file except in compliance with the License. You may obtain - a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - License for the specific language governing permissions and limitations - under the License. - -============== -Best Practices -============== - -Setting up Keystone -=================== - -Get your development environment set up according to -:doc:`development_environment`. It is recommended that you install -Keystone into a virtualenv. - - -Configuring Keystone -==================== - -Keystone requires a configuration file. There is a sample configuration file -that can be used to get started: - -.. code-block:: bash - - $ cp etc/keystone.conf.sample etc/keystone.conf - -The defaults are enough to get you going, but you can make any changes if -needed. - - -Running Keystone -================ - -To run the Keystone Admin and API server instances, use: - -.. code-block:: bash - - $ uwsgi --http 127.0.0.1:35357 --wsgi-file $(which keystone-wsgi-admin) - -This runs Keystone with the configuration the etc/ directory of the project. -See :doc:`../configuration` for details on how Keystone is configured. By default, -Keystone is configured with SQL backends. - - -Initializing Keystone -===================== - -Before using keystone, it is necessary to create the database tables and ensures -the database schemas are up to date, perform the following: - -.. code-block:: bash - - $ keystone-manage db_sync - -If the above commands result in a ``KeyError``, or they fail on a -``.pyc`` file with the message, ``You can only have one Python script per -version``, then it is possible that there are out-of-date compiled Python -bytecode files in the Keystone directory tree that are causing problems. This -can occur if you have previously installed and ran older versions of Keystone. -These out-of-date files can be easily removed by running a command like the -following from the Keystone root project directory: - -.. code-block:: bash - - $ find . -name "*.pyc" -delete - -Initial Sample Data -------------------- - -There is an included script which is helpful in setting up some initial sample -data for use with keystone: - -.. code-block:: bash - - $ ADMIN_PASSWORD=s3cr3t tools/sample_data.sh - -Once run, you can see the sample data that has been created by using the -`python-openstackclient`_ command-line interface: - -.. code-block:: bash - - $ export OS_USERNAME=admin - $ export OS_PASSWORD=s3cr3t - $ export OS_PROJECT_NAME=admin - $ export OS_USER_DOMAIN_ID=default - $ export OS_PROJECT_DOMAIN_ID=default - $ export OS_IDENTITY_API_VERSION=3 - $ export OS_AUTH_URL=http://localhost:5000/v3 - $ openstack user list - -The `python-openstackclient`_ can be installed using the following: - -.. code-block:: bash - - $ pip install python-openstackclient - -Interacting with Keystone -========================= - -You can also interact with keystone through its REST API. There is a Python -keystone client library `python-keystoneclient`_ which interacts exclusively -through the REST API, and a command-line interface `python-openstackclient`_ -command-line interface. - -.. _`python-keystoneclient`: https://git.openstack.org/cgit/openstack/python-keystoneclient -.. _`python-openstackclient`: https://git.openstack.org/cgit/openstack/python-openstackclient - -Building the Documentation -========================== - -The documentation is generated with Sphinx using the tox command. To create HTML -docs and man pages: - -.. code-block:: bash - - $ tox -e docs - -The results are in the ``doc/build/html`` and ``doc/build/man`` directories -respectively. - - -Generating a new Sample Config File -=================================== - -Keystone's sample configuration file ``etc/keystone.conf.sample`` is automatically -generated based upon all of the options available within Keystone. These options -are sourced from the many files around Keystone as well as some external libraries. - -The sample configuration file will be updated as the end of the development -cycle approaches. Developers should *NOT* generate the config file and propose -it as part of their patches, this will cause unnecessary conflicts. - -To generate a new sample configuration to see what it looks like, run: - -.. code-block:: bash - - $ tox -egenconfig -r - -The tox command will place an updated sample config in ``etc/keystone.conf.sample``. - -If there is a new external library (e.g. ``oslo.messaging``) that utilizes the -``oslo.config`` package for configuration, it can be added to the list of libraries -found in ``config-generator/keystone.conf``. - - -Release Notes -============= - -The Keystone team uses `reno -`_ to generate release -notes. These are important user-facing documents that must be included when a -user or operator facing change is performed, like a bug-fix or a new feature. A -release note should be included in the same patch the work is being performed. -Release notes should be easy to read and maintain; should link back to -appropriate documentation readers may need. The following conventions help the -team ensure all release notes achieve those goals. - -Most release notes either describe bug fixes or announce support for new -features, both of which are tracked using Launchpad. When creating a release -note that communicates a bug fix, use the bug number in the name of the note: - -.. code-block:: bash - - $ reno new bug-1652012 - Created new notes file in releasenotes/notes/bug-1652012-7c53b9702b10084d.yaml - -The body of the release note should clearly explain how the impact will affect -users and operators. It should also include why the change was necessary but -not be overspecific about implementation details, as that can be found in the -commit. It should contain a properly formatted link in reStructuredText that -points back to the original bug report used to track the fix. This makes -reading release notes easier because readers can get a quick summary of the -change, understand how it is going to impact them, and follow a link to more -detail if they choose. - -.. code-block:: yaml - - --- - fixes: - - | - [`bug 1652012 `_] - Changes the token_model to return is_admin_project False if the - attribute is not defined. Returning True for this has the potential to - be dangerous and the given reason for keeping it True was strictly for - backwards compatability. - -Release notes detailing feature work follow the same basic format, but instead -of using the bug number in the name of the release note, use the blueprint slug -used to track the feature work: - -.. code-block:: bash - - $ reno new bp-support-fizzbangs - Created new notes file in releasenotes/notes/bp-support-fizzbangs-d8f6a3d81c2a465f.yaml - -Just like release notes communicating bug fixes, release notes detailing -feature work must contain a link back to the blueprint. Readers should be able -to easily discover all patches that implement the feature, as well as find -links to the full specification and documentation. All of this is typically -found in the blueprint registered in Launchpad. - -.. code-block:: yaml - - --- - features: - - > - [`blueprint support-fizzbangs`_] - Keystone now fully supports the usage of fizzbangs. - -In the rare case there is a release note that does not pertain to a bug or -feature work, use a sensible slug and include any documentation relating to the -note. We can iterate on the content and application of the release note during -the review process. - -For more information on how and when to create release notes, see the -`project-team-guide `_. - -Testing Keystone -================ - -Running Tests -------------- - -Before running tests, you should have ``tox`` installed and available in your -environment (in addition to the other external dependencies in -:doc:`development_environment`): - -.. code-block:: bash - - $ pip install tox - -.. NOTE:: - - You may need to perform both the above operation and the next inside a - python virtualenv, or prefix the above command with ``sudo``, depending on - your preference. - -To execute the full suite of tests maintained within Keystone, simply run: - -.. code-block:: bash - - $ tox - -This iterates over multiple configuration variations, and uses external -projects to do light integration testing to verify the Identity API against -other projects. - -.. NOTE:: - - The first time you run ``tox``, it will take additional time to build - virtualenvs. You can later use the ``-r`` option with ``tox`` to rebuild - your virtualenv in a similar manner. - -To run tests for one or more specific test environments (for example, the most -common configuration of Python 2.7 and PEP-8), list the environments with the -``-e`` option, separated by spaces: - -.. code-block:: bash - - $ tox -e py27,pep8 - -See ``tox.ini`` for the full list of available test environments. - -Running with PDB -~~~~~~~~~~~~~~~~ - -Using PDB breakpoints with tox and testr normally doesn't work since the tests -just fail with a BdbQuit exception rather than stopping at the breakpoint. - -To run with PDB breakpoints during testing, use the ``debug`` tox environment -rather than ``py27``. Here's an example, passing the name of a test since -you'll normally only want to run the test that hits your breakpoint: - -.. code-block:: bash - - $ tox -e debug keystone.tests.unit.test_auth.AuthWithToken.test_belongs_to - -For reference, the ``debug`` tox environment implements the instructions -here: https://wiki.openstack.org/wiki/Testr#Debugging_.28pdb.29_Tests - -Disabling Stream Capture -~~~~~~~~~~~~~~~~~~~~~~~~ - -The stdout, stderr and log messages generated during a test are captured and -in the event of a test failure those streams will be printed to the terminal -along with the traceback. The data is discarded for passing tests. - -Each stream has an environment variable that can be used to force captured -data to be discarded even if the test fails: `OS_STDOUT_CAPTURE` for stdout, -`OS_STDERR_CAPTURE` for stderr and `OS_LOG_CAPTURE` for logging. If the value -of the environment variable is not one of (True, true, 1, yes) the stream will -be discarded. All three variables default to 1. - -For example, to discard logging data during a test run: - -.. code-block:: bash - - $ OS_LOG_CAPTURE=0 tox -e py27 - -Tests Structure ---------------- - -Not all of the tests in the ``keystone/tests/unit`` directory are strictly unit -tests. Keystone intentionally includes tests that run the service locally and -drives the entire configuration to achieve basic functional testing. - -For the functional tests, an in-memory key-value store or in-memory SQLite -database is used to keep the tests fast. - -Within the tests directory, the general structure of the backend tests is a -basic set of tests represented under a test class, and then subclasses of those -tests under other classes with different configurations to drive different -backends through the APIs. To add tests covering all drivers, update the base -test class in ``test_backend.py``. - -.. NOTE:: - - The structure of backend testing is in transition, migrating from having - all classes in a single file (test_backend.py) to one where there is a - directory structure to reduce the size of the test files. See: - - - :mod:`keystone.tests.unit.backend.role` - - :mod:`keystone.tests.unit.backend.domain_config` - -To add new drivers, subclass the base class at ``test_backend.py`` (look -towards ``test_backend_sql.py`` for examples) and update the configuration of -the test class in ``setUp()``. - -For example, ``test_backend.py`` has a sequence of tests under the class -:class:`~keystone.tests.unit.test_backend.IdentityTests` that will work with -the default drivers as configured in this project's etc/ directory. -``test_backend_sql.py`` subclasses those tests, changing the configuration by -overriding with configuration files stored in the ``tests/unit/config_files`` -directory aimed at enabling the SQL backend for the Identity module. - -:class:`keystone.tests.unit.test_v2_keystoneclient.ClientDrivenTestCase` -uses the installed python-keystoneclient, verifying it against a temporarily -running local keystone instance to explicitly verify basic functional testing -across the API. - -Testing Schema Migrations -------------------------- - -The application of schema migrations can be tested using SQLAlchemy Migrate’s -built-in test runner, one migration at a time. - -.. WARNING:: - - This may leave your database in an inconsistent state; attempt this in - non-production environments only! - -This is useful for testing the *next* migration in sequence (both forward & -backward) in a database under version control: - -.. code-block:: bash - - $ python keystone/common/sql/migrate_repo/manage.py test \ - --url=sqlite:///test.db \ - --repository=keystone/common/sql/migrate_repo/ - -This command references to a SQLite database (test.db) to be used. Depending on -the migration, this command alone does not make assertions as to the integrity -of your data during migration. - -LDAP Tests ----------- - -LDAP has a fake backend that performs rudimentary operations. If you -are building more significant LDAP functionality, you should test against -a live LDAP server. Devstack has an option to set up a directory server for -Keystone to use. Add ldap to the ``ENABLED_SERVICES`` environment variable, -and set environment variables ``KEYSTONE_IDENTITY_BACKEND=ldap`` and -``KEYSTONE_CLEAR_LDAP=yes`` in your ``localrc`` file. - -The unit tests can be run against a live server with -``keystone/tests/unit/test_ldap_livetest.py`` and -``keystone/tests/unit/test_ldap_pool_livetest.py``. The default password is -``test`` but if you have installed devstack with a different LDAP password, -modify the file ``keystone/tests/unit/config_files/backend_liveldap.conf`` and -``keystone/tests/unit/config_files/backend_pool_liveldap.conf`` to reflect your -password. - -.. NOTE:: - To run the live tests you need to set the environment variable - ``ENABLE_LDAP_LIVE_TEST`` to a non-negative value. - -"Work in progress" Tests ------------------------- - -Work in progress (WIP) tests are very useful in a variety of situations -including: - -* During a TDD process they can be used to add tests to a review while - they are not yet working and will not cause test failures. (They should - be removed before the final merge.) -* Often bug reports include small snippets of code to show broken - behaviors. Some of these can be converted into WIP tests that can later - be worked on by a developer. This allows us to take code that can be - used to catch bug regressions and commit it before any code is - written. - -The :func:`keystone.tests.unit.utils.wip` decorator can be used to mark a test -as WIP. A WIP test will always be run. If the test fails then a TestSkipped -exception is raised because we expect the test to fail. We do not pass -the test in this case so that it doesn't count toward the number of -successfully run tests. If the test passes an AssertionError exception is -raised so that the developer knows they made the test pass. This is a -reminder to remove the decorator. - -The :func:`~keystone.tests.unit.utils.wip` decorator requires that the author -provides a message. This message is important because it will tell other -developers why this test is marked as a work in progress. Reviewers will -require that these messages are descriptive and accurate. - -.. NOTE:: - The :func:`~keystone.tests.unit.utils.wip` decorator is not a replacement for - skipping tests. - -.. code-block:: python - - @wip('waiting on bug #000000') - def test(): - pass - -.. NOTE:: - Another strategy is to not use the wip decorator and instead show how the - code currently incorrectly works. Which strategy is chosen is up to the - developer. - -API & Scenario Tests --------------------- - -Keystone provides API and scenario tests via a `tempest plugin`_ located at -:func:`~keystone.keystone_tempest_plugin`. This tempest plugin is mainly -intended for specific scenarios that require a special deployment, such as -the tests for the ``Federated Identity`` feature. For the deployment of these -scenarios, keystone also provides a `devstack plugin`_. - -For example, to setup a working federated environment, add the following lines -in your `devstack` `local.conf`` file: - -.. code-block:: bash - - [[local|localrc]] - enable_plugin keystone git://git.openstack.org/openstack/keystone - enable_service keystone-saml2-federation - -Finally, to run keystone's API and scenario tests, deploy `tempest`_ with -`devstack`_ (using the configuration above) and then run the following command -from the tempest directory: - -.. code-block:: bash - - tox -e all-plugin -- keystone_tempest_plugin - -.. NOTE:: - Most of keystone's API tests are implemented in `tempest`_ and it is usually - the correct place to add new tests. - -.. _devstack: https://git.openstack.org/cgit/openstack-dev/devstack -.. _devstack plugin: https://docs.openstack.org/developer/devstack/plugins.html -.. _tempest: https://git.openstack.org/cgit/openstack/tempest -.. _tempest plugin: https://docs.openstack.org/developer/tempest/plugin.html - -Writing new API & Scenario Tests -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When writing tests for the keystone tempest plugin, we should follow the -official tempest guidelines, details about the guidelines can be found at the -`tempest coding guide`_. There are also specific guides for the API and -scenario tests: `Tempest Field Guide to API tests`_ and -`Tempest Field Guide to Scenario tests`_. - -The keystone tempest plugin also provides a base class. For most cases, the -tests should inherit from it: -:class:`keystone_tempest_plugin.tests.base.BaseIdentityTest`. This class -already setups the identity API version and is the container of all API -services clients. -New API services clients :mod:`keystone_tempest_plugin.services` -(which are used to communicate with the REST API from -the services) should also be added to this class. For example, below we have a -snippet from the tests at -:py:mod:`keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.py`. - -.. code-block:: python - - class IdentityProvidersTest(base.BaseIdentityTest): - - ... - - def _create_idp(self, idp_id, idp_ref): - idp = self.idps_client.create_identity_provider( - idp_id, **idp_ref)['identity_provider'] - self.addCleanup( - self.idps_client.delete_identity_provider, idp_id) - return idp - - @decorators.idempotent_id('09450910-b816-4150-8513-a2fd4628a0c3') - def test_identity_provider_create(self): - idp_id = data_utils.rand_uuid_hex() - idp_ref = fixtures.idp_ref() - idp = self._create_idp(idp_id, idp_ref) - - # The identity provider is disabled by default - idp_ref['enabled'] = False - - # The remote_ids attribute should be set to an empty list by default - idp_ref['remote_ids'] = [] - - self._assert_identity_provider_attributes(idp, idp_id, idp_ref) - -The test class extends -:class:`keystone_tempest_plugin.tests.base.BaseIdentityTest`. Also, the -``_create_idp`` method calls keystone's API using the ``idps_client``, -which is an instance from. -:class:`keystone_tempest_plugin.tests.services.identity.v3.identity_providers_client.IdentityProvidersClient`. - -Additionally, to illustrate the construction of a new test class, below we have -a snippet from the scenario test that checks the complete federated -authentication workflow ( -:py:mod:`keystone_tempest_plugin.tests.scenario.test_federated_authentication.py`). -In the test setup, all of the needed resources are created using the API -service clients. Since it is a scenario test, it is common to need some -customized settings that will come from the environment (in this case, from -the devstack plugin) - these settings are collected in the ``_setup_settings`` -method. - -.. code-block:: python - - class TestSaml2EcpFederatedAuthentication(base.BaseIdentityTest): - - ... - - def _setup_settings(self): - self.idp_id = CONF.fed_scenario.idp_id - self.idp_url = CONF.fed_scenario.idp_ecp_url - self.keystone_v3_endpoint = CONF.identity.uri_v3 - self.password = CONF.fed_scenario.idp_password - self.protocol_id = CONF.fed_scenario.protocol_id - self.username = CONF.fed_scenario.idp_username - - ... - - def setUp(self): - super(TestSaml2EcpFederatedAuthentication, self).setUp() - self._setup_settings() - - # Reset client's session to avoid getting garbage from another runs - self.saml2_client.reset_session() - - # Setup identity provider, mapping and protocol - self._setup_idp() - self._setup_mapping() - self._setup_protocol() - -Finally, the tests perform the complete workflow of the feature, asserting its -correctness in each step: - -.. code-block:: python - - def _request_unscoped_token(self): - resp = self.saml2_client.send_service_provider_request( - self.keystone_v3_endpoint, self.idp_id, self.protocol_id) - self.assertEqual(http_client.OK, resp.status_code) - saml2_authn_request = etree.XML(resp.content) - - relay_state = self._str_from_xml( - saml2_authn_request, self.ECP_RELAY_STATE) - sp_consumer_url = self._str_from_xml( - saml2_authn_request, self.ECP_SERVICE_PROVIDER_CONSUMER_URL) - - # Perform the authn request to the identity provider - resp = self.saml2_client.send_identity_provider_authn_request( - saml2_authn_request, self.idp_url, self.username, self.password) - self.assertEqual(http_client.OK, resp.status_code) - saml2_idp_authn_response = etree.XML(resp.content) - - idp_consumer_url = self._str_from_xml( - saml2_idp_authn_response, self.ECP_IDP_CONSUMER_URL) - - # Assert that both saml2_authn_request and saml2_idp_authn_response - # have the same consumer URL. - self.assertEqual(sp_consumer_url, idp_consumer_url) - - ... - - - @testtools.skipUnless(CONF.identity_feature_enabled.federation, - "Federated Identity feature not enabled") - def test_request_unscoped_token(self): - self._request_unscoped_token() - -Notice that the ``test_request_unscoped_token`` test only executes if the the -``federation`` feature flag is enabled. - -.. NOTE:: - For each patch submitted upstream, all of the tests from the keystone - tempest plugin are executed in the - ``gate-keystone-dsvm-functional-v3-only-*`` job. - -.. _Tempest Field Guide to Scenario tests: https://docs.openstack.org/developer/tempest/field_guide/scenario.html -.. _Tempest Field Guide to API tests: https://docs.openstack.org/developer/tempest/field_guide/api.html -.. _tempest coding guide: https://docs.openstack.org/developer/tempest/HACKING.html - -Developing ``doctor`` checks -============================ - -As noted in the section above, keystone's management CLI provides various tools -for administrating OpenStack Identity. One of those tools is called -``keystone-manage doctor`` and it is responsible for performing health checks -about the deployment. If ``keystone-manage doctor`` detects a symptom, it -will provide the operator with suggestions to improve the overall health of the -deployment. This section is dedicated to documenting how to write symptoms for -``doctor``. - -The ``doctor`` tool consists of a list of symptoms. Each symptom is something -that we can check against, and provide a warning for if we detect a -misconfiguration. The ``doctor`` module is located in -:py:mod:`keystone.cmd.doctor`. The current checks are based heavily on -inspecting configuration values. As a result, many of the submodules within the -``doctor`` module are named after the configuration section for the symptoms -they check. For example, if we want to ensure the ``keystone.conf [DEFAULT] -max_token_size`` option is properly configured for whatever ``keystone.conf -[token] provider`` is set to, we can place that symptom in a module called -:py:mod:`keystone.cmd.doctor.tokens`. The symptom will be loaded by -importing the ``doctor`` module, which is done when ``keystone-manage doctor`` -is invoked from the command line. When adding new symptoms, it's important to -remember to add new modules to the ``SYMPTOM_MODULES`` list in -:py:mod:`keystone.cmd.doctor.__init__`. Doing that will ensure ``doctor`` -discovers properly named symptoms when executed. - -Now that we know symptoms are organized according to configuration sections, -and how to add them, how exactly do we write a new symptom? ``doctor`` will -automatically discover new symptoms by inspecting the methods of each symptom -module (i.e. ``SYMPTOM_MODULES``). If a method declaration starts with -``def symptom_`` it is considered a symptom that ``doctor`` should check for, -and it should be run. The naming of the symptom, or method name, is extremely -important since ``doctor`` will use it to describe what it's doing to whoever -runs ``doctor``. In addition to a well named method, we also need to provide a -complete documentation string for the method. If ``doctor`` detects a symptom, -it will use the method's documentation string as feedback to the operator. It -should describe why the check is being done, why it was triggered, and possible -solutions to cure the symptom. For examples of this, see the existing symptoms -in any of ``doctor``'s symptom modules. - -The last step is evaluating the logic within the symptom. As previously stated, -``doctor`` will check for a symptom if methods within specific symptom modules -make a specific naming convention. In order for ``doctor`` to suggest feedback, -it needs to know whether or not the symptom is actually present. We accomplish -this by making all symptoms return ``True`` when a symptom is present. When a -symptom evaluates to ``False``, ``doctor`` will move along to the next symptom -in the list since. If the deployment isn't suffering for a specific symptom, -``doctor`` should not suggest any actions related to that symptom (i.e. if -you have your cholesterol under control, why would a physician recommend -cholesterol medication if you don't need it). - -To summarize: - -- Symptoms should live in modules named according to the most relevant - configuration section they apply to. This ensure we keep our symptoms - organized, grouped, and easy to find. -- When writing symptoms for a new section, remember to add the module name to - the ``SYMPTOM_MODULES`` list in :py:mod:`keystone.cmd.doctor.__init__`. -- Remember to use a good name for the symptom method signature and to prepend - it with ``symptom_`` in order for it to be discovered automatically by - ``doctor``. -- Symptoms have to evaluate to ``True`` in order to provide feedback to - operators. -- Symptoms should have very thorough documentation strings that describe the - symptom, side-effects of the symptom, and ways to remedy it. - -For examples, feel free to run ``doctor`` locally using ``keystone-manage`` and -inspect the existing symptoms. - -Database Migrations -=================== - -Starting with Newton, keystone supports upgrading both with and without -downtime. In order to support this, there are three separate migration -repositories (all under ``keystone/common/sql/``) that match the three phases -of an upgrade (schema expansion, data migration, and schema contraction): - -``expand_repo`` - For additive schema modifications and triggers to ensure data is kept in - sync between the old and new schema until the point when there are no - keystone instances running old code. - -``data_migration_repo`` - To ensure new tables/columns are fully populated with data from the old - schema. - -``contract_repo`` - Run after all old code versions have been upgraded to running the new code, - so remove any old schema columns/tables that are not used by the new - version of the code. Drop any triggers added in the expand phase. - -All migrations are required to have a migration script in each of these repos, -each with the same version number (which is indicated by the first three digits -of the name of the script, e.g. ``003_add_X_table.py``). If there is no work to -do in a specific phase, then include a no-op migration to simply ``pass`` (in -fact the ``001`` migration in each of these repositories is a no-op migration, -so that can be used as a template). - -.. NOTE:: - - Since rolling upgrade support was added part way through the Newton cycle, - some migrations had already been added to the legacy repository - (``keystone/common/sql/migrate_repo``). This repository is now closed and - no new migrations should be added (except for backporting of previous - placeholders). - -In order to support rolling upgrades, where two releases of keystone briefly -operate side-by-side using the same database without downtime, each phase of -the migration must adhere to following constraints: - -These triggers should be removed in the contract phase. There are further -restrictions as to what can and cannot be included in migration scripts in each -phase: - -Expand phase: - Only additive schema changes are allowed, such as new columns, tables, - indices, and triggers. - - Data insertion, modification, and removal is not allowed. - - Triggers must be created to keep data in sync between the previous release - and the next release. Data written by the previous release must be readable - by both the previous release and the next release. Data written by the next - release must be readable by both the next release and the previous release. - - In cases it is not possible for triggers to maintain data integrity across - multiple schemas, writing data should be forbidden using triggers. - -Data Migration phase: - Data is allowed to be inserted, updated, and deleted. - - No schema changes are allowed. - -Contract phase: - Only contractive schema changes are allowed, such as dropping or altering - columns, tables, indices, and triggers. - - Data insertion, modification, and removal is not allowed. - - Triggers created during the expand phase must be dropped. - -For more information on writing individual migration scripts refer to -`SQLAlchemy-migrate`_. - -.. _SQLAlchemy-migrate: https://git.openstack.org/cgit/openstack/sqlalchemy-migrate - - -Filtering responsibilities between controllers and drivers -========================================================== - -Keystone supports the specification of filtering on list queries as part of the -v3 identity API. By default these queries are satisfied in the controller -class when a controller calls the ``wrap_collection`` method at the end of a -``list_{entity}`` method. However, to enable optimum performance, any driver -can implement some or all of the specified filters (for example, by adding -filtering to the generated SQL statements to generate the list). - -The communication of the filter details between the controller level and its -drivers is handled by the passing of a reference to a Hints object, -which is a list of dicts describing the filters. A driver that satisfies a -filter must delete the filter from the Hints object so that when it is returned -to the controller level, it knows to only execute any unsatisfied -filters. - -The contract for a driver for ``list_{entity}`` methods is therefore: - -* It MUST return a list of entities of the specified type -* It MAY either just return all such entities, or alternatively reduce the - list by filtering for one or more of the specified filters in the passed - Hints reference, and removing any such satisfied filters. An exception to - this is that for identity drivers that support domains, then they should - at least support filtering by domain_id. - -Entity list truncation by drivers -================================= - -Keystone supports the ability for a deployment to restrict the number of -entries returned from ``list_{entity}`` methods, typically to prevent poorly -formed searches (e.g. without sufficient filters) from becoming a performance -issue. - -These limits are set in the configuration file, either for a specific driver or -across all drivers. These limits are read at the Manager level and passed into -individual drivers as part of the Hints list object. A driver should try and -honor any such limit if possible, but if it is unable to do so then it may -ignore it (and the truncation of the returned list of entities will happen at -the controller level). - -Identity entity ID management between controllers and drivers -============================================================= - -Keystone supports the option of having domain-specific backends for the -identity driver (i.e. for user and group storage), allowing, for example, -a different LDAP server for each domain. To ensure that Keystone can determine -to which backend it should route an API call, starting with Juno, the -identity manager will, provided that domain-specific backends are enabled, -build on-the-fly a persistent mapping table between Keystone Public IDs that -are presented to the controller and the domain that holds the entity, along -with whatever local ID is understood by the driver. This hides, for instance, -the LDAP specifics of whatever ID is being used. - -To ensure backward compatibility, the default configuration of either a -single SQL or LDAP backend for Identity will not use the mapping table, -meaning that public facing IDs will be the unchanged. If keeping these IDs -the same for the default LDAP backend is not required, then setting the -configuration variable ``backward_compatible_ids`` to ``False`` will enable -the mapping for the default LDAP driver, hence hiding the LDAP specifics of the -IDs being used. - - -Translated responses -==================== - -The Keystone server can provide error responses translated into the language in -the ``Accept-Language`` header of the request. In order to test this in your -development environment, there's a couple of things you need to do. - -1. Build the message files. Run the following command in your keystone - directory: - -.. code-block:: bash - - $ python setup.py compile_catalog - -This will generate .mo files like keystone/locale/[lang]/LC_MESSAGES/[lang].mo - -2. When running Keystone, set the ``KEYSTONE_LOCALEDIR`` environment variable - to the keystone/locale directory. For example: - -.. code-block:: bash - - $ KEYSTONE_LOCALEDIR=/opt/stack/keystone/keystone/locale uwsgi --http 127.0.0.1:35357 --wsgi-file $(which keystone-wsgi-admin) - - -Now you can get a translated error response: - -.. code-block:: bash - - $ curl -s -H "Accept-Language: zh" http://localhost:5000/notapath | python -mjson.tool - { - "error": { - "code": 404, - "message": "\u627e\u4e0d\u5230\u8cc7\u6e90\u3002", - "title": "Not Found" - } - } - - -Caching Layer -============= - -The caching layer is designed to be applied to any ``manager`` object within Keystone -via the use of the ``on_arguments`` decorator provided in the ``keystone.common.cache`` -module. This decorator leverages `dogpile.cache`_ caching system to provide a flexible -caching backend. - -It is recommended that each of the managers have an independent toggle within the config -file to enable caching. The easiest method to utilize the toggle within the -configuration file is to define a ``caching`` boolean option within that manager's -configuration section (e.g. ``identity``). Once that option is defined you can -pass function to the ``on_arguments`` decorator with the named argument ``should_cache_fn``. -In the ``keystone.common.cache`` module, there is a function called ``should_cache_fn``, -which will provide a reference, to a function, that will consult the global cache -``enabled`` option as well as the specific manager's caching enable toggle. - - .. NOTE:: - If a section-specific boolean option is not defined in the config section specified when - calling ``should_cache_fn``, the returned function reference will default to enabling - caching for that ``manager``. - -Example use of cache and ``should_cache_fn`` (in this example, ``token`` is the manager): - -.. code-block:: python - - from keystone.common import cache - SHOULD_CACHE = cache.should_cache_fn('token') - - @cache.on_arguments(should_cache_fn=SHOULD_CACHE) - def cacheable_function(arg1, arg2, arg3): - ... - return some_value - -With the above example, each call to the ``cacheable_function`` would check to see if -the arguments passed to it matched a currently valid cached item. If the return value -was cached, the caching layer would return the cached value; if the return value was -not cached, the caching layer would call the function, pass the value to the ``SHOULD_CACHE`` -function reference, which would then determine if caching was globally enabled and enabled -for the ``token`` manager. If either caching toggle is disabled, the value is returned but -not cached. - -It is recommended that each of the managers have an independent configurable time-to-live (TTL). -If a configurable TTL has been defined for the manager configuration section, it is possible to -pass it to the ``cache.on_arguments`` decorator with the named-argument ``expiration_time``. For -consistency, it is recommended that this option be called ``cache_time`` and default to ``None``. -If the ``expiration_time`` argument passed to the decorator is set to ``None``, the expiration -time will be set to the global default (``expiration_time`` option in the ``[cache]`` -configuration section. - -Example of using a section specific ``cache_time`` (in this example, ``identity`` is the manager): - -.. code-block:: python - - from keystone.common import cache - SHOULD_CACHE = cache.should_cache_fn('identity') - - @cache.on_arguments(should_cache_fn=SHOULD_CACHE, - expiration_time=CONF.identity.cache_time) - def cachable_function(arg1, arg2, arg3): - ... - return some_value - -For cache invalidation, the ``on_arguments`` decorator will add an ``invalidate`` method -(attribute) to your decorated function. To invalidate the cache, you pass the same arguments -to the ``invalidate`` method as you would the normal function. - -Example (using the above cacheable_function): - -.. code-block:: python - - def invalidate_cache(arg1, arg2, arg3): - cacheable_function.invalidate(arg1, arg2, arg3) - -.. WARNING:: - The ``on_arguments`` decorator does not accept keyword-arguments/named arguments. An - exception will be raised if keyword arguments are passed to a caching-decorated function. - -.. NOTE:: - In all cases methods work the same as functions except if you are attempting to invalidate - the cache on a decorated bound-method, you need to pass ``self`` to the ``invalidate`` - method as the first argument before the arguments. - -.. _`dogpile.cache`: http://dogpilecache.readthedocs.org/ - - -dogpile.cache based MongoDB (NoSQL) backend -------------------------------------------- - -The ``dogpile.cache`` based MongoDB backend implementation allows for various MongoDB -configurations, e.g., standalone, a replica set, sharded replicas, with or without SSL, -use of TTL type collections, etc. - -Example of typical configuration for MongoDB backend: - -.. code-block:: python - - from dogpile.cache import region - - arguments = { - 'db_hosts': 'localhost:27017', - 'db_name': 'ks_cache', - 'cache_collection': 'cache', - 'username': 'test_user', - 'password': 'test_password', - - # optional arguments - 'son_manipulator': 'my_son_manipulator_impl' - } - - region.make_region().configure('keystone.cache.mongo', - arguments=arguments) - -The optional `son_manipulator` is used to manipulate custom data type while its saved in -or retrieved from MongoDB. If the dogpile cached values contain built-in data types and no -custom classes, then the provided implementation class is sufficient. For further details, refer -http://api.mongodb.org/python/current/examples/custom_type.html#automatic-encoding-and-decoding - -Similar to other backends, this backend can be added via Keystone configuration in -``keystone.conf``:: - - [cache] - # Global cache functionality toggle. - enabled = True - - # Referring to specific cache backend - backend = keystone.cache.mongo - - # Backend specific configuration arguments - backend_argument = db_hosts:localhost:27017 - backend_argument = db_name:ks_cache - backend_argument = cache_collection:cache - backend_argument = username:test_user - backend_argument = password:test_password - -This backend is registered in ``keystone.common.cache.core`` module. So, its usage -is similar to other dogpile caching backends as it implements the same dogpile APIs. diff --git a/doc/source/devref/development_environment.rst b/doc/source/devref/development_environment.rst index 79c0751e9..2be6cb551 100644 --- a/doc/source/devref/development_environment.rst +++ b/doc/source/devref/development_environment.rst @@ -76,7 +76,7 @@ Once set up, you should be able to invoke Python and import the libraries: $ .tox/py27/bin/python -c "import keystone" If you can import keystone without a traceback, you should be ready to move on -to :doc:`development_best_practices`. +to the next sections. Database setup ============== diff --git a/doc/source/devref/doctor-checks.rst b/doc/source/devref/doctor-checks.rst new file mode 100644 index 000000000..cd3ecf094 --- /dev/null +++ b/doc/source/devref/doctor-checks.rst @@ -0,0 +1,85 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Developing ``doctor`` checks +============================ + +As noted in the section above, keystone's management CLI provides various tools +for administrating OpenStack Identity. One of those tools is called +``keystone-manage doctor`` and it is responsible for performing health checks +about the deployment. If ``keystone-manage doctor`` detects a symptom, it +will provide the operator with suggestions to improve the overall health of the +deployment. This section is dedicated to documenting how to write symptoms for +``doctor``. + +The ``doctor`` tool consists of a list of symptoms. Each symptom is something +that we can check against, and provide a warning for if we detect a +misconfiguration. The ``doctor`` module is located in +:py:mod:`keystone.cmd.doctor`. The current checks are based heavily on +inspecting configuration values. As a result, many of the submodules within the +``doctor`` module are named after the configuration section for the symptoms +they check. For example, if we want to ensure the ``keystone.conf [DEFAULT] +max_token_size`` option is properly configured for whatever ``keystone.conf +[token] provider`` is set to, we can place that symptom in a module called +:py:mod:`keystone.cmd.doctor.tokens`. The symptom will be loaded by +importing the ``doctor`` module, which is done when ``keystone-manage doctor`` +is invoked from the command line. When adding new symptoms, it's important to +remember to add new modules to the ``SYMPTOM_MODULES`` list in +:py:mod:`keystone.cmd.doctor.__init__`. Doing that will ensure ``doctor`` +discovers properly named symptoms when executed. + +Now that we know symptoms are organized according to configuration sections, +and how to add them, how exactly do we write a new symptom? ``doctor`` will +automatically discover new symptoms by inspecting the methods of each symptom +module (i.e. ``SYMPTOM_MODULES``). If a method declaration starts with +``def symptom_`` it is considered a symptom that ``doctor`` should check for, +and it should be run. The naming of the symptom, or method name, is extremely +important since ``doctor`` will use it to describe what it's doing to whoever +runs ``doctor``. In addition to a well named method, we also need to provide a +complete documentation string for the method. If ``doctor`` detects a symptom, +it will use the method's documentation string as feedback to the operator. It +should describe why the check is being done, why it was triggered, and possible +solutions to cure the symptom. For examples of this, see the existing symptoms +in any of ``doctor``'s symptom modules. + +The last step is evaluating the logic within the symptom. As previously stated, +``doctor`` will check for a symptom if methods within specific symptom modules +make a specific naming convention. In order for ``doctor`` to suggest feedback, +it needs to know whether or not the symptom is actually present. We accomplish +this by making all symptoms return ``True`` when a symptom is present. When a +symptom evaluates to ``False``, ``doctor`` will move along to the next symptom +in the list since. If the deployment isn't suffering for a specific symptom, +``doctor`` should not suggest any actions related to that symptom (i.e. if +you have your cholesterol under control, why would a physician recommend +cholesterol medication if you don't need it). + +To summarize: + +- Symptoms should live in modules named according to the most relevant + configuration section they apply to. This ensure we keep our symptoms + organized, grouped, and easy to find. +- When writing symptoms for a new section, remember to add the module name to + the ``SYMPTOM_MODULES`` list in :py:mod:`keystone.cmd.doctor.__init__`. +- Remember to use a good name for the symptom method signature and to prepend + it with ``symptom_`` in order for it to be discovered automatically by + ``doctor``. +- Symptoms have to evaluate to ``True`` in order to provide feedback to + operators. +- Symptoms should have very thorough documentation strings that describe the + symptom, side-effects of the symptom, and ways to remedy it. + +For examples, feel free to run ``doctor`` locally using ``keystone-manage`` and +inspect the existing symptoms. diff --git a/doc/source/devref/filtering-responsibilities.rst b/doc/source/devref/filtering-responsibilities.rst new file mode 100644 index 000000000..8c9f863ea --- /dev/null +++ b/doc/source/devref/filtering-responsibilities.rst @@ -0,0 +1,41 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Filtering responsibilities between controllers and drivers +========================================================== + +Keystone supports the specification of filtering on list queries as part of the +v3 identity API. By default these queries are satisfied in the controller +class when a controller calls the ``wrap_collection`` method at the end of a +``list_{entity}`` method. However, to enable optimum performance, any driver +can implement some or all of the specified filters (for example, by adding +filtering to the generated SQL statements to generate the list). + +The communication of the filter details between the controller level and its +drivers is handled by the passing of a reference to a Hints object, +which is a list of dicts describing the filters. A driver that satisfies a +filter must delete the filter from the Hints object so that when it is returned +to the controller level, it knows to only execute any unsatisfied +filters. + +The contract for a driver for ``list_{entity}`` methods is therefore: + +* It MUST return a list of entities of the specified type +* It MAY either just return all such entities, or alternatively reduce the + list by filtering for one or more of the specified filters in the passed + Hints reference, and removing any such satisfied filters. An exception to + this is that for identity drivers that support domains, then they should + at least support filtering by domain_id. diff --git a/doc/source/devref/id-manage.rst b/doc/source/devref/id-manage.rst new file mode 100644 index 000000000..d4728e0c0 --- /dev/null +++ b/doc/source/devref/id-manage.rst @@ -0,0 +1,36 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Identity entity ID management between controllers and drivers +============================================================= + +Keystone supports the option of having domain-specific backends for the +identity driver (i.e. for user and group storage), allowing, for example, +a different LDAP server for each domain. To ensure that Keystone can determine +to which backend it should route an API call, starting with Juno, the +identity manager will, provided that domain-specific backends are enabled, +build on-the-fly a persistent mapping table between Keystone Public IDs that +are presented to the controller and the domain that holds the entity, along +with whatever local ID is understood by the driver. This hides, for instance, +the LDAP specifics of whatever ID is being used. + +To ensure backward compatibility, the default configuration of either a +single SQL or LDAP backend for Identity will not use the mapping table, +meaning that public facing IDs will be the unchanged. If keeping these IDs +the same for the default LDAP backend is not required, then setting the +configuration variable ``backward_compatible_ids`` to ``False`` will enable +the mapping for the default LDAP driver, hence hiding the LDAP specifics of the +IDs being used. diff --git a/doc/source/devref/index.rst b/doc/source/devref/index.rst index 81aa717fd..c79b4efad 100644 --- a/doc/source/devref/index.rst +++ b/doc/source/devref/index.rst @@ -1,3 +1,19 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + ========================= Contributor Documentation ========================= @@ -5,7 +21,21 @@ Contributor Documentation .. toctree:: :maxdepth: 1 - development_best_practices.rst + setup.rst + conf.rst + run-keystone.rst + initialise.rst + interact.rst + build-docs.rst + release-notes.rst + testing-keystone.rst + doctor-checks.rst + database-migrations.rst + filtering-responsibilities.rst + list-truncation.rst + id-manage.rst + translated-responses.rst + caching-layer.rst development_environment.rst developing_drivers.rst api_change_tutorial.rst diff --git a/doc/source/devref/initialise.rst b/doc/source/devref/initialise.rst new file mode 100644 index 000000000..a7afa29c7 --- /dev/null +++ b/doc/source/devref/initialise.rst @@ -0,0 +1,69 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Initializing Keystone +===================== + +Before using keystone, it is necessary to create the database tables and ensures +the database schemas are up to date, perform the following: + +.. code-block:: bash + + $ keystone-manage db_sync + +If the above commands result in a ``KeyError``, or they fail on a +``.pyc`` file with the message, ``You can only have one Python script per +version``, then it is possible that there are out-of-date compiled Python +bytecode files in the Keystone directory tree that are causing problems. This +can occur if you have previously installed and ran older versions of Keystone. +These out-of-date files can be easily removed by running a command like the +following from the Keystone root project directory: + +.. code-block:: bash + + $ find . -name "*.pyc" -delete + +Initial Sample Data +------------------- + +There is an included script which is helpful in setting up some initial sample +data for use with keystone: + +.. code-block:: bash + + $ ADMIN_PASSWORD=s3cr3t tools/sample_data.sh + +Once run, you can see the sample data that has been created by using the +`python-openstackclient`_ command-line interface: + +.. code-block:: bash + + $ export OS_USERNAME=admin + $ export OS_PASSWORD=s3cr3t + $ export OS_PROJECT_NAME=admin + $ export OS_USER_DOMAIN_ID=default + $ export OS_PROJECT_DOMAIN_ID=default + $ export OS_IDENTITY_API_VERSION=3 + $ export OS_AUTH_URL=http://localhost:5000/v3 + $ openstack user list + +The `python-openstackclient`_ can be installed using the following: + +.. code-block:: bash + + $ pip install python-openstackclient + +.. _`python-openstackclient`: https://git.openstack.org/cgit/openstack/python-openstackclient diff --git a/doc/source/devref/interact.rst b/doc/source/devref/interact.rst new file mode 100644 index 000000000..37f5476e0 --- /dev/null +++ b/doc/source/devref/interact.rst @@ -0,0 +1,26 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Interacting with Keystone +========================= + +You can also interact with keystone through its REST API. There is a Python +keystone client library `python-keystoneclient`_ which interacts exclusively +through the REST API, and a command-line interface `python-openstackclient`_ +command-line interface. + +.. _`python-keystoneclient`: https://git.openstack.org/cgit/openstack/python-keystoneclient +.. _`python-openstackclient`: https://git.openstack.org/cgit/openstack/python-openstackclient diff --git a/doc/source/devref/list-truncation.rst b/doc/source/devref/list-truncation.rst new file mode 100644 index 000000000..1f10200a0 --- /dev/null +++ b/doc/source/devref/list-truncation.rst @@ -0,0 +1,30 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Entity list truncation by drivers +================================= + +Keystone supports the ability for a deployment to restrict the number of +entries returned from ``list_{entity}`` methods, typically to prevent poorly +formed searches (e.g. without sufficient filters) from becoming a performance +issue. + +These limits are set in the configuration file, either for a specific driver or +across all drivers. These limits are read at the Manager level and passed into +individual drivers as part of the Hints list object. A driver should try and +honor any such limit if possible, but if it is unable to do so then it may +ignore it (and the truncation of the returned list of entities will happen at +the controller level). diff --git a/doc/source/devref/release-notes.rst b/doc/source/devref/release-notes.rst new file mode 100644 index 000000000..1012b6c82 --- /dev/null +++ b/doc/source/devref/release-notes.rst @@ -0,0 +1,87 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Release Notes +============= + +The Keystone team uses `reno +`_ to generate release +notes. These are important user-facing documents that must be included when a +user or operator facing change is performed, like a bug-fix or a new feature. A +release note should be included in the same patch the work is being performed. +Release notes should be easy to read and maintain; should link back to +appropriate documentation readers may need. The following conventions help the +team ensure all release notes achieve those goals. + +Most release notes either describe bug fixes or announce support for new +features, both of which are tracked using Launchpad. When creating a release +note that communicates a bug fix, use the bug number in the name of the note: + +.. code-block:: bash + + $ reno new bug-1652012 + Created new notes file in releasenotes/notes/bug-1652012-7c53b9702b10084d.yaml + +The body of the release note should clearly explain how the impact will affect +users and operators. It should also include why the change was necessary but +not be overspecific about implementation details, as that can be found in the +commit. It should contain a properly formatted link in reStructuredText that +points back to the original bug report used to track the fix. This makes +reading release notes easier because readers can get a quick summary of the +change, understand how it is going to impact them, and follow a link to more +detail if they choose. + +.. code-block:: yaml + + --- + fixes: + - | + [`bug 1652012 `_] + Changes the token_model to return is_admin_project False if the + attribute is not defined. Returning True for this has the potential to + be dangerous and the given reason for keeping it True was strictly for + backwards compatability. + +Release notes detailing feature work follow the same basic format, but instead +of using the bug number in the name of the release note, use the blueprint slug +used to track the feature work: + +.. code-block:: bash + + $ reno new bp-support-fizzbangs + Created new notes file in releasenotes/notes/bp-support-fizzbangs-d8f6a3d81c2a465f.yaml + +Just like release notes communicating bug fixes, release notes detailing +feature work must contain a link back to the blueprint. Readers should be able +to easily discover all patches that implement the feature, as well as find +links to the full specification and documentation. All of this is typically +found in the blueprint registered in Launchpad. + +.. code-block:: yaml + + --- + features: + - > + [`blueprint support-fizzbangs`_] + Keystone now fully supports the usage of fizzbangs. + +In the rare case there is a release note that does not pertain to a bug or +feature work, use a sensible slug and include any documentation relating to the +note. We can iterate on the content and application of the release note during +the review process. + +For more information on how and when to create release notes, see the +`project-team-guide `_. diff --git a/doc/source/devref/run-keystone.rst b/doc/source/devref/run-keystone.rst new file mode 100644 index 000000000..779319c29 --- /dev/null +++ b/doc/source/devref/run-keystone.rst @@ -0,0 +1,28 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Running Keystone +================ + +To run the Keystone Admin and API server instances, use: + +.. code-block:: bash + + $ uwsgi --http 127.0.0.1:35357 --wsgi-file $(which keystone-wsgi-admin) + +This runs Keystone with the configuration the etc/ directory of the project. +See :doc:`../configuration` for details on how Keystone is configured. By default, +Keystone is configured with SQL backends. diff --git a/doc/source/devref/setup.rst b/doc/source/devref/setup.rst new file mode 100644 index 000000000..8c5a6b310 --- /dev/null +++ b/doc/source/devref/setup.rst @@ -0,0 +1,22 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Setting up Keystone +=================== + +Get your development environment set up according to +:doc:`development_environment`. It is recommended that you install +Keystone into a virtualenv. diff --git a/doc/source/devref/testing-keystone.rst b/doc/source/devref/testing-keystone.rst new file mode 100644 index 000000000..2d98ccb59 --- /dev/null +++ b/doc/source/devref/testing-keystone.rst @@ -0,0 +1,402 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Testing Keystone +================ + +Running Tests +------------- + +Before running tests, you should have ``tox`` installed and available in your +environment (in addition to the other external dependencies in +:doc:`development_environment`): + +.. code-block:: bash + + $ pip install tox + +.. NOTE:: + + You may need to perform both the above operation and the next inside a + python virtualenv, or prefix the above command with ``sudo``, depending on + your preference. + +To execute the full suite of tests maintained within Keystone, simply run: + +.. code-block:: bash + + $ tox + +This iterates over multiple configuration variations, and uses external +projects to do light integration testing to verify the Identity API against +other projects. + +.. NOTE:: + + The first time you run ``tox``, it will take additional time to build + virtualenvs. You can later use the ``-r`` option with ``tox`` to rebuild + your virtualenv in a similar manner. + +To run tests for one or more specific test environments (for example, the most +common configuration of Python 2.7 and PEP-8), list the environments with the +``-e`` option, separated by spaces: + +.. code-block:: bash + + $ tox -e py27,pep8 + +See ``tox.ini`` for the full list of available test environments. + +Running with PDB +~~~~~~~~~~~~~~~~ + +Using PDB breakpoints with tox and testr normally doesn't work since the tests +just fail with a BdbQuit exception rather than stopping at the breakpoint. + +To run with PDB breakpoints during testing, use the ``debug`` tox environment +rather than ``py27``. Here's an example, passing the name of a test since +you'll normally only want to run the test that hits your breakpoint: + +.. code-block:: bash + + $ tox -e debug keystone.tests.unit.test_auth.AuthWithToken.test_belongs_to + +For reference, the ``debug`` tox environment implements the instructions +here: https://wiki.openstack.org/wiki/Testr#Debugging_.28pdb.29_Tests + +Disabling Stream Capture +~~~~~~~~~~~~~~~~~~~~~~~~ + +The stdout, stderr and log messages generated during a test are captured and +in the event of a test failure those streams will be printed to the terminal +along with the traceback. The data is discarded for passing tests. + +Each stream has an environment variable that can be used to force captured +data to be discarded even if the test fails: `OS_STDOUT_CAPTURE` for stdout, +`OS_STDERR_CAPTURE` for stderr and `OS_LOG_CAPTURE` for logging. If the value +of the environment variable is not one of (True, true, 1, yes) the stream will +be discarded. All three variables default to 1. + +For example, to discard logging data during a test run: + +.. code-block:: bash + + $ OS_LOG_CAPTURE=0 tox -e py27 + +Tests Structure +--------------- + +Not all of the tests in the ``keystone/tests/unit`` directory are strictly unit +tests. Keystone intentionally includes tests that run the service locally and +drives the entire configuration to achieve basic functional testing. + +For the functional tests, an in-memory key-value store or in-memory SQLite +database is used to keep the tests fast. + +Within the tests directory, the general structure of the backend tests is a +basic set of tests represented under a test class, and then subclasses of those +tests under other classes with different configurations to drive different +backends through the APIs. To add tests covering all drivers, update the base +test class in ``test_backend.py``. + +.. NOTE:: + + The structure of backend testing is in transition, migrating from having + all classes in a single file (test_backend.py) to one where there is a + directory structure to reduce the size of the test files. See: + + - :mod:`keystone.tests.unit.backend.role` + - :mod:`keystone.tests.unit.backend.domain_config` + +To add new drivers, subclass the base class at ``test_backend.py`` (look +towards ``test_backend_sql.py`` for examples) and update the configuration of +the test class in ``setUp()``. + +For example, ``test_backend.py`` has a sequence of tests under the class +:class:`~keystone.tests.unit.test_backend.IdentityTests` that will work with +the default drivers as configured in this project's etc/ directory. +``test_backend_sql.py`` subclasses those tests, changing the configuration by +overriding with configuration files stored in the ``tests/unit/config_files`` +directory aimed at enabling the SQL backend for the Identity module. + +:class:`keystone.tests.unit.test_v2_keystoneclient.ClientDrivenTestCase` +uses the installed python-keystoneclient, verifying it against a temporarily +running local keystone instance to explicitly verify basic functional testing +across the API. + +Testing Schema Migrations +------------------------- + +The application of schema migrations can be tested using SQLAlchemy Migrate’s +built-in test runner, one migration at a time. + +.. WARNING:: + + This may leave your database in an inconsistent state; attempt this in + non-production environments only! + +This is useful for testing the *next* migration in sequence (both forward & +backward) in a database under version control: + +.. code-block:: bash + + $ python keystone/common/sql/migrate_repo/manage.py test \ + --url=sqlite:///test.db \ + --repository=keystone/common/sql/migrate_repo/ + +This command references to a SQLite database (test.db) to be used. Depending on +the migration, this command alone does not make assertions as to the integrity +of your data during migration. + +LDAP Tests +---------- + +LDAP has a fake backend that performs rudimentary operations. If you +are building more significant LDAP functionality, you should test against +a live LDAP server. Devstack has an option to set up a directory server for +Keystone to use. Add ldap to the ``ENABLED_SERVICES`` environment variable, +and set environment variables ``KEYSTONE_IDENTITY_BACKEND=ldap`` and +``KEYSTONE_CLEAR_LDAP=yes`` in your ``localrc`` file. + +The unit tests can be run against a live server with +``keystone/tests/unit/test_ldap_livetest.py`` and +``keystone/tests/unit/test_ldap_pool_livetest.py``. The default password is +``test`` but if you have installed devstack with a different LDAP password, +modify the file ``keystone/tests/unit/config_files/backend_liveldap.conf`` and +``keystone/tests/unit/config_files/backend_pool_liveldap.conf`` to reflect your +password. + +.. NOTE:: + To run the live tests you need to set the environment variable + ``ENABLE_LDAP_LIVE_TEST`` to a non-negative value. + +"Work in progress" Tests +------------------------ + +Work in progress (WIP) tests are very useful in a variety of situations +including: + +* During a TDD process they can be used to add tests to a review while + they are not yet working and will not cause test failures. (They should + be removed before the final merge.) +* Often bug reports include small snippets of code to show broken + behaviors. Some of these can be converted into WIP tests that can later + be worked on by a developer. This allows us to take code that can be + used to catch bug regressions and commit it before any code is + written. + +The :func:`keystone.tests.unit.utils.wip` decorator can be used to mark a test +as WIP. A WIP test will always be run. If the test fails then a TestSkipped +exception is raised because we expect the test to fail. We do not pass +the test in this case so that it doesn't count toward the number of +successfully run tests. If the test passes an AssertionError exception is +raised so that the developer knows they made the test pass. This is a +reminder to remove the decorator. + +The :func:`~keystone.tests.unit.utils.wip` decorator requires that the author +provides a message. This message is important because it will tell other +developers why this test is marked as a work in progress. Reviewers will +require that these messages are descriptive and accurate. + +.. NOTE:: + The :func:`~keystone.tests.unit.utils.wip` decorator is not a replacement for + skipping tests. + +.. code-block:: python + + @wip('waiting on bug #000000') + def test(): + pass + +.. NOTE:: + Another strategy is to not use the wip decorator and instead show how the + code currently incorrectly works. Which strategy is chosen is up to the + developer. + +API & Scenario Tests +-------------------- + +Keystone provides API and scenario tests via a `tempest plugin`_ located at +:func:`~keystone.keystone_tempest_plugin`. This tempest plugin is mainly +intended for specific scenarios that require a special deployment, such as +the tests for the ``Federated Identity`` feature. For the deployment of these +scenarios, keystone also provides a `devstack plugin`_. + +For example, to setup a working federated environment, add the following lines +in your `devstack` `local.conf`` file: + +.. code-block:: bash + + [[local|localrc]] + enable_plugin keystone git://git.openstack.org/openstack/keystone + enable_service keystone-saml2-federation + +Finally, to run keystone's API and scenario tests, deploy `tempest`_ with +`devstack`_ (using the configuration above) and then run the following command +from the tempest directory: + +.. code-block:: bash + + tox -e all-plugin -- keystone_tempest_plugin + +.. NOTE:: + Most of keystone's API tests are implemented in `tempest`_ and it is usually + the correct place to add new tests. + +.. _devstack: https://git.openstack.org/cgit/openstack-dev/devstack +.. _devstack plugin: https://docs.openstack.org/developer/devstack/plugins.html +.. _tempest: https://git.openstack.org/cgit/openstack/tempest +.. _tempest plugin: https://docs.openstack.org/developer/tempest/plugin.html + +Writing new API & Scenario Tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When writing tests for the keystone tempest plugin, we should follow the +official tempest guidelines, details about the guidelines can be found at the +`tempest coding guide`_. There are also specific guides for the API and +scenario tests: `Tempest Field Guide to API tests`_ and +`Tempest Field Guide to Scenario tests`_. + +The keystone tempest plugin also provides a base class. For most cases, the +tests should inherit from it: +:class:`keystone_tempest_plugin.tests.base.BaseIdentityTest`. This class +already setups the identity API version and is the container of all API +services clients. +New API services clients :mod:`keystone_tempest_plugin.services` +(which are used to communicate with the REST API from +the services) should also be added to this class. For example, below we have a +snippet from the tests at +:py:mod:`keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.py`. + +.. code-block:: python + + class IdentityProvidersTest(base.BaseIdentityTest): + + ... + + def _create_idp(self, idp_id, idp_ref): + idp = self.idps_client.create_identity_provider( + idp_id, **idp_ref)['identity_provider'] + self.addCleanup( + self.idps_client.delete_identity_provider, idp_id) + return idp + + @decorators.idempotent_id('09450910-b816-4150-8513-a2fd4628a0c3') + def test_identity_provider_create(self): + idp_id = data_utils.rand_uuid_hex() + idp_ref = fixtures.idp_ref() + idp = self._create_idp(idp_id, idp_ref) + + # The identity provider is disabled by default + idp_ref['enabled'] = False + + # The remote_ids attribute should be set to an empty list by default + idp_ref['remote_ids'] = [] + + self._assert_identity_provider_attributes(idp, idp_id, idp_ref) + +The test class extends +:class:`keystone_tempest_plugin.tests.base.BaseIdentityTest`. Also, the +``_create_idp`` method calls keystone's API using the ``idps_client``, +which is an instance from. +:class:`keystone_tempest_plugin.tests.services.identity.v3.identity_providers_client.IdentityProvidersClient`. + +Additionally, to illustrate the construction of a new test class, below we have +a snippet from the scenario test that checks the complete federated +authentication workflow ( +:py:mod:`keystone_tempest_plugin.tests.scenario.test_federated_authentication.py`). +In the test setup, all of the needed resources are created using the API +service clients. Since it is a scenario test, it is common to need some +customized settings that will come from the environment (in this case, from +the devstack plugin) - these settings are collected in the ``_setup_settings`` +method. + +.. code-block:: python + + class TestSaml2EcpFederatedAuthentication(base.BaseIdentityTest): + + ... + + def _setup_settings(self): + self.idp_id = CONF.fed_scenario.idp_id + self.idp_url = CONF.fed_scenario.idp_ecp_url + self.keystone_v3_endpoint = CONF.identity.uri_v3 + self.password = CONF.fed_scenario.idp_password + self.protocol_id = CONF.fed_scenario.protocol_id + self.username = CONF.fed_scenario.idp_username + + ... + + def setUp(self): + super(TestSaml2EcpFederatedAuthentication, self).setUp() + self._setup_settings() + + # Reset client's session to avoid getting garbage from another runs + self.saml2_client.reset_session() + + # Setup identity provider, mapping and protocol + self._setup_idp() + self._setup_mapping() + self._setup_protocol() + +Finally, the tests perform the complete workflow of the feature, asserting its +correctness in each step: + +.. code-block:: python + + def _request_unscoped_token(self): + resp = self.saml2_client.send_service_provider_request( + self.keystone_v3_endpoint, self.idp_id, self.protocol_id) + self.assertEqual(http_client.OK, resp.status_code) + saml2_authn_request = etree.XML(resp.content) + + relay_state = self._str_from_xml( + saml2_authn_request, self.ECP_RELAY_STATE) + sp_consumer_url = self._str_from_xml( + saml2_authn_request, self.ECP_SERVICE_PROVIDER_CONSUMER_URL) + + # Perform the authn request to the identity provider + resp = self.saml2_client.send_identity_provider_authn_request( + saml2_authn_request, self.idp_url, self.username, self.password) + self.assertEqual(http_client.OK, resp.status_code) + saml2_idp_authn_response = etree.XML(resp.content) + + idp_consumer_url = self._str_from_xml( + saml2_idp_authn_response, self.ECP_IDP_CONSUMER_URL) + + # Assert that both saml2_authn_request and saml2_idp_authn_response + # have the same consumer URL. + self.assertEqual(sp_consumer_url, idp_consumer_url) + + ... + + + @testtools.skipUnless(CONF.identity_feature_enabled.federation, + "Federated Identity feature not enabled") + def test_request_unscoped_token(self): + self._request_unscoped_token() + +Notice that the ``test_request_unscoped_token`` test only executes if the the +``federation`` feature flag is enabled. + +.. NOTE:: + For each patch submitted upstream, all of the tests from the keystone + tempest plugin are executed in the + ``gate-keystone-dsvm-functional-v3-only-*`` job. + +.. _Tempest Field Guide to Scenario tests: https://docs.openstack.org/developer/tempest/field_guide/scenario.html +.. _Tempest Field Guide to API tests: https://docs.openstack.org/developer/tempest/field_guide/api.html +.. _tempest coding guide: https://docs.openstack.org/developer/tempest/HACKING.html diff --git a/doc/source/devref/translated-responses.rst b/doc/source/devref/translated-responses.rst new file mode 100644 index 000000000..cf860eff6 --- /dev/null +++ b/doc/source/devref/translated-responses.rst @@ -0,0 +1,52 @@ +.. + Copyright 2011-2012 OpenStack Foundation + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +Translated responses +==================== + +The Keystone server can provide error responses translated into the language in +the ``Accept-Language`` header of the request. In order to test this in your +development environment, there's a couple of things you need to do. + +1. Build the message files. Run the following command in your keystone + directory: + +.. code-block:: bash + + $ python setup.py compile_catalog + +This will generate .mo files like keystone/locale/[lang]/LC_MESSAGES/[lang].mo + +2. When running Keystone, set the ``KEYSTONE_LOCALEDIR`` environment variable + to the keystone/locale directory. For example: + +.. code-block:: bash + + $ KEYSTONE_LOCALEDIR=/opt/stack/keystone/keystone/locale uwsgi --http 127.0.0.1:35357 --wsgi-file $(which keystone-wsgi-admin) + + +Now you can get a translated error response: + +.. code-block:: bash + + $ curl -s -H "Accept-Language: zh" http://localhost:5000/notapath | python -mjson.tool + { + "error": { + "code": 404, + "message": "\u627e\u4e0d\u5230\u8cc7\u6e90\u3002", + "title": "Not Found" + } + } diff --git a/doc/source/getting-started/installing.rst b/doc/source/getting-started/installing.rst index 325091b1e..51c8097ff 100644 --- a/doc/source/getting-started/installing.rst +++ b/doc/source/getting-started/installing.rst @@ -21,8 +21,7 @@ Installing Keystone This document describes how to install Keystone in order to use it. If you are intending to develop on or with Keystone, please read -:doc:`../devref/development_best_practices` and -:doc:`../devref/development_environment`. +:doc:`../devref/index` and :doc:`../devref/development_environment`. Installing from Source ----------------------