diff --git a/doc/source/contributor/additional-roles.rst b/doc/source/contributor/additional-roles.rst deleted file mode 100644 index 4804f25110..0000000000 --- a/doc/source/contributor/additional-roles.rst +++ /dev/null @@ -1,372 +0,0 @@ -================================== -Contributing to Roles and Services -================================== - -If you would like to contribute towards a role to introduce an OpenStack -or infrastructure service, or to improve an existing role, the -OpenStack-Ansible project would welcome that contribution and your assistance -in maintaining it. - -Recommended procedure to develop a role ---------------------------------------- - -#. Deploy OpenStack-Ansible (possibly using - `an AIO`_ - deploy) so that you have the rest of an OpenStack cluster to integrate with - in your testing. -#. Deploy your service on another VM, or possibly directly on the AIO host, by - hand. Configure the service to coordinate with the OpenStack cluster - appropriately. When all the related systems are communicating with each - other you can use the resulting configuration as a reference later. -#. Develop a role for your service. A recommended process is detailed below. - -.. _an AIO: quickstart-aio.html - -Writing a new role ------------------- - -Here are the steps to write the role: - -#. You can review roles which may be currently in development by checking our - `specs repository`_ and `unmerged specs`_ on review.openstack.org. If you - do not find a spec for the role, propose a blueprint/spec `(see also the - spec template)`_ outlining the new Role. By proposing a draft spec you can - help the OpenStack-Ansible community keep track of what roles are being - developed and perhaps connect you with others who may be interested and - able to help you in the process. -#. Create a source repository (e.g. on Github) to start your work on the Role. -#. Generate the reference directory structure for an Ansible role which is - the necessary subset of the documented `Best Practice`_. You might use - Ansible Galaxy tools to do this for you (e.g. ``ansible-galaxy init``). - You may additionally want to include directories such as ``docs`` and - ``examples`` and ``tests`` for your role. -#. Generate a meta/main.yml right away. This file is important to Ansible to - ensure your dependent roles are installed and available and provides others - with the information they will need to understand the purpose of your role. - -#. Develop task files for each of the install stages in turn, creating any - handlers and templates as needed. Ensure that you notify handlers after any - task which impacts the way the service would run (such as configuration - file modifications). Also take care that file ownership and permissions are - appropriate. - - .. HINT:: Fill in variable defaults, libraries, and prerequisites as you - discover a need for them. You can also develop documentation for your - role at the same time. - -#. Add tests to the role. -#. Ensuring the role matches OpenStack-Ansible's latest standards. -#. Deploying the role on an AIO. - -Writing tasks in a role -^^^^^^^^^^^^^^^^^^^^^^^ - -Most OpenStack services will follow a common series of stages to install or -update a service deployment. This is apparent when you review `tasks/main.yml` -for existing roles. - -#. pre-install: prepare the service user group and filesystem directory paths - on the host or container -#. install: install system packages, prepare the (optional) service virtual - environment, install service and requirements (into a virtual environment) -#. post-install: apply all configuration files -#. service add: register the service (each of: service type, service project, - service user, and endpoints) within Keystone's service catalog. -#. service setup: install a service-startup script (init, upstart, systemd, - etc.) so that the service will start up when the container or host next - starts. -#. service init/startup: signal to the host or container to start the services, - make sure the service runs on boot. - -There may be other specialized steps required by some services but most of the -roles will perform all of these at a minimum. Begin by reviewing a role for a -service that has something in common with your service and think about how you -can fit most of the common service setup and configuration steps into that -model. - -.. HINT:: Following the patterns you find in other roles can help ensure your role - is easier to use and maintain. - -.. _(see also the spec template): https://git.openstack.org/cgit/openstack/openstack-ansible-specs/tree/specs/templates/template.rst -.. _specs repository: https://git.openstack.org/cgit/openstack/openstack-ansible-specs -.. _unmerged specs: https://review.openstack.org/#/q/status:+open+project:openstack/openstack-ansible-specs -.. _Best Practice: https://docs.ansible.com/ansible/playbooks_best_practices.html#directory-layout - -Keep in mind a role candidate for inclusion should respect our -`Ansible Style Guide`_. - -.. _Ansible Style Guide: contribute.html#ansible-style-guide - -Adding tests to a role -^^^^^^^^^^^^^^^^^^^^^^ - -Each of the role tests is in its tests/ folder. - -This folder contains at least the following files: - -#. ``test.yml`` ("super" playbook acting as test router to sub-playbooks) -#. ``-overrides.yml``. This var file is automatically loaded - by our shell script in our `tests repository`_. -#. ``inventory``. A static inventory for role testing. - It's possible some roles have multiple inventories. See for example the - neutron role with its ``lxb_inventory``, ``calico_inventory``. -#. ``group_vars`` and ``host_vars``. These folders will hold override the - necessary files for testing. For example, this is where you override - the IP addresses, IP ranges, and ansible connection details. -#. ``ansible-role-requirements.yml``. This should be fairly straightforward: - this file contains all the roles to clone before running your role. - The roles' relative playbooks will have to be listed in the ``test.yml`` - file. However, keep in mind to NOT re-invent the wheel. For example, - if your role needs keystone, you don't need to create your own keystone - install playbook, because we have a generic keystone install playbook - in the `tests repository`. - -.. _tests repository: https://git.openstack.org/cgit/openstack/openstack-ansible-tests - -Ensuring the role matches OpenStack-Ansible's standards -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -#. To facilitate the development and tests implemented across all - OpenStack-Ansible roles, a base set of folders and files need to be - implemented. A base set of configuration and test facilitation scripts must - include at least the following: - - * ``tox.ini``: - The lint testing, documentation build, release note build and - functional build execution process for the role's gate tests are all - defined in this file. - * ``test-requirements.txt``: - The Python requirements that must be installed when executing the - tests. - * ``bindep.txt``: - The binary requirements that must be installed on the host the tests - are executed on for the Python requirements and the tox execution to - work. This must be copied from the - ``openstack-ansible-tests`` repository and will be automatically - be overriden by our proposal bot should any change happen. - * ``setup.cfg`` and ``setup.py``: - Information about the repository used when building artifacts. - * ``run_tests.sh``: - A script for developers to execute all standard tests on a - suitable host. This must be copied from the - ``openstack-ansible-tests`` repository and will be automatically - be overriden by our proposal bot should any change happen. - * ``Vagrantfile``: - A configuration file to allow a developer to easily create a - test virtual machine using `Vagrant`_. This must automatically execute - ``run_tests.sh``. This must be copied from the - ``openstack-ansible-tests`` repository and will be automatically - be overriden by our proposal bot should any change happen. - * ``README.rst``, ``LICENSE``, ``CONTRIBUTING.rst``: - A set of standard files whose content is self-explanatory. - * ``.gitignore``: - A standard git configuration file for the repository which should be - pretty uniform across all the repositories. This must be copied from the - ``openstack-ansible-tests`` repository and will be automatically - be overriden by our proposal bot should any change happen. - * ``.gitreview``: - A standard file configured for the project to inform the ``git-review`` - plugin where to find the upstream gerrit remote for the repository. - * ``tests/tests-repo-clone.sh`` needs to be copied from the - ``openstack-ansible-tests`` repository. - - Please have a look at a role like os_cinder, os_keystone, or os_neutron - for latest files. - -#. The role development should initially be focused on implementing a set of - tasks and a test playbook which converge. The convergence must: - - * Implement ``developer_mode`` to build from a git source into a Python - virtual environment. - * Deploy the applicable configuration files in the right places. - * Ensure that the service starts. - - The convergence may involve consuming other OpenStack-Ansible roles (For - example: ``galera_server``, ``galera_client``, ``rabbitmq_server``) in - order to ensure that the appropriate infrastructure is in place. Re-using - existing roles in OpenStack-Ansible or Ansible Galaxy is strongly - encouraged. - -#. The role *must* support Ubuntu 16.04 LTS. It should - ideally also support CentOS 7 and openSUSE 42.X but this is not required - at this time. The patterns to achieve this include: - - * The separation of platform specific variables into role vars files. - * The detection and handling of different init systems (init.d, SystemD). - * The detection and handling of different package managers (apt, yum). - * The detection and handling of different network configuration methods. - - There are several examples of these patterns implemented across many of - the OpenStack-Ansible roles. Developers are advised to inspect the - established patterns and either implement or improve upon them. - -#. The role implementation should be done in such a way that it is agnostic - with regards to whether it is implemented in a container, or on a - physical host. The test infrastructure may make use of LXC containers for - the separation of services, but if a role is used by a playbook that - targets a host, it must work regardless of whether that host is a - container, a virtual server, or a physical server. The use of LXC - containers for role tests is not required but it may be useful in order - to simulate a multi-node build out as part of the testing infrastructure. - -#. Any secrets (For example: passwords) should not be provided with default - values in the tasks, role vars, or role defaults. The tasks should be - implemented in such a way that any secrets required, but not provided, - should result in the task execution failure. It is important for a - secure-by-default implementation to ensure that an environment is not - vulnerable due to the production use of default secrets. Deployers - must be forced to properly provide their own secret variable values. - -#. Once the initial convergence is working and the services are running, - the role development should focus on implementing some level of - functional testing. Ideally, the functional tests for an OpenStack role - should make use of Tempest to execute the functional tests. The ideal - tests to execute are scenario tests as they test the functions that - the service is expected to do in a production deployment. In the absence - of any scenario tests for the service a fallback option is to implement - the smoke tests instead. - -#. The role must include documentation. The `Documentation and Release Note - Guidelines`_ provide specific guidelines with regards to style and - conventions. The documentation must include a description of the - mandatory infrastructure (For example: a database and a message queue are - required), variables (For example: the database name and credentials) and - group names (For example: The role expects a group named ``foo_all`` to - be present and it expects the host to be a member of it) for the role's - execution to succeed. - - .. _Documentation and Release Note Guidelines: contribute.html#documentation-and-release-note-guidelines - .. _Vagrant: https://www.vagrantup.com/ - -Deploying the role -^^^^^^^^^^^^^^^^^^ - -#. Include your role on the deploy host. See also `Adding Galaxy roles`_. -#. Perform any other host preparation (such as the tasks performed by the - ``bootstrap-aio.yml`` playbook). This includes any preparation tasks that - are particular to your service. -#. Generate files to include your service in the Ansible inventory - using `env.d`_ and `conf.d`_ files for use on your deploy host. - - .. HINT:: You can follow examples from other roles, making the appropriate - modifications being sure that group labels in ``env.d`` and ``conf.d`` - files are consistent. - - .. HINT:: A description of how these work can be - found in :deploy_guide:`Appendix C ` - of the Deployment Guide. - -#. Generate secrets, if any, as described in the :deploy_guide:`Configure - Service Credentials `. - You can append your keys to an existing ``user_secrets.yml`` file or add a - new file to the ``openstack_deploy`` directory to contain them. Provide - overrides for any other variables you will need at this time as well, either - in ``user_variables.yml`` or another file. This is explained in more depth - under `Extending OpenStack-Ansible`_. - Any secrets required for the role to work must be noted in the - ``etc/openstack_deploy/user_secrets.yml`` file for reuse by other users. - -#. If your service is installed from source or relies on python packages which - need to be installed from source, specify a repository for the source - code of each requirement by adding a file to your deploy host under - ``playbooks/defaults/repo_packages`` in the OpenStack-Ansible source - repository and following the pattern of files currently in that directory. - You could also simply add an entry to an existing file there. Be sure to - run the ``repo-build.yml`` play later so that wheels for your packages will - be included in the repository infrastructure. -#. Make any required adjustments to the load balancer configuration - (e.g. modify ``inventory/group_vars/all/haproxy.yml`` in the - OpenStack-Ansible source repository on your deploy host) so that your - service can be reached through a load balancer, if appropriate, and be sure - to run the ``haproxy-install.yml`` play later so your changes will be - applied. Please note, you can also use ``haproxy_extra_services`` variable - if you don't want to provide your service as default for everyone. -#. Put together a service install playbook file for your role. This can also - be modeled from any existing service playbook that has similar - dependencies to your service (database, messaging, storage drivers, - container mount points, etc.). A common place to keep playbook files in a - Galaxy role is in an ``examples`` directory off the root of the role. - If the playbook is meant for installing an OpenStack service, name it - ``os--install.yml`` and target it at the appropriate - group defined in the service ``env.d`` file. - It is crucial that the implementation of the service is optional and - that the deployer must opt-in to the deployment through the population - of a host in the applicable host group. If the host group has no - hosts, Ansible skips the playbook's tasks automatically. -#. Any variables needed by other roles to connect to the new role, or by the - new role to connect to other roles, should be implemented in - ``inventory/group_vars``. The group vars are essentially the - glue which playbooks use to ensure that all roles are given the - appropriate information. When group vars are implemented it should be a - minimum set to achieve the goal of integrating the new role into the - integrated build. -#. Documentation must be added in the role to describe how to implement - the new service in an integrated environement. This content must - adhere to the `Documentation and Release Note Guidelines`_. Until the - role has integrated functional testing implemented (see also the - Role development maturity paragraph), the documentation - must make it clear that the service inclusion in OpenStack-Ansible is - experimental and is not fully tested by OpenStack-Ansible in an - integrated build. -#. A feature release note must be added to announce the new service - availability and to refer to the role documentation for further - details. This content must adhere to the - `Documentation and Release Note Guidelines`_. -#. It must be possible to execute a functional, integrated test which - executes a deployment in the same way as a production environment. The - test must execute a set of functional tests using Tempest. This is the - required last step before a service can remove the experimental warning - from the documentation. - -.. HINT:: If you adhere to the pattern of isolating your role's extra - deployment requirements (secrets and var files, HAProxy yml fragments, - repo_package files, etc.) in their own files it makes it easy for you to - automate these additional steps when testing your role. - -.. _Adding Galaxy roles: extending.html#adding-galaxy-roles -.. _env.d: extending.html#env-d -.. _conf.d: extending.html#conf-d -.. _Extending OpenStack-Ansible: extending.html#user-yml-files - -Role development maturity -------------------------- - -A role may be fully mature, even if it is not integrated in the -``openstack-ansible`` repository. - -A role can be in one of the four maturity levels: - -* ``Complete`` -* ``Incubated`` -* ``Unmaintained`` -* ``Retired`` - -Here are a series of rules that define maturity levels: - -* A role can be retired at any time if it is not relevant anymore. -* A role can be ``Incubated`` for maximum 2 cycles. -* An ``Incubated`` role that passes functional testing will be upgraded - to the ``Complete`` status, and cannot return in ``Incubated`` status. -* An ``Incubated`` role that didn't implement functional testing in - the six month timeframe will become ``Unmaintained``. -* A role in ``Complete`` status can be downgraded to ``Unmaintained``. - status, according to the maturity downgrade procedure. - -Maturity downgrade procedure -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If a role has failed periodics or gate test for two weeks, a bug -should be filed, and a message to the mailing list will be sent, -referencing the bug. - -The next community meeting should discuss about role deprecation, -and if no contributor comes forward to fix the role, periodic -testing will be turned off, and the role will move to an -``unmaintained`` state. - -Maturity Matrix -^^^^^^^^^^^^^^^ - -View the following role maturity table to see each role's status. - -.. _role maturity table: role-maturity.html diff --git a/doc/source/contributor/bug-triage.rst b/doc/source/contributor/bugs.rst similarity index 77% rename from doc/source/contributor/bug-triage.rst rename to doc/source/contributor/bugs.rst index 489732c7ff..35597a622b 100644 --- a/doc/source/contributor/bug-triage.rst +++ b/doc/source/contributor/bugs.rst @@ -1,9 +1,60 @@ -============================ -OpenStack-Ansible Bug Triage -============================ +============================== +OpenStack-Ansible Bug Handling +============================== + +.. _bug_reporting: + +Bug Reporting +============= + +Bugs should be filed on the `OpenStack-Ansible Launchpad project`_. + +When submitting a bug, or working on a bug, please ensure the following +criteria are met: + +* The description clearly states or describes the original problem or root + cause of the problem. +* The description clearly states the expected outcome of the user action. +* Include historical information on how the problem was identified. +* Any relevant logs or user configuration are included, either directly + or through a pastebin. +* If the issue is a bug that needs fixing in a branch other than master, + please note the associated branch within the launchpad issue. +* The provided information should be totally self-contained. External access + to web services/sites should not be needed. +* Steps to reproduce the problem if possible. + +.. _OpenStack-Ansible Launchpad project: https://bugs.launchpad.net/openstack-ansible + +Bug Tags +^^^^^^^^ +If the reported needs fixing in a branch in addition to master, add a +'\-backport-potential' tag (e.g. ``liberty-backport-potential``). +There are predefined tags that will auto-complete. + +Status +^^^^^^ +Please leave the **status** of an issue alone until someone confirms it or +a member of the bugs team triages it. While waiting for the issue to be +confirmed or triaged the status should remain as **New**. + +Importance +^^^^^^^^^^ +Should only be touched if it is a Blocker/Gating issue. If it is, please +set to **High**, and only use **Critical** if you have found a bug that +can take down whole infrastructures. Once the importance has been changed +the status should be changed to **Triaged** by someone other than the bug +creator. + +The triaging process is explained here below. + +.. _bug_triage: + +Bug triage +========== What is a bug triage -~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^ "Bug triage is a process where tracker issues are screened and prioritised. Triage should help ensure we appropriately manage all @@ -24,7 +75,7 @@ The bug triage practices in OpenStack-Ansible are based on the .. _OpenStack Bug Triage Documentation: https://docs.openstack.org/infra/manual/developers.html#working-on-bugs Bug classification -~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^ The **Status** of a bug is one of the following: @@ -53,7 +104,7 @@ The **Importance** of a bug is one of the following: * *Undecided* - Not decided yet. It might need more discussion. Bug triage meeting duties -~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^ If the bug description is incomplete, or the report is lacking the information necessary to reproduce the issue, ask the reporter to @@ -75,7 +126,7 @@ supervisors rights to also prioritize bugs per importance (on top of classifying them on status). Bug skimming duty -~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^ To help triaging bugs, one person of the bug team can be on "bug skimming duty". @@ -116,7 +167,7 @@ skimming duty". an importance set. Bug skimming duty weekly checklist -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Prioritize or reprioritize OpenStack-Ansible `confirmed bugs`_. diff --git a/doc/source/contributor/code-rules.rst b/doc/source/contributor/code-rules.rst new file mode 100644 index 0000000000..3e822117ec --- /dev/null +++ b/doc/source/contributor/code-rules.rst @@ -0,0 +1,382 @@ +.. _code_rules: + +========== +Code rules +========== + +.. _codeguidelines: + +General Guidelines for Submitting Code +====================================== + +* Write good commit messages. We follow the OpenStack + "`Git Commit Good Practice`_" guide. if you have any questions regarding how + to write good commit messages please review the upstream OpenStack + documentation. +* Changes to the project should be submitted for review via the Gerrit tool, + following the `workflow documented here`_. +* Pull requests submitted through GitHub will be ignored and closed without + regard. +* Patches should be focused on solving one problem at a time. If the review is + overly complex or generally large the initial commit will receive a "**-2**" + and the contributor will be asked to split the patch up across multiple + reviews. In the case of complex feature additions the design and + implementation of the feature should be done in such a way that it can be + submitted in multiple patches using dependencies. Using dependent changes + should always aim to result in a working build throughout the dependency + chain. Documentation is available for `advanced gerrit usage`_ too. +* All patch sets should adhere to the :ref:`Ansible Style Guide` listed here as + well as adhere to the `Ansible best practices`_ when possible. +* All changes should be clearly listed in the commit message, with an + associated bug id/blueprint along with any extra information where + applicable. +* Refactoring work should never include additional "rider" features. Features + that may pertain to something that was re-factored should be raised as an + issue and submitted in prior or subsequent patches. +* New features, breaking changes and other patches of note must include a + release note generated using `the reno tool`_. Please see the + :ref:`Documentation and Release Note Guidelines` + for more information. +* All patches including code, documentation and release notes should be built + and tested locally with the appropriate test suite before submitting for + review. See :ref:`Development and Testing` + for more information. + +.. _Git Commit Good Practice: https://wiki.openstack.org/wiki/GitCommitMessages +.. _workflow documented here: https://docs.openstack.org/infra/manual/developers.html#development-workflow +.. _advanced gerrit usage: http://www.mediawiki.org/wiki/Gerrit/Advanced_usage +.. _Ansible best practices: http://docs.ansible.com/playbooks_best_practices.html +.. _the reno tool: https://docs.openstack.org/developer/reno/usage.html + +.. _documentation: + +Documentation and Release Note Guidelines +========================================= + +Documentation is a critical part of ensuring that the deployers of +OpenStack-Ansible are appropriately informed about: + +* How to use the project's tooling effectively to deploy OpenStack. +* How to implement the right configuration to meet the needs of their specific + use-case. +* Changes in the project over time which may affect an existing deployment. + +To meet these needs developers must submit +:ref:`code comments`, documentation (see also the +:ref:`documentation locations section`) and +:ref:`release notes` with any code submissions. + +All forms of documentation should comply with the guidelines provided +in the `OpenStack Documentation Contributor +Guide`_, with particular reference to the following sections: + +* Writing style +* RST formatting conventions + +.. _OpenStack Documentation Contributor Guide: https://docs.openstack.org/contributor-guide/ + +.. _codecomments: + +Code Comments +------------- + +Code comments for variables should be used to explain the purpose of the +variable. This is particularly important for the role defaults file as the file +is included verbatim in the role's documentation. Where there is an optional +variable, the variable and an explanation of what it is used for should be +added to the defaults file. + +Code comments for bash/python scripts should give guidance to the purpose of +the code. This is important to provide context for reviewers before the patch +has merged, and for later modifications to remind the contributors what the +purpose was and why it was done that way. + +.. _docslocations: + +Documentation locations +----------------------- + +OpenStack-Ansible has multiple forms of documentation with different intent. + +The :deploy_guide:`Deployment Guide ` +intends to help deployers deploy OpenStack-Ansible for the first time. + +The :dev_docs:`User Guide ` intends to provide user +stories on how to do specific things with OpenStack-Ansible. + +The :dev_docs:`Operations Guide ` provide help +on how to manage and operate OpenStack-Ansible. + +The in-depth technical information is located in the +:dev_docs:`OpenStack-Ansible Reference `. + +The role documentation (for example, the `keystone role documentation`_) +intends to explain all the options available for the role and how to implement +more advanced requirements. To reduce duplication, the role documentation +directly includes the role's default variables file which includes the +comments explaining the purpose of the variables. The long hand documentation +for the roles should focus less on explaining variables and more on explaining +how to implement advanced use cases. + +The role documentation must include a description of the mandatory +infrastructure (For example: a database and a message queue are required), +variables (For example: the database name and credentials) and group names +(For example: The role expects a group named ``foo_all`` to +be present and it expects the host to be a member of it) for the role's +execution to succeed. + +Where possible the documentation in OpenStack-Ansible should steer clear of +trying to explain OpenStack concepts. Those explanations belong in the +OpenStack Manuals or service documentation and OpenStack-Ansible documentation +should link to those documents when available, rather than duplicate their +content. + +.. _keystone role documentation: https://docs.openstack.org/developer/openstack-ansible-os_keystone/ + +.. _reno: + +Release Notes +------------- + +Release notes are generated using `the reno tool`_. Release notes must be +written with the following guidelines in mind: + +* Each list item must make sense to read without the context of the patch or + the repository the patch is being submitted into. The reason for this is that + all release notes are consolidated and presented in a long list without + reference to the source patch or the context of the repository. +* Each note should be brief and to the point. Try to avoid multi-paragraph + notes. For features the note should typically refer to documentation for more + details. For bug fixes the note can refer to a registered bug for more + details. + +In most cases only the following sections should be used for new release notes +submitted with patches: + +* ``features``: This should inform the deployer briefly about a new feature and + should describe how to use it either by referencing the variables to set or + by referring to documentation. +* ``issues``: This should inform the deployer about known issues. This may be + used when fixing an issue and wanting to inform deployers about a workaround + that can be used for versions prior to that which contains the patch that + fixes the issue. Issue notes should specifically make mention of what + versions of OpenStack-Ansible are affected by the issue. +* ``upgrade``: This should inform the deployer about changes which may affect + them when upgrading from a previous major or minor version. Typically, these + notes would describe changes to default variable values or variables that + have been removed. +* ``deprecations``: If a variable has been deprecated (ideally using the + deprecation filter), then it should be communicated through notes in this + section. Note that if a variable has been removed entirely then it has not + been deprecated and the removal should be noted in the ``upgrade`` section. + +.. _specs: + +Submitting a specification +========================== + +By proposing a draft spec you can help the OpenStack-Ansible +community keep track of what roles or large changes are being developed, +and perhaps connect you with others who may be interested and able +to help you in the process. + +Our specifications repository follows the usual OpenStack and +OpenStack-Ansible guidelines for submitting code. + +However, to help you in the writing of the specification, we have a +`specification template`_ that can be copied into the latest release +name folder. Rename and edit it for your needs. + +.. _specification template: https://git.openstack.org/cgit/openstack/openstack-ansible-specs/tree/specs/templates/template.rst + +.. _Ansible Style Guide: + +Ansible Style Guide +=================== + +YAML formatting +--------------- + +When creating tasks and other roles for use in Ansible please create them +using the YAML dictionary format. + +Example YAML dictionary format: + +.. code-block:: yaml + + - name: The name of the tasks + module_name: + thing1: "some-stuff" + thing2: "some-other-stuff" + tags: + - some-tag + - some-other-tag + + +Example what **NOT** to do: + +.. code-block:: yaml + + - name: The name of the tasks + module_name: thing1="some-stuff" thing2="some-other-stuff" + tags: some-tag + +.. code-block:: yaml + + - name: The name of the tasks + module_name: > + thing1="some-stuff" + thing2="some-other-stuff" + tags: some-tag + + +Usage of the ">" and "|" operators should be limited to Ansible conditionals +and command modules such as the Ansible ``shell`` or ``command``. + +Tags and tags conventions +------------------------- + +Tags are assigned based on the relevance of each individual item. +Higher level includes (for example in the ``tasks/main.yml``) need high +level tags. For example, ``*-config`` or ``*-install``. +Included tasks can have more detailed tags. + +The following convention is used: + +* A tag including the word ``install`` handles software installation tasks. + Running a playbook with ``--tags -install`` only deploys the + necessary software on the target, and will not configure it to your + needs or run any service. + +* A tag including the word ``config`` prepares the configuration of the + software (adapted to your needs), and all the components necessary + to run the service(s) configured in the role. Running a playbook with + ``--tags -config`` is only possible if the target already ran + the tags ``-install``. + +* A tag including the word ``upgrade`` handles all the upgrade tasks. + +Variable files conventions +-------------------------- + +The variables files in a role are split in 3 locations: + +#. The `defaults/main.yml` file +#. The `vars/main.yml` file +#. The `vars/.yml` file + +The variables with lower priority should be in the `defaults/main.yml`. +This allows their overriding with group variables or host variables. +A good example for this are default database connection details, default +queues connection details, or debug mode. + +In other words, `defaults/main.yml` contains variables that are meant to +be overridable by a deployer or a continuous integration system. +These variables should be limited as much as possible, to avoid +increasing the test matrix. + +The `vars/main.yml` is always included. It contains generic +variables that aren't meant to be changed by a deployer. This includes +for example static information that aren't distribution specific (like +aggregation of role internal variables for example). + +The `vars/.yml` is the place where distribution +specific content will be stored. For example, this file will hold +the package names, repositories urls and keys, file paths, service +names/init scripts. + +Secrets +^^^^^^^ + +Any secrets (For example: passwords) should not be provided with default +values in the tasks, role vars, or role defaults. The tasks should be +implemented in such a way that any secrets required, but not provided, +should result in the task execution failure. It is important for a +secure-by-default implementation to ensure that an environment is not +vulnerable due to the production use of default secrets. Deployers +must be forced to properly provide their own secret variable values. + +Task files conventions +---------------------- + +Most OpenStack services will follow a common series of stages to +install, configure, or update a service deployment. This is apparent +when you review `tasks/main.yml` for existing roles. + +If developing a new role, please follow the conventions set by +existing roles. + +Tests conventions +----------------- + +The conventions for writing tests are described in the +:ref:`tests` page. + +Other OpenStack-Ansible conventions +----------------------------------- + +To facilitate the development and tests implemented across all +OpenStack-Ansible roles, a base set of folders and files need to be +implemented. A base set of configuration and test facilitation scripts must +include at least the following: + +* ``tox.ini``: + The lint testing, documentation build, release note build and + functional build execution process for the role's gate tests are all + defined in this file. +* ``test-requirements.txt``: + The Python requirements that must be installed when executing the + tests. +* ``bindep.txt``: + The binary requirements that must be installed on the host the tests + are executed on for the Python requirements and the tox execution to + work. This must be copied from the + ``openstack-ansible-tests`` repository and will be automatically + be overriden by our proposal bot should any change happen. +* ``setup.cfg`` and ``setup.py``: + Information about the repository used when building artifacts. +* ``run_tests.sh``: + A script for developers to execute all standard tests on a + suitable host. This must be copied from the + ``openstack-ansible-tests`` repository and will be automatically + be overriden by our proposal bot should any change happen. +* ``Vagrantfile``: + A configuration file to allow a developer to easily create a + test virtual machine using `Vagrant`_. This must automatically execute + ``run_tests.sh``. This must be copied from the + ``openstack-ansible-tests`` repository and will be automatically + be overriden by our proposal bot should any change happen. +* ``README.rst``, ``LICENSE``, ``CONTRIBUTING.rst``: + A set of standard files whose content is self-explanatory. +* ``.gitignore``: + A standard git configuration file for the repository which should be + pretty uniform across all the repositories. This must be copied from the + ``openstack-ansible-tests`` repository and will be automatically + be overriden by our proposal bot should any change happen. +* ``.gitreview``: + A standard file configured for the project to inform the ``git-review`` + plugin where to find the upstream gerrit remote for the repository. +* ``docs/`` and ``releasenotes/`` folders need to be exist and be + properly configured. + +Please have a look at a role like os_cinder, os_keystone, or os_neutron +for latest files. + +.. _Vagrant: https://www.vagrantup.com/ + +Container technology independence +--------------------------------- + +The role implementation should be done in such a way that it is agnostic +with regards to whether it is implemented in a container, or on a +physical host. The test infrastructure may make use of containers for +the separation of services, but if a role is used by a playbook that +targets a host, it must work regardless of whether that host is a +container, a virtual server, or a physical server. The use of +containers for role tests is not required but it may be useful in order +to simulate a multi-node build out as part of the testing infrastructure. + +Minimum supported distributions +------------------------------- + +See our :ref:`supported-distros` page. diff --git a/doc/source/contributor/contribute.rst b/doc/source/contributor/contribute.rst index 0e75fd07ed..2cf57de14d 100644 --- a/doc/source/contributor/contribute.rst +++ b/doc/source/contributor/contribute.rst @@ -1,97 +1,83 @@ +.. _contributing: + ====================== Contributor Guidelines ====================== -Reporting Bugs -~~~~~~~~~~~~~~ +Before submitting code +====================== -Bugs should be filed on `Bug Launchpad`_ for OpenStack-Ansible. +Before jumping ahead and working on code, a series of steps should +be taken: -When submitting a bug, or working on a bug, please ensure the following -criteria are met: +* Is there a bug for it? Can your track if someone else has seen + the same bug? +* Are you sure nobody is working on this problem at the moment? + Could there be a review pending fixing the same issue? +* Have you checked if your issue/feature request + hasn't been solved in another branch? -* The description clearly states or describes the original problem or root - cause of the problem. -* Include historical information on how the problem was identified. -* Any relevant logs are included. -* If the issue is a bug that needs fixing in a branch other than master, - please note the associated branch within the launchpad issue. -* The provided information should be totally self-contained. External access - to web services/sites should not be needed. -* Steps to reproduce the problem if possible. +If you're willing to submit code, please remember the following rules: -Tags ----- -If it's a bug that needs fixing in a branch in addition to master, add a -'\-backport-potential' tag (e.g. ``liberty-backport-potential``). -There are predefined tags that will auto-complete. +* All code should match our + :ref:`codeguidelines`. +* All code requires to go through our :ref:`reviews`. +* Documentation should be provided with the + code directly. See also :ref:`documentation`. +* Fixing bugs and increasing test coverage have priority to new features. + See also the section :ref:`bugfixing`. +* New features are following a process, explained in the section + :ref:`newfeatures`. + New features are less likely to be :ref:`backported` + to previous branches. -Status ------- -Please leave the **status** of an issue alone until someone confirms it or -a member of the bugs team triages it. While waiting for the issue to be -confirmed or triaged the status should remain as **New**. +.. _reviews: -Importance ----------- -Should only be touched if it is a Blocker/Gating issue. If it is, please -set to **High**, and only use **Critical** if you have found a bug that -can take down whole infrastructures. Once the importance has been changed -the status should be changed to *Triaged* by someone other than the bug -creator. +Review process +============== -The triaging process is explained on the `bug triage documentation`_ page. +Any new code will be reviewed before merging into our repositories. -.. _Bug Launchpad: https://bugs.launchpad.net/openstack-ansible -.. _bug triage documentation: bug-triage.html +We follow openstack guidelines for the code reviewing process. -General Guidelines for Submitting Code -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Please be aware that any patch can be refused by the community if they +don't match the :ref:`codeguidelines`. -* Write good commit messages. We follow the OpenStack - "`Git Commit Good Practice`_" guide. if you have any questions regarding how - to write good commit messages please review the upstream OpenStack - documentation. -* Changes to the project should be submitted for review via the Gerrit tool, - following the `workflow documented here`_. -* Pull requests submitted through GitHub will be ignored and closed without - regard. -* Patches should be focused on solving one problem at a time. If the review is - overly complex or generally large the initial commit will receive a "**-2**" - and the contributor will be asked to split the patch up across multiple - reviews. In the case of complex feature additions the design and - implementation of the feature should be done in such a way that it can be - submitted in multiple patches using dependencies. Using dependent changes - should always aim to result in a working build throughout the dependency - chain. Documentation is available for `advanced gerrit usage`_ too. -* All patch sets should adhere to the `Ansible Style Guide`_ listed here as - well as adhere to the `Ansible best practices`_ when possible. -* All changes should be clearly listed in the commit message, with an - associated bug id/blueprint along with any extra information where - applicable. -* Refactoring work should never include additional "rider" features. Features - that may pertain to something that was re-factored should be raised as an - issue and submitted in prior or subsequent patches. -* New features, breaking changes and other patches of note must include a - release note generated using `the reno tool`_. Please see the - `Documentation and Release Note Guidelines`_ for more information. -* All patches including code, documentation and release notes should be built - and tested locally with the appropriate test suite before submitting for - review. See `Development and Testing`_ for more information. +.. _bugfixing: -.. _Git Commit Good Practice: https://wiki.openstack.org/wiki/GitCommitMessages -.. _workflow documented here: https://docs.openstack.org/infra/manual/developers.html#development-workflow -.. _advanced gerrit usage: http://www.mediawiki.org/wiki/Gerrit/Advanced_usage -.. _Ansible best practices: http://docs.ansible.com/playbooks_best_practices.html -.. _the reno tool: https://docs.openstack.org/developer/reno/usage.html -.. _Development and Testing: scripts.html#development-and-testing +Working on bug fixes +==================== -Working on Features -~~~~~~~~~~~~~~~~~~~ +Any bug fix should have, in its commit message: -* All feature additions/deletions should be accompanied by a blueprint/spec. - e.g. adding additional active agents to neutron, developing a new service - role, etc... + Closes-Bug: #bugnumber + +or + + Related-Bug: #bugnumber + +where #bugnumber refers to a Launchpad issue. + +See also the `working on bugs`_ section of the openstack documentation. + +.. _working on bugs: https://docs.openstack.org/infra/manual/developers.html#working-on-bugs + +.. _newfeatures: + +Working on new features +======================= + +If you would like to contribute towards a role to introduce an OpenStack +or infrastructure service, or to improve an existing role, the +OpenStack-Ansible project would welcome that contribution and your assistance +in maintaining it. + +Here are a few rules to get started: + +* All large feature additions/deletions should be accompanied by a + blueprint/spec. e.g. adding additional active agents to neutron, + developing a new service role, etc... See also + :ref:`specs`. * Before creating blueprint/spec an associated 'Wishlist Bug' can be raised on launchpad. This issue will be triaged and a determination will be made on how large the change is and whether or not the change warrants a @@ -101,12 +87,13 @@ Working on Features * All blueprints/specs should be voted on and approved by core reviewers before any associated code will be merged. For more information on blueprints/specs please review the OpenStack documentation regarding - `Working on Specifications and Blueprints`_. + `Working on Specifications and Blueprints`_ and our own + :ref:`specs`. * Once the blueprint work is completed the author(s) can request a backport of the blueprint work into a stable branch. Each backport will be evaluated on a case by case basis with cautious consideration based on how the - backport affects any existing deployments. See the `Backporting`_ section - for more information. + backport affects any existing deployments. See the + :ref:`backport` section for more information. * Any new OpenStack services implemented which have `Tempest`_ tests available must be implemented along with suitable functional tests enabled as part of the feature development in order to ensure that any changes @@ -115,12 +102,110 @@ Working on Features OpenStack documentation about what the feature is and how it works. The documentation should then describe how it is implemented in OpenStack-Ansible and what configuration options there are. + See also the :ref:`documentation` section. .. _Working on Specifications and Blueprints: https://docs.openstack.org/infra/manual/developers.html#working-on-specifications-and-blueprints .. _Tempest: https://docs.openstack.org/developer/tempest/ + +Example process to develop a new role +------------------------------------- + +Here are the steps to write the role: + +#. You can review roles which may be currently in development by checking our + `specs repository`_ and `unmerged specs`_ on review.openstack.org. If you + do not find a spec for the role, propose a blueprint/spec. + See also :ref:`specs`. +#. Create a source repository (e.g. on Github) to start your work on the Role. +#. Generate the reference directory structure for an Ansible role which is + the necessary subset of the documented `Best Practice`_. You might use + Ansible Galaxy tools to do this for you (e.g. ``ansible-galaxy init``). + You may additionally want to include directories such as ``docs`` and + ``examples`` and ``tests`` for your role. +#. Generate a meta/main.yml right away. This file is important to Ansible to + ensure your dependent roles are installed and available and provides others + with the information they will need to understand the purpose of your role. + +#. Develop task files for each of the install stages in turn, creating any + handlers and templates as needed. Ensure that you notify handlers after any + task which impacts the way the service would run (such as configuration + file modifications). Also take care that file ownership and permissions are + appropriate. + + .. HINT:: Fill in variable defaults, libraries, and prerequisites as you + discover a need for them. You can also develop documentation for your + role at the same time. + +#. Add tests to the role. See also our :ref:`tests` page. +#. Ensuring the role matches OpenStack-Ansible's latest standards. + See also our :ref:`code_rules` page. +#. Ensure the role converges: + + * Implement **developer_mode** to build from a git source into + a Python virtual environment. + * Deploy the applicable configuration files in the right places. + * Ensure that the service starts. + + The convergence may involve consuming other OpenStack-Ansible roles + (For example: **galera_server, galera_client, rabbitmq_server**) + in order to ensure that the appropriate infrastructure is in place. + Re-using existing roles in OpenStack-Ansible or Ansible Galaxy is + strongly encouraged. +#. Once the initial convergence is working and the services are running, + the role development should focus on implementing some level of + functional testing. See also :ref:`tempest-testing`. +#. Test the role on a new machine, using our provided scripts. +#. Submit your role for review. +#. If required, ask the OpenStack-Ansible PTL to import the github + role into the openstack-ansible namespace (This can only be done + early in the development cycle, and may be postponed to next + cycle). +#. If necessary, work on the integration within the + openstack-ansible integrated repository, and deploy + the role on an AIO. See also :ref:`integrate-new-role-with-aio`. + +.. _specs repository: https://git.openstack.org/cgit/openstack/openstack-ansible-specs +.. _unmerged specs: https://review.openstack.org/#/q/status:+open+project:openstack/openstack-ansible-specs +.. _Best Practice: https://docs.ansible.com/ansible/playbooks_best_practices.html#directory-layout + +Example process for adding a feature to an existing role +-------------------------------------------------------- + +#. Search for in the `OpenStack-Ansible Launchpad project`_ for + the feature request. +#. If no "Wishlist" item exist in Launchpad for your feature, create + a bug for it. Don't hesitate to ask if a spec is required in + the bug. +#. The :ref:`bug_triage` will classify if this new feature requires + a spec or not. +#. Work on the role files, following our :ref:`code_rules`. +#. Add an extra role test scenario, to ensure your code path is + tested and working. +#. Test your new scenario with a new machine. + See also the :ref:`devel_and_testing` page. +#. Submit your code for review, with its necessary documentation and + release notes. + +.. _OpenStack-Ansible Launchpad project: https://bugs.launchpad.net/openstack-ansible + + +Example process to incubate a new "ops" project +----------------------------------------------- + +A new project in "openstack-ansible-ops" can be started at any time, +with no constraint like writing a specification, or creating a bug. + +Instead, the new code has to be isolated on a separate folder of the +`openstack-ansible-ops repo`_. + +.. _openstack-ansible-ops repo: https://git.openstack.org/cgit/openstack/openstack-ansible-ops + + +.. _backport: + Backporting -~~~~~~~~~~~ +=========== * Backporting is defined as the act of reproducing a change from another branch. Unclean/squashed/modified cherry-picks and complete @@ -135,7 +220,7 @@ Backporting commit message. This can be done with ``cherry-pick -x``. Here's more information on `Submitting a change to a branch for review`_. * Every backport commit must still only solve one problem, as per the - guidelines in `General Guidelines for Submitting Code`_. + guidelines in :ref:`codeguidelines`. * If a backport is a squashed set of cherry-picked commits, the original SHAs should be referenced in the commit message and the reason for squashing the commits should be clearly explained. @@ -148,239 +233,3 @@ Backporting .. _Submitting a change to a branch for review: http://www.mediawiki.org/wiki/Gerrit/Advanced_usage#Submitting_a_change_to_a_branch_for_review_.28.22backporting.22.29 .. _OpenStack Guidelines for stable branches: https://docs.openstack.org/project-team-guide/stable-branches.html - -Documentation and Release Note Guidelines -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Documentation is a critical part of ensuring that the deployers of -OpenStack-Ansible are appropriately informed about: - -* How to use the project's tooling effectively to deploy OpenStack. -* How to implement the right configuration to meet the needs of their specific - use-case. -* Changes in the project over time which may affect an existing deployment. - -To meet these needs developers must submit code comments, documentation and -release notes with any code submissions. All forms of documentation should -comply with the guidelines provided in the `OpenStack Documentation Contributor -Guide`_, with particular reference to the following sections: - -* Writing style -* RST formatting conventions - -.. _OpenStack Documentation Contributor Guide: https://docs.openstack.org/contributor-guide/ - -Code Comments -------------- - -Code comments for variables should be used to explain the purpose of the -variable. This is particularly important for the role defaults file as the file -is included verbatim in the role's documentation. Where there is an optional -variable, the variable and an explanation of what it is used for should be -added to the defaults file. - -Code comments for bash/python scripts should give guidance to the purpose of -the code. This is important to provide context for reviewers before the patch -has merged, and for later modifications to remind the contributors what the -purpose was and why it was done that way. - -Documentation -------------- - -OpenStack-Ansible has multiple forms of documentation with different intent. - -.. note:: - - The statements below regarding the Install Guide and Role Documentation are - statements of intent. The work to fulfill the intent is ongoing. Any new - documentation submissions should try to help this intent where possible. - -The `Deployment Guide `_ -intends to help deployers deploy OpenStack-Ansible for the first time. - -The role documentation (for example, the `keystone role documentation`_) -intends to explain all the options available for the role and how to implement -more advanced requirements. To reduce duplication, the role documentation -directly includes the role's default variables file which includes the -comments explaining the purpose of the variables. The long hand documentation -for the roles should focus less on explaining variables and more on explaining -how to implement advanced use cases. - -Where possible the documentation in OpenStack-Ansible should steer clear of -trying to explain OpenStack concepts. Those explanations belong in the -OpenStack Manuals or service documentation and OpenStack-Ansible documentation -should link to those documents when available, rather than duplicate their -content. - -.. _keystone role documentation: https://docs.openstack.org/developer/openstack-ansible-os_keystone/ - -Release Notes -------------- - -Release notes are generated using `the reno tool`_. Release notes must be -written with the following guidelines in mind: - -* Each list item must make sense to read without the context of the patch or - the repository the patch is being submitted into. The reason for this is that - all release notes are consolidated and presented in a long list without - reference to the source patch or the context of the repository. -* Each note should be brief and to the point. Try to avoid multi-paragraph - notes. For features the note should typically refer to documentation for more - details. For bug fixes the note can refer to a registered bug for more - details. - -In most cases only the following sections should be used for new release notes -submitted with patches: - -* ``features``: This should inform the deployer briefly about a new feature and - should describe how to use it either by referencing the variables to set or - by referring to documentation. -* ``issues``: This should inform the deployer about known issues. This may be - used when fixing an issue and wanting to inform deployers about a workaround - that can be used for versions prior to that which contains the patch that - fixes the issue. Issue notes should specifically make mention of what - versions of OpenStack-Ansible are affected by the issue. -* ``upgrade``: This should inform the deployer about changes which may affect - them when upgrading from a previous major or minor version. Typically, these - notes would describe changes to default variable values or variables that - have been removed. -* ``deprecations``: If a variable has been deprecated (ideally using the - deprecation filter), then it should be communicated through notes in this - section. Note that if a variable has been removed entirely then it has not - been deprecated and the removal should be noted in the ``upgrade`` section. - -Ansible Style Guide -~~~~~~~~~~~~~~~~~~~ - -YAML formatting ---------------- - -When creating tasks and other roles for use in Ansible please create them -using the YAML dictionary format. - -Example YAML dictionary format: - -.. code-block:: yaml - - - name: The name of the tasks - module_name: - thing1: "some-stuff" - thing2: "some-other-stuff" - tags: - - some-tag - - some-other-tag - - -Example what **NOT** to do: - -.. code-block:: yaml - - - name: The name of the tasks - module_name: thing1="some-stuff" thing2="some-other-stuff" - tags: some-tag - -.. code-block:: yaml - - - name: The name of the tasks - module_name: > - thing1="some-stuff" - thing2="some-other-stuff" - tags: some-tag - - -Usage of the ">" and "|" operators should be limited to Ansible conditionals -and command modules such as the Ansible ``shell`` or ``command``. - -Tags and tags conventions -------------------------- - -Tags are assigned based on the relevance of each individual item. -Higher level includes (for example in the ``tasks/main.yml``) need high -level tags. For example, ``*-config`` or ``*-install``. -Included tasks can have more detailed tags. - -The following convention is used: - -* A tag including the word ``install`` handles software installation tasks. - Running a playbook with ``--tags -install`` only deploys the - necessary software on the target, and will not configure it to your - needs or run any service. - -* A tag including the word ``config`` prepares the configuration of the - software (adapted to your needs), and all the components necessary - to run the service(s) configured in the role. Running a playbook with - ``--tags -config`` is only possible if the target already ran - the tags ``-install``. - -* A tag including the word ``upgrade`` handles all the upgrade tasks. - -Variable files conventions --------------------------- - -The variables files in a role are split in 3 locations: - -#. The `defaults/main.yml` file -#. The `vars/main.yml` file -#. The `vars/.yml` file - -The variables with lower priority should be in the `defaults/main.yml`. -This allows their overriding with group variables or host variables. -A good example for this are default database connection details, default -queues connection details, or debug mode. - -In other words, `defaults/main.yml` contains variables that are meant to -be overridable by a deployer or a continuous integration system. -These variables should be limited as much as possible, to avoid -increasing the test matrix. - -The `vars/main.yml` is always included. It contains generic -variables that aren't meant to be changed by a deployer. This includes -for example static information that aren't distribution specific (like -aggregation of role internal variables for example). - -The `vars/.yml` is the place where distribution -specific content will be stored. For example, this file will hold -the package names, repositories urls and keys, file paths, service -names/init scripts. - -Development cycle checklist -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -On top of the normal cycle goals, a contributor can help the OpenStack-Ansible -development team by performing one of the following recurring tasks: - -* By milestone 1: - - * Community goal acknowledgement. - -* By milestone 2: - - * Handle deprecations from upstream project's previous cycle. - - * Handle OpenStack-Ansible roles deprecations from the previous cycle. - - * Refresh static elements in roles. For example, update a specific version of - the software packages. - - * Bump ``ceph_stable_release`` to latest Ceph LTS release in the integrated - OpenStack-Ansible repo, and inside the ``ceph_client`` role defaults. - - * Check and bump galera versions if required. - - * Check and bump rabbitmq versions if required. - - * Check outstanding reviews and move to merge or abandon them if no longer - valid. - -* By milestone 3: - - * Implement features - -* After milestone 3: - - * Feature freeze, bug fixes, and testing improvements - -* After official project release, before official OpenStack-Ansible release: - - * Bump RDO, Ubuntu Cloud Archive and openSUSE OBS OpenStack Cloud - repositories if they are ready on time. diff --git a/doc/source/developer-docs/distributions.rst b/doc/source/contributor/distributions.rst similarity index 90% rename from doc/source/developer-docs/distributions.rst rename to doc/source/contributor/distributions.rst index bd2d9acaf7..62102d6599 100644 --- a/doc/source/developer-docs/distributions.rst +++ b/doc/source/contributor/distributions.rst @@ -2,14 +2,16 @@ Distribution support ==================== +.. _supported-distros: + Supported distributions -~~~~~~~~~~~~~~~~~~~~~~~ +======================= The list of supported distributions can be found in the :deploy_guide:`Deployment Guide ` Minimum requirements for OpenStack-Ansible roles -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +================================================ Existing and new distributions are expected to meet the following requirements in order for them to be accepted in the individual OpenStack-Ansible roles: @@ -17,7 +19,7 @@ in order for them to be accepted in the individual OpenStack-Ansible roles: * Pass functional tests Graduation -~~~~~~~~~~ +========== For a distribution to be considered supported by the OpenStack-Ansible project, it has to meet the following minimum requirements: @@ -28,7 +30,7 @@ project, it has to meet the following minimum requirements: Temptest testing framework. Voting -~~~~~~ +====== Distributions can be promoted to voting jobs on individual roles once they move to the `Graduation` phase and their stability has been confirmed by the core diff --git a/doc/source/contributor/index.rst b/doc/source/contributor/index.rst index 39b93a0a5e..21a02e0485 100644 --- a/doc/source/contributor/index.rst +++ b/doc/source/contributor/index.rst @@ -24,11 +24,14 @@ Contents: .. toctree:: :maxdepth: 2 + project-onboarding + bugs + contribute + code-rules + testing + periodic-work inventory-and-vars scripts - contribute - bug-triage core-reviewers - role-services distributions diff --git a/doc/source/contributor/periodic-work.rst b/doc/source/contributor/periodic-work.rst new file mode 100644 index 0000000000..fdab598f42 --- /dev/null +++ b/doc/source/contributor/periodic-work.rst @@ -0,0 +1,112 @@ +.. _periodicwork: + +============= +Periodic Work +============= + +Releasing +========= + +Our release schedule for stable branches is around every two weeks: + +* We request an OpenStack release on the second Friday of the month +* We request an OpenStack release on the last Friday of the month + +The releases take generally a few days to get "tagged" and be publicly +available. + +Dependency Updates +------------------ + +The dependencies for OpenStack-Ansible are updated +through the use of ``scripts/sources-branch-updater.sh``. This script +updates all pinned SHA's for OpenStack services, OpenStack-Ansible roles, +and other python dependencies which are not handled by the OpenStack global +requirements management process. This script also updates the statically +held templates/files in each role to ensure that they are always up to date. +Finally, it also increments the patch version of the +``openstack_release`` variable. + +The update script is used as follows: + +.. parsed-literal:: + + # change directory to the openstack-ansible checkout + cd ~/code/openstack-ansible + + # ensure that the correct branch is checked out + git checkout |current_release_git_branch_name| + + # ensure that the branch is up to date + git pull + + # create the local branch for the update + git checkout -b sha-update + + # execute the script for all openstack services + ./scripts/sources-branch-updater.sh -b |current_release_git_branch_name| -o |current_release_git_branch_name| + + # execute the script for gnocchi + ./scripts/sources-branch-updater.sh -s playbooks/defaults/repo_packages/gnocchi.yml -b |current_release_gnocchi_git_branch_name| -o |current_release_git_branch_name| + + # the console code should only be updated when necessary for a security fix, or for the OSA master branch + ./scripts/sources-branch-updater.sh -s playbooks/defaults/repo_packages/nova_consoles.yml -b master + + # the testing repositories should not be updated for stable branches as the new tests + # or other changes introduced may not work for older branches + ./scripts/sources-branch-updater.sh -s playbooks/defaults/repo_packages/openstack_testing.yml -b master + + # commit the changes + new_version=$(awk '/^openstack_release/ {print $2}' inventory/group_vars/all/all.yml) + git add --all + git commit -a -m "Update all SHAs for ${new_version}" \ + -m "This patch updates all the roles to the latest available stable + SHA's, copies the release notes from the updated roles into the + integrated repo, updates all the OpenStack Service SHA's, and + updates the appropriate python requirements pins. + + # push the changes up to gerrit + git review + + +Development cycle checklist +=========================== + +On top of the normal cycle goals, a contributor can help the OpenStack-Ansible +development team by performing one of the following recurring tasks: + +* By milestone 1: + + * Community goal acknowledgement. + +* By milestone 2: + + * Handle deprecations from upstream project's previous cycle. + + * Handle OpenStack-Ansible roles deprecations from the previous cycle. + + * Refresh static elements in roles. For example, update a specific version of + the software packages. + + * Bump ``ceph_stable_release`` to latest Ceph LTS release in the integrated + OpenStack-Ansible repo, and inside the ``ceph_client`` role defaults. + + * Check and bump galera versions if required. + + * Check and bump rabbitmq versions if required. + + * Check outstanding reviews and move to merge or abandon them if no longer + valid. + +* By milestone 3: + + * Implement features + +* After milestone 3: + + * Feature freeze, bug fixes, and testing improvements + +* After official project release, before official OpenStack-Ansible release: + + * Bump RDO, Ubuntu Cloud Archive and openSUSE OBS OpenStack Cloud + repositories if they are ready on time. diff --git a/doc/source/contributor/project-onboarding.rst b/doc/source/contributor/project-onboarding.rst new file mode 100644 index 0000000000..e6cc8de350 --- /dev/null +++ b/doc/source/contributor/project-onboarding.rst @@ -0,0 +1,118 @@ +================== +Project Onboarding +================== + +This document should help you understand how to contribute to +OpenStack-Ansible. + +Project repositories +==================== + +The OpenStack-Ansible project has different kinds of git repositories, +each of them with specific use cases, and different sets of practices. + +.. list-table:: + :header-rows: 1 + + * - Repository type or name + - Code location + - Repository purpose + * - | **OpenStack-Ansible** + | Also called *integrated repository* + - * https://github.com/openstack/openstack-ansible + - Our main repository, used by deployers. + Uses the other repositories. + * - | The **OpenStack-Ansible roles** repositories + - * https://github.com/openstack/openstack-ansible-os_nova + * https://github.com/openstack/openstack-ansible-os_glance + * https://github.com/openstack/ansible-role-systemd_mount + * https://github.com/openstack/ansible-config_template + * https://github.com/openstack/ansible-hardening + * ... + - Each role is in charge of deploying **exactly one** + component of an OpenStack-Ansible deployment. + * - | The **tests repository** + - * https://github.com/openstack/openstack-ansible-tests + - | The tests repository is the location for common code used in + the integrated repo and role repos tests. + | It allows us to not repeat ourselves: it is the location of + common playbooks, common tasks and scripts. + * - | The **specs** repository + - * https://github.com/openstack/openstack-ansible-specs + - This repository contains all the information concerning + large bodies of work done in OpenStack-Ansible, + split by cycle. + * - | The **ops** repository + - * https://github.com/openstack/openstack-ansible-ops + - This repository is an incubator for new projects, each project + solving a particular operational problem. Each project has its + own folder in this repository. + * - | External repositories + - * https://github.com/ceph/ceph-ansible + * https://github.com/logan2211/ansible-resolvconf + * https://github.com/willshersystems/ansible-sshd + * https://github.com/opendaylight/integration-packaging-ansible-opendaylight + * https://github.com/evrardjp/ansible-keepalived + * ... + - OpenStack-Ansible is not re-inventing the wheel, and tries to + reuse as much as possible existing roles. A bugfix for one of + those repositories must be handled to these repositories' + maintainers. + +How to contribute on code or issues +=================================== + +* For contributing code and documentation, you must follow the + OpenStack practices. Nothing special is required for OpenStack-Ansible. + + See also the `OpenStack developers getting started page`_. + and our :ref:`contributor guidelines` before hacking. + +* For helping on or submitting bugs, you must have an account on + ubuntu Launchpad. + All our repositories share the same `Launchpad project`_. + + Please check our :ref:`bug report` and + :ref:`bug triage` processes. + + Easy to fix bugs are marked with the tag *low hanging fruit*, and + should be the target of first time contributors. + +* For sharing your user experience, stories, and helping other users, + please join us in our :ref:`IRC channel`. + +* The OpenStack-Ansible project has recurring tasks that need + attention, like releasing, or other code duties. + See our page :ref:`Periodic work`. + +.. _OpenStack developers getting started page: https://docs.openstack.org/infra/manual/developers.html#getting-started +.. _Launchpad project: https://bugs.launchpad.net/openstack-ansible + +Community communication channels +================================ + +.. _irc: + +IRC channel +^^^^^^^^^^^ + +The OpenStack-Ansible community communicates a lot through IRC, in +the #openstack-ansible channel, on freenode. This channel is +logged, and its logs are published +`here `_. + +Weekly meetings are held in our IRC channel. The schedule and +logs can be found `here`_. +Next meeting agenda can be found on our +`Meetings wiki page `_. + +Mailing lists +^^^^^^^^^^^^^ + +A member of the OpenStack-Ansible community should monitor the +**OpenStack-dev** and **OpenStack-operators** `mailing lists`_. + +.. _mailing lists: http://lists.openstack.org/cgi-bin/mailman/listinfo + +All our communications should be prefixed with **[openstack-ansible]**. + diff --git a/doc/source/contributor/role-maturity.rst b/doc/source/contributor/role-maturity.rst deleted file mode 100644 index c5c3735821..0000000000 --- a/doc/source/contributor/role-maturity.rst +++ /dev/null @@ -1,11 +0,0 @@ -============= -Role maturity -============= - -All of the OpenStack-Ansible roles do not have the same level of maturity and -testing. - -Here is a dashboard of the current status of the roles: - -.. raw:: html - :file: role-maturity-matrix.html diff --git a/doc/source/contributor/role-services.rst b/doc/source/contributor/role-services.rst deleted file mode 100644 index 0cbce7f0c3..0000000000 --- a/doc/source/contributor/role-services.rst +++ /dev/null @@ -1,14 +0,0 @@ -================== -Roles and services -================== - -If you would like to contribute towards a role to introduce an OpenStack -service and/or an infrastructure service to support an OpenStack deployment, -the OpenStack-Ansible project would welcome that contribution and your -assistance in maintaining it. - -.. toctree:: - :maxdepth: 2 - - additional-roles - role-maturity diff --git a/doc/source/contributor/scripts.rst b/doc/source/contributor/scripts.rst deleted file mode 100644 index 95e53af2c4..0000000000 --- a/doc/source/contributor/scripts.rst +++ /dev/null @@ -1,244 +0,0 @@ -================ -Included Scripts -================ - -The repository contains several helper scripts to manage gate jobs, -install base requirements, and update repository information. Execute -these scripts from the root of the repository clone. For example: - -.. code:: bash - - $ scripts/.sh - -Bootstrapping -^^^^^^^^^^^^^ - -bootstrap-ansible.sh --------------------- - -The ``bootstrap-ansible.sh`` script installs Ansible, including the `core`_ and -`extras`_ module repositories and Galaxy roles. - -While there are several configurable environment variables which this script -uses, the following are commonly used: - -* ``ANSIBLE_PACKAGE`` - The version of Ansible to install. - -For example: - -.. code:: bash - - $ export ANSIBLE_PACKAGE="ansible==2.1.0" - -Installing directly from git is also supported. For example, from the tip of -Ansible development branch: - -.. code:: bash - - $ export ANSIBLE_PACKAGE="git+https://github.com/ansible/ansible@devel#egg=ansible" - -* ``ANSIBLE_ROLE_FILE`` - The location of a YAML file which ansible-galaxy can - consume which specifies which roles to download and install. The default - value for this is ``ansible-role-requirements.yml``. - -The script also creates the ``openstack-ansible`` wrapper tool that provides -the variable files to match ``/etc/openstack_deploy/user_*.yml`` as -arguments to ``ansible-playbook`` as a convenience. - -.. _core: https://github.com/ansible/ansible-modules-core -.. _extras: https://github.com/ansible/ansible-modules-extras - -bootstrap-aio.sh ----------------- - -The ``bootstrap-aio.sh`` script prepares a host for an -:ref:`All-In-One ` (AIO) -deployment for the purposes of development and gating. The script creates the -necessary partitions, directories, and configurations. The script can be -configured using environment variables - more details are provided on the -:ref:`All-In-One ` page. - -Development and Testing -^^^^^^^^^^^^^^^^^^^^^^^ - -Lint Tests ----------- - -Python coding conventions are tested using `PEP8`_, with the following -convention exceptions: - -* F403 - 'from ansible.module_utils.basic import \*' - -Testing may be done locally by executing: - -.. code-block:: bash - - tox -e pep8 - -Bash coding conventions are tested using `Bashate`_, with the following -convention exceptions: - -* E003: Indent not multiple of 4. We prefer to use multiples of 2 instead. -* E006: Line longer than 79 columns. As many scripts are deployed as templates - and use jinja templating, this is very difficult to achieve. It is - still considered a preference and should be a goal to improve - readability, within reason. -* E040: Syntax error determined using `bash -n`. As many scripts are deployed - as templates and use jinja templating, this will often fail. This - test is reasonably safely ignored as the syntax error will be - identified when executing the resulting script. - -Testing may be done locally by executing: - -.. code-block:: bash - - tox -e bashate - -Ansible is lint tested using `ansible-lint`_. - -Testing may be done locally by executing: - -.. code-block:: bash - - tox -e ansible-lint - -Ansible playbook syntax is tested using ansible-playbook. - -Testing may be done locally by executing: - -.. code-block:: bash - - tox -e ansible-syntax - -A consolidated set of all lint tests may be done locally by executing: - -.. code-block:: bash - - tox -e linters - -.. _PEP8: https://www.python.org/dev/peps/pep-0008/ -.. _Bashate: https://git.openstack.org/cgit/openstack-dev/bashate -.. _ansible-lint: https://github.com/willthames/ansible-lint - -Documentation Build -------------------- - -Documentation is developed in reStructuredText_ (RST) and compiled into -HTML using Sphinx. - -Documentation may be built locally by executing: - -.. code-block:: bash - - tox -e docs - tox -e deploy-guide - -.. _reStructuredText: http://docutils.sourceforge.net/rst.html - -Release Notes Build -------------------- - -Release notes are generated using the `the reno tool`_ and compiled into -HTML using Sphinx. - -Release notes may be built locally by executing: - -.. code-block:: bash - - tox -e releasenotes - -.. _the reno tool: https://docs.openstack.org/developer/reno/usage.html - -.. note:: - - The ``releasenotes`` build argument only tests committed changes. - Ensure your local changes are committed before running the - ``releasenotes`` build. - -Gating -^^^^^^ - -Every commit to the OpenStack-Ansible integrated build is verified by -OpenStack-CI through the following jobs: - -* ``gate-openstack-ansible-releasenotes``: This job executes the - `Release Notes Build`_. - -* ``gate-openstack-ansible-docs-ubuntu-xenial``: This job executes the - `Documentation Build`_. - -* ``gate-openstack-ansible-linters-ubuntu-xenial``: This job executes - the `Lint Tests`_. - -* ``gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial``: where - ``aio`` is the scenario, ``ubuntu`` is the distribution, and ``xenial`` - is the version of the distribution. - - The same test is executed against multiple distribution versions, and - may be executed against multiple distributions and multiple scenarios - too. - - This job executes the ``gate-check-commit.sh`` script which executes a - convergence test and then a functional test. - - The convergence test is the execution of an AIO build which aims to test - the primary code path for a functional environment. The functional test - then executes OpenStack's Tempest testing suite to verify that the - environment that has deployed successfully actually works. - - While this script is primarily developed and maintained for use in - OpenStack-CI, it can be used in other environments. - -Dependency Updates -^^^^^^^^^^^^^^^^^^ - -The dependencies for OpenStack-Ansible are updated approximately every two -weeks through the use of ``scripts/sources-branch-updater.sh``. This script -updates all pinned SHA's for OpenStack services, OpenStack-Ansible roles, -and other python dependencies which are not handled by the OpenStack global -requirements management process. This script also updates the statically -held templates/files in each role to ensure that they are always up to date. -Finally, it also does a minor version increment of the value for -``openstack_release``. - -The update script is used as follows: - -.. parsed-literal:: - - # change directory to the openstack-ansible checkout - cd ~/code/openstack-ansible - - # ensure that the correct branch is checked out - git checkout |current_release_git_branch_name| - - # ensure that the branch is up to date - git pull - - # create the local branch for the update - git checkout -b sha-update - - # execute the script for all openstack services - ./scripts/sources-branch-updater.sh -b |current_release_git_branch_name| -o |current_release_git_branch_name| - - # execute the script for gnocchi - ./scripts/sources-branch-updater.sh -s playbooks/defaults/repo_packages/gnocchi.yml -b |current_release_gnocchi_git_branch_name| -o |current_release_git_branch_name| - - # the console code should only be updated when necessary for a security fix, or for the OSA master branch - ./scripts/sources-branch-updater.sh -s playbooks/defaults/repo_packages/nova_consoles.yml -b master - - # the testing repositories should not be updated for stable branches as the new tests - # or other changes introduced may not work for older branches - ./scripts/sources-branch-updater.sh -s playbooks/defaults/repo_packages/openstack_testing.yml -b master - - # commit the changes - new_version=$(awk '/^openstack_release/ {print $2}' inventory/group_vars/all/all.yml) - git add --all - git commit -a -m "Update all SHAs for ${new_version}" \ - -m "This patch updates all the roles to the latest available stable - SHA's, copies the release notes from the updated roles into the - integrated repo, updates all the OpenStack Service SHA's, and - updates the appropriate python requirements pins. - - # push the changes up to gerrit - git review - diff --git a/doc/source/contributor/testing.rst b/doc/source/contributor/testing.rst new file mode 100644 index 0000000000..08f1acb6db --- /dev/null +++ b/doc/source/contributor/testing.rst @@ -0,0 +1,368 @@ +.. _tests: + +======= +Testing +======= + +Adding tests to a new role +========================== + +Each of the role tests is in its tests/ folder. + +This folder contains at least the following files: + +* ``test.yml`` ("super" playbook acting as test router to sub-playbooks) +* ``-overrides.yml``. This var file is automatically loaded + by our shell script in our `tests repository`_. +* ``inventory``. A static inventory for role testing. + It's possible some roles have multiple inventories. See for example the + neutron role with its ``lxb_inventory``, ``calico_inventory``. +* ``group_vars`` and ``host_vars``. These folders will hold override the + necessary files for testing. For example, this is where you override + the IP addresses, IP ranges, and ansible connection details. +* ``ansible-role-requirements.yml``. This should be fairly straightforward: + this file contains all the roles to clone before running your role. + The roles' relative playbooks will have to be listed in the ``test.yml`` + file. However, keep in mind to NOT re-invent the wheel. For example, + if your role needs keystone, you don't need to create your own keystone + install playbook, because we have a generic keystone install playbook + in the `tests repository`. +* Only add a ``zuul.d`` folder when your role is imported into the + openstack-ansible namespace. + +.. _tests repository: https://git.openstack.org/cgit/openstack/openstack-ansible-tests + +Extending tests of an existing role +=================================== + +#. Modify the tox.ini to add your new scenario. If required, you can + override the inventory, and/or the variable files. +#. Add a new non-voting job in ``zuul.d/jobs.yaml``, and wire it in + the project tests file ``zuul.d/project.yaml``. + +.. _tempest-testing: + +Improve testing with tempest +============================ + +Once the initial convergence is working and the services are running, +the role development should focus on implementing some level of +functional testing. + +Ideally, the functional tests for an OpenStack role +should make use of Tempest to execute the functional tests. The ideal +tests to execute are scenario tests as they test the functions that +the service is expected to do in a production deployment. In the absence +of any scenario tests for the service a fallback option is to implement +the smoke tests instead. + +If no tempest is provided, some other functional testing should be done. +For APIs, you can probably check the HTTP response codes, with +specially crafted requests. + +.. _integrate-new-role-with-aio: + +Testing a new role with an AIO +============================== + +#. Include your role on the deploy host. + See also :ref:`extend_osa_roles`. +#. Perform any other host preparation (such as the tasks performed by the + ``bootstrap-aio.yml`` playbook). This includes any preparation tasks that + are particular to your service. +#. Generate files to include your service in the Ansible inventory + using ``env.d`` and ``conf.d`` files for use on your deploy host. + + .. HINT:: You can follow examples from other roles, making the appropriate + modifications being sure that group labels in ``env.d`` and ``conf.d`` + files are consistent. + + .. HINT:: A description of how these work can be + found in :ref:`inventory-confd` and :ref:`inventory-envd`. + +#. Generate secrets, if any, as described in the :deploy_guide:`Configure + Service Credentials `. + You can append your keys to an existing ``user_secrets.yml`` file or add a + new file to the ``openstack_deploy`` directory to contain them. Provide + overrides for any other variables you will need at this time as well, either + in ``user_variables.yml`` or another file. + + See also our :ref:`user-overrides` page. + + Any secrets required for the role to work must be noted in the + ``etc/openstack_deploy/user_secrets.yml`` file for reuse by other users. + +#. If your service is installed from source or relies on python packages which + need to be installed from source, specify a repository for the source + code of each requirement by adding a file to your deploy host under + ``playbooks/defaults/repo_packages`` in the OpenStack-Ansible source + repository and following the pattern of files currently in that directory. + You could also simply add an entry to an existing file there. Be sure to + run the ``repo-build.yml`` play later so that wheels for your packages will + be included in the repository infrastructure. + +#. Make any required adjustments to the load balancer configuration + (e.g. modify ``inventory/group_vars/all/haproxy.yml`` in the + OpenStack-Ansible source repository on your deploy host) so that your + service can be reached through a load balancer, if appropriate, and be sure + to run the ``haproxy-install.yml`` play later so your changes will be + applied. Please note, you can also use ``haproxy_extra_services`` variable + if you don't want to provide your service as default for everyone. + +#. Put together a service install playbook file for your role. This can also + be modeled from any existing service playbook that has similar + dependencies to your service (database, messaging, storage drivers, + container mount points, etc.). A common place to keep playbook files in a + Galaxy role is in an ``examples`` directory off the root of the role. + If the playbook is meant for installing an OpenStack service, name it + ``os--install.yml`` and target it at the appropriate + group defined in the service ``env.d`` file. + It is crucial that the implementation of the service is optional and + that the deployer must opt-in to the deployment through the population + of a host in the applicable host group. If the host group has no + hosts, Ansible skips the playbook's tasks automatically. + +#. Any variables needed by other roles to connect to the new role, or by the + new role to connect to other roles, should be implemented in + ``inventory/group_vars``. The group vars are essentially the + glue which playbooks use to ensure that all roles are given the + appropriate information. When group vars are implemented it should be a + minimum set to achieve the goal of integrating the new role into the + integrated build. + +#. Documentation must be added in the role to describe how to implement + the new service in an integrated environement. This content must + adhere to the :ref:`documentation`. Until the + role has integrated functional testing implemented (see also the + Role development maturity paragraph), the documentation + must make it clear that the service inclusion in OpenStack-Ansible is + experimental and is not fully tested by OpenStack-Ansible in an + integrated build. Alternatively, an user story can be created. + +#. A feature release note must be added to announce the new service + availability and to refer to the role documentation for further + details. This content must adhere to the + :ref:`documentation`. + +#. It must be possible to execute a functional, integrated test which + executes a deployment in the same way as a production environment. The + test must execute a set of functional tests using Tempest. This is the + required last step before a service can remove the experimental warning + from the documentation. + +.. HINT:: If you adhere to the pattern of isolating your role's extra + deployment requirements (secrets and var files, HAProxy yml fragments, + repo_package files, etc.) in their own files it makes it easy for you to + automate these additional steps when testing your role. + +.. _devel_and_testing: + +Running tests locally +===================== + +Linting +------- + +Python coding conventions are tested using `PEP8`_, with the following +convention exceptions: + +* F403 - 'from ansible.module_utils.basic import \*' + +Testing may be done locally by executing: + +.. code-block:: bash + + ./run_tests.sh pep8 + +Bash coding conventions are tested using `Bashate`_, with the following +convention exceptions: + +* E003: Indent not multiple of 4. We prefer to use multiples of 2 instead. +* E006: Line longer than 79 columns. As many scripts are deployed as templates + and use jinja templating, this is very difficult to achieve. It is + still considered a preference and should be a goal to improve + readability, within reason. +* E040: Syntax error determined using `bash -n`. As many scripts are deployed + as templates and use jinja templating, this will often fail. This + test is reasonably safely ignored as the syntax error will be + identified when executing the resulting script. + +Testing may be done locally by executing: + +.. code-block:: bash + + ./run_tests.sh bashate + +Ansible is lint tested using `ansible-lint`_. + +Testing may be done locally by executing: + +.. code-block:: bash + + ./run_tests.sh ansible-lint + +Ansible playbook syntax is tested using ansible-playbook. + +Testing may be done locally by executing: + +.. code-block:: bash + + ./run_tests.sh ansible-syntax + +A consolidated set of all lint tests may be done locally by executing: + +.. code-block:: bash + + ./run_tests.sh linters + +.. _PEP8: https://www.python.org/dev/peps/pep-0008/ +.. _Bashate: https://git.openstack.org/cgit/openstack-dev/bashate +.. _ansible-lint: https://github.com/willthames/ansible-lint + +Documentation building +---------------------- + +Documentation is developed in reStructuredText_ (RST) and compiled into +HTML using Sphinx. + +Documentation may be built locally by executing: + +.. code-block:: bash + + ./run_tests.sh docs + +.. _reStructuredText: http://docutils.sourceforge.net/rst.html + +Deployment guide building +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The OpenStack-Ansible integrated repo also has an extra documentation +building process, to build the deployment guide. + +This guide may be built locally by executing: + +.. code-block:: bash + + ./run_tests.sh deploy-guide + +Release notes building +---------------------- + +Release notes are generated using the `the reno tool`_ and compiled into +HTML using Sphinx. + +Release notes may be built locally by executing: + +.. code-block:: bash + + ./run_tests.sh releasenotes + +.. _the reno tool: https://docs.openstack.org/developer/reno/usage.html + +.. note:: + + The ``releasenotes`` build argument only tests committed changes. + Ensure your local changes are committed before running the + ``releasenotes`` build. + +Roles functional or scenario testing +------------------------------------ + +To run a functional test of the role, execute: + +.. code-block:: bash + + ./run_tests.sh functional + +Some roles have extra tests, like neutron, defined in ``tox.ini``. + +To run a functional test named "calico", execute: + +.. code-block:: bash + + ./run_tests.sh calico + +Integrated repo functional or scenario testing +---------------------------------------------- + +To test the integrated repo, follow the +:deploy_guide:`Deployment Guide ` + +Alternatively, you can check the :ref:`aio guide`, +or even run the gate wrapper script, +named ``scripts/gate-check-commit.sh``, described below. + +The OpenStack Infrastructure automated tests +============================================ + +There should be no difference between running tests in the openstack +infrastructure, versus running locally. + +The tests in the openstack infrastructure are triggered by jobs +defined in each repo ``zuul.d`` folder. + +See also the `zuul user guide`_. + +However, for reliability purposes, a few variables are defined +to point to the OpenStack infra pypi and packages mirrors. + +.. _zuul user guide: https://docs.openstack.org/infra/zuul/user/index.html + +The integrated repo functional test is using the +``scripts/gate-check-commit.sh`` script, which receives arguments +from the zuul run playbook definition. + +While this script is primarily developed and maintained for use in +OpenStack-CI, it can be used in other environments. + +.. _role-maturity: + +Role development maturity +========================= + +A role may be fully mature, even if it is not integrated in the +``openstack-ansible`` repository. The maturity depends on its +testing levels. + +A role can be in one of the four maturity levels: + +* ``Complete`` +* ``Incubated`` +* ``Unmaintained`` +* ``Retired`` + +Here are a series of rules that define maturity levels: + +* A role can be retired at any time if it is not relevant anymore. +* A role can be ``Incubated`` for maximum 2 cycles. +* An ``Incubated`` role that passes functional testing will be upgraded + to the ``Complete`` status, and cannot return in ``Incubated`` status. +* An ``Incubated`` role that didn't implement functional testing in + the six month timeframe will become ``Unmaintained``. +* A role in ``Complete`` status can be downgraded to ``Unmaintained``. + status, according to the maturity downgrade procedure. + +Maturity downgrade procedure +---------------------------- + +If a role has failed periodics or gate test for two weeks, a bug +should be filed, and a message to the mailing list will be sent, +referencing the bug. + +The next community meeting should discuss about role deprecation, +and if no contributor comes forward to fix the role, periodic +testing will be turned off, and the role will move to an +``unmaintained`` state. + +.. _role-maturity-matrix: + +Maturity Matrix +--------------- + +All of the OpenStack-Ansible roles do not have the same level of maturity and +testing. + +Here is a dashboard of the current status of the roles: + +.. raw:: html + :file: role-maturity-matrix.html diff --git a/doc/source/reference/configuration/extending-osa.rst b/doc/source/reference/configuration/extending-osa.rst index aed9414040..bea44d1b02 100644 --- a/doc/source/reference/configuration/extending-osa.rst +++ b/doc/source/reference/configuration/extending-osa.rst @@ -2,7 +2,7 @@ Extending OpenStack-Ansible =========================== Including OpenStack-Ansible in your project -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +------------------------------------------- Including the openstack-ansible repository within another project can be done in several ways: @@ -18,22 +18,21 @@ Also note that copying files into directories such as ``env.d`` or project. Including OpenStack-Ansible with your Ansible structure -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +------------------------------------------------------- You can create your own playbook, variable, and role structure while still including the OpenStack-Ansible roles and libraries by setting environment variables or by adjusting ``/usr/local/bin/openstack-ansible.rc``. -The relevant environment variables for Ansible 1.9 (included in -OpenStack-Ansible) are as follows: +The relevant environment variables for OpenStack-Ansible are as follows: ``ANSIBLE_LIBRARY`` This variable should point to - ``openstack-ansible/playbooks/library``. Doing so allows roles and + ``/etc/ansible/plugins/library``. Doing so allows roles and playbooks to access OpenStack-Ansible's included Ansible modules. ``ANSIBLE_ROLES_PATH`` This variable should point to - ``openstack-ansible/playbooks/roles``. This allows Ansible to + ``/etc/ansible/roles`` by default. This allows Ansible to properly look up any OpenStack-Ansible roles that extension roles may reference. ``ANSIBLE_INVENTORY`` @@ -58,3 +57,27 @@ Consider this directory structure:: The environment variables set would use ``../openstack-ansible/playbooks/``. + +.. _extend_osa_roles: + +Adding new or overriding roles in your OpenStack-Ansible installation +--------------------------------------------------------------------- + +By default OpenStack-Ansible uses its `ansible-role-requirements`_ file +to fetch the roles it requires for the installation process. + +The roles will be fetched into the standard ``ANSIBLE_ROLES_PATH``, +which defaults to ``/etc/ansible/roles``. + +``ANSIBLE_ROLE_FILE`` is an environment variable pointing to +the location of a YAML file which ansible-galaxy can consume, +specifying which roles to download and install. +The default value for this is ``ansible-role-requirements.yml``. + +You can override the ansible-role-requirement file used by defining +the environment variable ``ANSIBLE_ROLE_FILE`` before running the +``bootstrap-ansible.sh`` script. + +.. _ansible-role-requirements: https://git.openstack.org/cgit/openstack/openstack-ansible/tree/ansible-role-requirements.yml + + diff --git a/doc/source/reference/inventory/understanding-inventory.rst b/doc/source/reference/inventory/understanding-inventory.rst index cdfe7ad1b2..158d79be17 100644 --- a/doc/source/reference/inventory/understanding-inventory.rst +++ b/doc/source/reference/inventory/understanding-inventory.rst @@ -22,6 +22,8 @@ To customize the layout of the components for your deployment, modify the host groups and container groups appropriately before running the installation playbooks. +.. _inventory-confd: + Understanding host groups (conf.d structure) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -51,6 +53,8 @@ variables to any component containers on the specific host. particularly for new services, by using a new file in the ``conf.d/`` directory. +.. _inventory-envd: + Understanding container groups (env.d structure) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/source/user/aio/quickstart.rst b/doc/source/user/aio/quickstart.rst index 10301def7e..f766afb242 100644 --- a/doc/source/user/aio/quickstart.rst +++ b/doc/source/user/aio/quickstart.rst @@ -191,8 +191,10 @@ Notes: # TMPDIR=/var/tmp scripts/bootstrap-ansible.sh In order for all the services to run, the host must be prepared with the -appropriate disks, packages, network configuration and a base configuration -for the OpenStack Deployment. For the default AIO scenario, this preparation +appropriate disks partitioning, packages, network configuration and +configurations for the OpenStack Deployment. + +For the default AIO scenario, this preparation is completed by executing: .. code-block:: shell-session diff --git a/doc/source/user/aio/scenario-table-gen.html b/doc/source/user/aio/scenario-table-gen.html index e69de29bb2..75dbdffc3b 100644 --- a/doc/source/user/aio/scenario-table-gen.html +++ b/doc/source/user/aio/scenario-table-gen.html @@ -0,0 +1 @@ +
aio_basekitaio_lxcaio_metalaio_nspawncephcongressoctaviatackertranslations
ceph    X    
cinderXXXXX   X
congress     X   
designate X X    X
glanceXXXXX X X
haproxyXX XXXXXX
heat XXX   XX
horizon X X    X
keystoneXXXXXXXXX
magnum        X
neutronXXXXXXX X
novaXXXXXXX X
octavia      X X
sahara        X
swift XXX    X
tacker       X 
trove        X
\ No newline at end of file diff --git a/doc/source/user/index.rst b/doc/source/user/index.rst index dac25f870e..59e2239409 100644 --- a/doc/source/user/index.rst +++ b/doc/source/user/index.rst @@ -29,3 +29,4 @@ For in-depth technical information, see the l3pods/example.rst ceph/full-deploy.rst security/index.rst + source-overrides/index.rst diff --git a/doc/source/user/source-overrides/index.rst b/doc/source/user/source-overrides/index.rst new file mode 100644 index 0000000000..26b1c6fdf7 --- /dev/null +++ b/doc/source/user/source-overrides/index.rst @@ -0,0 +1,77 @@ +========================== +Source overriding examples +========================== + +There are situations where a deployer want to override sources with +its own fork. + +This chapter gives case-by-case examples on how to override the +default sources. + +Overriding Ansible version +========================== + +Overriding the default Ansible version is not recommended, as +each branch of OpenStack-Ansible has been built with the a specific +Ansible version in mind, and many Ansible changes are neither backwards +nor forward compatible. + +The ``bootstrap-ansible.sh`` script installs Ansible, and uses +a variable ``ANSIBLE_PACKAGE`` to describe which version to install. + +For example to install ansible version 2.5.0: + +.. code:: bash + + $ export ANSIBLE_PACKAGE="ansible==2.5.0" + + +Installing directly from git is also supported. For example, from the tip of +Ansible development branch: + +.. code:: bash + + $ export ANSIBLE_PACKAGE="git+https://github.com/ansible/ansible@devel#egg=ansible" + + +Overriding the roles +==================== + +Overriding the role file has been explained in the reference guide, +on the :ref:`extend_osa_roles` section. + +Overriding other upstream projects source code +============================================== + +All the upstream repositories used are defined in the +``openstack-ansible`` integrated repository, in the +``playbooks/defaults/repo_packages`` folder. + +For example, if you want to override ``glance`` repository with your +own, you need to define the following: + +:: + + glance_git_repo: https:// + glance_git_install_branch: + glance_git_project_group: glance_all + +Please note, for this glance example, that you do not need to edit the +``playbooks/defaults/repo_packages/openstack_services.yml`` file. + +Instead, the usual overrides mechanism can take place, and you +can define these 3 variables in a ``user_*.yml`` file. +See also the :ref:`user-overrides` page. + +.. note:: + + These variables behave a little differently than standard ansible + precedence, because they are also consumed by a custom lookup plugin. + + The ``py_pkgs lookup`` will ignore all _git_ variables unless the + ``_git_repo`` variable is present. + + So even if you only want to override the ``_git_install_branch`` for + a repository, you should also define the ``_git_repo`` variable + in your user variables. +