diff --git a/.travis.yml b/.travis.yml index 1aab0ae..5e5e305 100644 --- a/.travis.yml +++ b/.travis.yml @@ -31,6 +31,9 @@ matrix: # Run coverage checks. - python: 3.5 env: TOX_ENV=cover + # Generate docs. + - python: 3.5 + env: TOX_ENV=docs install: # Install tox in a virtualenv to ensure we have an up to date version. diff --git a/README.md b/README.md deleted file mode 100644 index 7bdc014..0000000 --- a/README.md +++ /dev/null @@ -1,120 +0,0 @@ -# Tenks - -Tenks is a utility that manages virtual bare metal clusters for development and -testing purposes. - -## Getting Started - -### Pre-Requisites - -Tenks has dependencies on Ansible roles that are hosted by Ansible Galaxy. -Given that your virtualenv of choice is active and Ansible (>=2.6) is -installed inside it, Tenks' role dependencies can be installed by -`ansible-galaxy install --role-file=requirements.yml ---roles-path=ansible/roles/`. - -### Hosts - -Tenks uses Ansible inventory to manage hosts. A multi-host setup is therefore -supported, although the default hosts configuration will deploy an all-in-one -setup on the host where the `ansible-playbook` command is executed -(`localhost`). - -* Configuration management of the Tenks cluster is always performed on - `localhost`. -* The `hypervisors` group should not directly contain any hosts. Its sub-groups - must contain one or more system. Systems in its sub-groups will host a subset - of the nodes deployed by Tenks. - - * The `libvirt` group is a sub-group of `hypervisors`. Systems in this - group will act as hypervisors using the Libvirt provider. - -### Configuration - -An override file should be created to configure Tenks. Any variables specified -in this file will take precedence over their default settings in Tenks. This -will allow you to set options as necessary for your setup, without needing to -directly modify Tenks' variable files. An example override file can be found -in `ansible/override.yml.example`. - -Most of the configuration you will need to do relates to variables defined in -`ansible/host_vars/localhost`. You can set your own values for these in your -override file (mentioned above). In addition to other options, you will need to -define the types of node you'd like to be able to manage as a dict in -`node_types`, as well as the desired deployment specifications in `specs`. -Format and guidance for available options will be found within the variable -file. - -Broadly, most variables in `ansible/group_vars/*` have sensible defaults which -may be left as-is unless you have a particular need to configure them. A -notable exception to this is the variable `physnet_mappings` in -`ansible/group_vars/hypervisors`, which should map physical network names to -the device to use for that network: this can be a network interface, or an -existing OVS or Linux bridge. If these mappings are the same for all hosts in -your `hypervisors` group, you may set a single dict `physnet_mappings` in your -overrides file, and this will be used for all hosts. If different mappings are -required for different hosts, you will need to individually specify them: for a -host with hostname *myhost*, set `physnet_mappings` within the file -`ansible/host_vars/myhost`. - -### Commands - -Tenks has a variable `cmd` which specifies the command to be run. This variable -can be set in your override file (see above). The possible values it can take -are: - -* `deploy`: create a virtual cluster to the specification given. This is the - default command. -* `teardown`: tear down any existing virtual cluster with the specification - given. - -### Running Tenks - -Currently, Tenks does not have a CLI or wrapper. Deployment can be run by -calling `ansible-playbook --inventory ansible/inventory ansible/deploy.yml ---extra-vars=@override.yml`, where `override.yml` is the path to your override -file. - -The `deploy.yml` playbook will run deployment from start to finish; -`teardown.yml` is `deploy.yml`'s "mirror image" to tear down a cluster. These -playbooks automatically set `cmd` appropriately, and they contain various -constituent playbooks which perform different parts of the deployment. An -individual section of Tenks can be run separately by substituting -`ansible/deploy.yml` in the command above with the path to the playbook(s) you -want to run. The current playbooks can be seen in the Ansible structure diagram -in the *Development* section. Bear in mind that you will have to set `cmd` in -your override file if you are running any of the sub-playbooks individually. - -Once a cluster has been deployed, it can be reconfigured by modifying the Tenks -configuration and rerunning `deploy.yml`. Node specs can be changed (including -increasing/decreasing the number of nodes); node types can also be -reconfigured. Existing nodes will be preserved where possible. - -## Limitations - -The following is a non-exhaustive list of current known limitations of Tenks: - -* When using the Libvirt provider (currently the only provider), Tenks - hypervisors cannot co-exist with a containerised Libvirt daemon (for example, - as deployed by Kolla in the nova-libvirt container). Tenks will configure an - uncontainerised Libvirt daemon instance on the hypervisor, and this may - conflict with an existing containerised daemon. A workaround is to disable - the Nova virtualised compute service on each Tenks hypervisor if it is - present (for example, `docker stop nova_libvirt`) before running Tenks. - -## Development - -A diagram representing the Ansible structure of Tenks can be seen below. Blue -rectangles represent playbooks, green rounded rectangles represent task books, -red ellipses represent roles and yellow rhombi represent action plugins. - - - -![Tenks Ansible structure](assets/tenks_ansible_structure.png) diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..c456510 --- /dev/null +++ b/README.rst @@ -0,0 +1,8 @@ +Tenks +===== + +Tenks is a utility that manages virtual bare metal clusters for development and +testing purposes. + +Refer to the documentation in ``doc/`` for installation instructions, +configuration guidance and other information. diff --git a/doc/requirements.txt b/doc/requirements.txt new file mode 100644 index 0000000..913b117 --- /dev/null +++ b/doc/requirements.txt @@ -0,0 +1 @@ +sphinx>=1.8.0 # BSD diff --git a/assets/tenks_ansible_structure.png b/doc/source/_static/tenks_ansible_structure.png similarity index 100% rename from assets/tenks_ansible_structure.png rename to doc/source/_static/tenks_ansible_structure.png diff --git a/doc/source/_static/tenks_networking_structure.png b/doc/source/_static/tenks_networking_structure.png new file mode 100644 index 0000000..a3964da Binary files /dev/null and b/doc/source/_static/tenks_networking_structure.png differ diff --git a/doc/source/architecture.rst b/doc/source/architecture.rst new file mode 100644 index 0000000..82b21f5 --- /dev/null +++ b/doc/source/architecture.rst @@ -0,0 +1,49 @@ +.. _architecture: + +Architecture +============ + +Ansible +------- + +A diagram representing the Ansible structure of Tenks can be seen below. Blue +rectangles represent playbooks, green rounded rectangles represent task books, +red ellipses represent roles and yellow rhombi represent action plugins. + +.. This diagram will need to be updated when the Ansible structure changes. The + original draw.io diagram can be found below. The link below contains the + diagram's XML which can be imported into draw.io and edited, then a new PNG + artifact can be produced. + https://drive.google.com/file/d/1MlmaTvJ2BPkhrOCLin4GPH265JDJqD1E/view?usp=sharing + +.. image:: _static/tenks_ansible_structure.png + :target: _static/tenks_ansible_structure.png + +Networking +---------- + +Tenks has a concept of 'physical network' which currently must map one-to-one +to the hardware networks plugged into the hypervisors. It requires device +mappings to be specified on a hypervisor for each physical network that is to +be connected to nodes on that hypervisor. This device can be an interface, a +Linux bridge or an Open vSwitch bridge. For each physical network that is give +a mapping on a hypervisor, a new Tenks-managed Open vSwitch bridge is created. +If the device mapped to this physnet is an interface, it is plugged directly +into the new bridge. If the device is an existing Linux bridge, a veth pair is +created to connect the existing bridge to the new bridge. If the device is an +existing Open vSwitch bridge, an Open vSwitch patch port is created to link th +two bridges. + +A new veth pair is created for each physical network that each node on each +hypervisor is connected to, and one end of the pair is plugged into the Tenks +Open vSwitch bridge for that physical network; the other end will be plugged +into the node itself. Creation of these veth pairs is necessary (at least for +the Libvirt provider) to ensure that an interface is present in Open vSwitch +even when the node itself is powered off. + +An example of the networking structure of Tenks is shown below. In this +example, one node was requested to be connected to physnet0 and physnet1, and +two nodes were requested to be connected just to physnet1. + +.. image:: _static/tenks_networking_structure.png + :target: _static/tenks_networking_structure.png diff --git a/doc/source/conf.py b/doc/source/conf.py new file mode 100644 index 0000000..e604c27 --- /dev/null +++ b/doc/source/conf.py @@ -0,0 +1,173 @@ +# -*- coding: utf-8 -*- +# +# Configuration file for the Sphinx documentation builder. +# +# This file does only contain a selection of the most common options. For a +# full list see the documentation: +# http://www.sphinx-doc.org/en/master/config + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + + +# -- Project information ----------------------------------------------------- + +project = u'Tenks' +copyright = u'2018, Will Miller' +author = u'Will Miller' + +# The short X.Y version +version = u'' +# The full version, including alpha/beta/rc tags +release = u'' + + +# -- General configuration --------------------------------------------------- + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = None + + +# -- Options for HTML output ------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'alabaster' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# The default sidebars (for documents that don't match any pattern) are +# defined by theme itself. Builtin themes are using these templates by +# default: ``['localtoc.html', 'relations.html', 'sourcelink.html', +# 'searchbox.html']``. +# +# html_sidebars = {} + + +# -- Options for HTMLHelp output --------------------------------------------- + +# Output file base name for HTML help builder. +htmlhelp_basename = 'Tenksdoc' + + +# -- Options for LaTeX output ------------------------------------------------ + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, 'Tenks.tex', u'Tenks Documentation', + u'Will Miller', 'manual'), +] + + +# -- Options for manual page output ------------------------------------------ + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, 'tenks', u'Tenks Documentation', + [author], 1) +] + + +# -- Options for Texinfo output ---------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, 'Tenks', u'Tenks Documentation', + author, 'Tenks', 'One line description of project.', + 'Miscellaneous'), +] + + +# -- Options for Epub output ------------------------------------------------- + +# Bibliographic Dublin Core info. +epub_title = project + +# The unique identifier of the text. This can be a ISBN number +# or the project homepage. +# +# epub_identifier = '' + +# A unique identification for the text. +# +# epub_uid = '' + +# A list of files that should not be packed into the epub file. +epub_exclude_files = ['search.html'] \ No newline at end of file diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst new file mode 100644 index 0000000..e65636c --- /dev/null +++ b/doc/source/configuration.rst @@ -0,0 +1,50 @@ +.. _configuration: + +Configuration +============= + +Hosts +----- + +Tenks uses Ansible inventory to manage hosts. A multi-host setup is therefore +supported, although the default hosts configuration will deploy an all-in-one +setup on the host where the ``ansible-playbook`` command is executed +(*localhost*). + +* Configuration management of the Tenks cluster is always performed on + *localhost*. +* The ``hypervisors`` group should not directly contain any hosts. Its sub-groups + must contain one or more system. Systems in its sub-groups will host a subset + of the nodes deployed by Tenks. + + * The ``libvirt`` group is a sub-group of ``hypervisors``. Systems in this + group will act as hypervisors using the Libvirt provider. + +Variable Configuration +---------------------- + +An override file should be created to configure Tenks. Any variables specified +in this file will take precedence over their default settings in Tenks. This +will allow you to set options as necessary for your setup, without needing to +directly modify Tenks' variable files. An example override file can be found +in ``ansible/override.yml.example``. + +Most of the configuration you will need to do relates to variables defined in +``ansible/host_vars/localhost``. You can set your own values for these in your +override file (mentioned above). In addition to other options, you will need to +define the types of node you'd like to be able to manage as a dict in +``node_types``, as well as the desired deployment specifications in ``specs``. +Format and guidance for available options will be found within the variable +file. + +Broadly, most variables in ``ansible/group_vars/*`` have sensible defaults which +may be left as-is unless you have a particular need to configure them. A +notable exception to this is the variable ``physnet_mappings`` in +``ansible/group_vars/hypervisors``, which should map physical network names to +the device to use for that network: this can be a network interface, or an +existing OVS or Linux bridge. If these mappings are the same for all hosts in +your ``hypervisors`` group, you may set a single dict ``physnet_mappings`` in your +overrides file, and this will be used for all hosts. If different mappings are +required for different hosts, you will need to individually specify them: for a +host with hostname *myhost*, set ``physnet_mappings`` within the file +``ansible/host_vars/myhost``. diff --git a/doc/source/development.rst b/doc/source/development.rst new file mode 100644 index 0000000..0b7362f --- /dev/null +++ b/doc/source/development.rst @@ -0,0 +1,10 @@ +Development +=========== + +Contribution +------------ + +Contribution to Tenks' development is welcomed. Currently, the correct workflow +is to raise an issue in the `GitHub repository +`_. Patches should be submitted for review +using a GitHub pull request. diff --git a/doc/source/index.rst b/doc/source/index.rst new file mode 100644 index 0000000..eff830a --- /dev/null +++ b/doc/source/index.rst @@ -0,0 +1,20 @@ +.. Tenks documentation master file, created by + sphinx-quickstart on Thu Sep 20 15:43:21 2018. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Welcome to Tenks's documentation! +================================= + +Tenks is a utility that manages virtual bare metal clusters for development and +testing purposes. + +.. toctree:: + :maxdepth: 2 + + architecture + configuration + development + install + limitations + run diff --git a/doc/source/install.rst b/doc/source/install.rst new file mode 100644 index 0000000..200c86e --- /dev/null +++ b/doc/source/install.rst @@ -0,0 +1,8 @@ +Installation +=============== + +Tenks has dependencies on Ansible roles that are hosted by Ansible Galaxy. +Given that your virtualenv of choice is active and Ansible (>=2.6) is +installed inside it, Tenks' role dependencies can be installed by +``ansible-galaxy install --role-file=requirements.yml +--roles-path=ansible/roles/``. diff --git a/doc/source/limitations.rst b/doc/source/limitations.rst new file mode 100644 index 0000000..c0dd86d --- /dev/null +++ b/doc/source/limitations.rst @@ -0,0 +1,12 @@ +Limitations +=========== + +The following is a non-exhaustive list of current known limitations of Tenks: + +* When using the Libvirt provider (currently the only provider), Tenks + hypervisors cannot co-exist with a containerised Libvirt daemon (for example, + as deployed by Kolla in the nova-libvirt container). Tenks will configure an + uncontainerised Libvirt daemon instance on the hypervisor, and this may + conflict with an existing containerised daemon. A workaround is to disable + the Nova virtualised compute service on each Tenks hypervisor if it is + present (for example, ``docker stop nova_libvirt``) before running Tenks. diff --git a/doc/source/run.rst b/doc/source/run.rst new file mode 100644 index 0000000..fac0fd0 --- /dev/null +++ b/doc/source/run.rst @@ -0,0 +1,38 @@ +Running Tenks +============= + +Commands +-------- + +Tenks has a variable ``cmd`` which specifies the command to be run. This +variable can be set in your override file (see :ref:`configuration`). The +possible values it can take are: + +* ``deploy``: create a virtual cluster to the specification given. This is the + default command. +* ``teardown``: tear down any existing virtual cluster with the specification + given. + +Execution +--------- + +Currently, Tenks does not have a CLI or wrapper. Deployment can be run by +calling ``ansible-playbook --inventory ansible/inventory ansible/deploy.yml +--extra-vars=@override.yml``, where ``override.yml`` is the path to your +override file. + +The ``deploy.yml`` playbook will run deployment from start to finish; +``teardown.yml`` is ``deploy.yml``'s "mirror image" to tear down a cluster. +These playbooks automatically set ``cmd`` appropriately, and they contain +various constituent playbooks which perform different parts of the deployment. +An individual section of Tenks can be run separately by substituting +``ansible/deploy.yml`` in the command above with the path to the playbook(s) +you want to run. The current playbooks can be seen in the Ansible structure +diagram in :ref:`architecture`. Bear in mind that you will have to set +``cmd`` in your override file if you are running any of the sub-playbooks +individually. + +Once a cluster has been deployed, it can be reconfigured by modifying the Tenks +configuration and rerunning ``deploy.yml``. Node specs can be changed +(including increasing/decreasing the number of nodes); node types can also be +reconfigured. Existing nodes will be preserved where possible. diff --git a/setup.cfg b/setup.cfg index 00233c4..4d1e64c 100644 --- a/setup.cfg +++ b/setup.cfg @@ -2,7 +2,7 @@ name = tenks summary = Deployment of virtual bare metal clusters with Tenks description-file = - README.md + README.rst author = Will Miller author-email = willm@stackhpc.com home-page = https://stackhpc.com diff --git a/tox.ini b/tox.ini index bec29c4..dfa3407 100644 --- a/tox.ini +++ b/tox.ini @@ -29,6 +29,14 @@ basepython = python2.7 commands = flake8 {posargs} +[testenv:docs] +basepython = python3 +deps = + -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/rocky} + -r{toxinidir}/requirements.txt + -r{toxinidir}/doc/requirements.txt +commands = sphinx-build -W -b html doc/source doc/build/html + [testenv:cover] basepython = python3 setenv =