openstack-doc-tools/autogenerate_config_docs

autogenerate_config_docs

Automatically generate configuration tables to document OpenStack.

Dependencies: python-git (at least version 0.3.2 RC1), oslo-incubator.

Using the wrapper

The autohelp-wrapper script installs a virtual environment and all the needed dependencies, clones or update the projects and manuals repositories, then runs the autohelp.py script.

The workflow is:

$ ./autohelp-wrapper update
$ vim sources/openstack-manuals/tools/autogenerate-config-flagmappings/*.flagmappings
$ ./autohelp-wrapper docbook
# check the results
$ git commit -a

For details about the autohelp.py script and its manual setup, see the next sections.

Setting up your environment

Note: This tool is best run in a fresh VM environment or a python virtual environment, as running it requires installing the dependencies of the particular OpenStack product you are working with. Installing all of that on your normal machine could leave you with a bunch of cruft!

Install git and python-pip:

$ sudo apt-get install git python-pip

Install oslo.config and GitPython:

$ sudo pip install oslo.config "GitPython>=0.3.2.RC1"

Install oslo-incubator from git:

$ git clone git://git.openstack.org/openstack/oslo-incubator
$ cd oslo-incubator
$ python setup.py build
$ sudo python setup.py install

Check out the repository you are working with:

$ git clone https://git.openstack.org/openstack/nova.git

(This guide makes reference to a /repos directory, so you should record the directory you are using and replace as appropriate below.)

Check out the tool itself:

$ git clone https://git.openstack.org/openstack/openstack-doc-tools.git

Install the dependencies for the product you are working with:

$ sudo pip install -r nova/requirements.txt

Note 1: Occasionally, the requirements.txt file does not reference all dependencies needed to import all modules in a package. You may need to manually install some dependencies with pip or your distro's package manager. Always check the log for failed imports.

Note 2: Although this tool imports from your checkout, occasionally conflicts happen with installed files (e.g. a change from foo/bar.py to foo/bar/__init__.py). For best results, uninstall the old product and install from git using setup.py.

Using the tool

This tool is divided into three parts:

1. Extraction of flags names:

    $ openstack-doc-tools/autogenerate_config_docs/autohelp.py create nova -i /repos/nova
   

   You only need to use the create action to add a new product. Otherwise, use update:

    $ openstack-doc-tools/autogenerate_config_docs/autohelp.py update nova -i /repos/nova
   

   The create action will create a new nova.flagmappings file, possibly overriding an existing file. The update action will create a nova.flagmappings.new file, merging in group information from the existing nova.flagmappings file.

2. Grouping of flags

   This is currently done manually, by using the flag name file and placing a category after a space:

    $ head nova.flagmappings
    aggregate\_image\_properties\_isolation\_namespace scheduling
    aggregate\_image\_properties\_isolation\_separator scheduling
    allow\_instance\_snapshots policy
    allow\_migrate\_to\_same\_host policy
    allow\_resize\_to\_same\_host policy
    allow\_same\_net\_traffic network
    allowed\_direct\_url\_schemes glance
    ...
   

3. Creation of docbook-formatted configuration table files:

   Example:

    $ openstack-doc-tools/autogenerate_config_docs/autohelp.py docbook nova -i /repos/nova
   

Duplicate options

Sometimes the log will tell you there was a duplicate option name. This often happens if a parent module does import * from a child module. For example, nova.db does from nova.db.api import *. If one module name is a child of the other, the tool will use the option from the more specific module and tell you so. This is probably not a problem.

If the tool tells you it's chosing one at random, then something else is happening, and you should investigate. Otherwise, you may lose an option.

A worked example - updating the docs for Icehouse

Update automatically generated tables - from scratch, using a virtual environment. Using a python virtual environment will avoid modifications and conflicts with python packages installed on the base system. The virtual environment can be destroyed by deleting the venv directory.

Note that the virtual environment must be activated for every new shell you start. This can be done by sourcing the venv/bin/activate file.

$ mkdir WORKDIR
$ cd WORKDIR
$ sudo apt-get update
$ sudo apt-get install git python-virtualenv
$ virtualenv venv
$ source venv/bin/activate
$ git clone git://git.openstack.org/openstack/openstack-doc-tools.git
$ git clone git://git.openstack.org/openstack/openstack-manuals.git
$ git clone git://git.openstack.org/openstack/oslo-incubator
$ cd oslo-incubator
$ python setup.py install
$ cd ..
$ pip install "GitPython>=0.3.2.RC1"
$ cd openstack-manuals/tools/autogenerate-config-flagmappings

Now, download and install the projects. Installation is necessary to satisfy dependencies between projects. This will also install required dependencies.

$ mkdir -p sources && cd sources
$ PROJECTS="ceilometer cinder glance heat neutron nova trove"
$ for p in $PROJECTS; do git clone git://git.openstack.org/openstack/$p.git; done
$ for p in $PROJECTS; do cd $p && python setup.py install && cd ..; done
$ cd ..

Install some missing requirements:

$ pip install fixtures swift hp3parclient ryu pymongo bson pbr

Note that some dependencies will still be missing. They can't be installed using pip, they should be installed manually. Not installing those dependencies will not prevent the tool to work, but some configuration options might not be discovered.

Update the flag names:

$ for p in $PROJECTS; do
> ../../../openstack-doc-tools/autogenerate_config_docs/autohelp.py -vvv update $p
> done

At this point, search through the *.flagmappings.new files for anything labelled Unknown and fix it. Once that is done rename the *.flagmappings.new files and run autohelp.py with the docbook subcommand:

$ for p in $PROJECTS; do
> mv $p.flagmappings.new $p.flagmappings
> ../../../openstack-doc-tools/autogenerate_config_docs/autohelp.py -vvv docbook $p
> done

to generate the XML files and move those into the appropriate part of the git repo.