Browse Source

Remove 'autogenerate_config_docs'

This tool has been superseded by the 'oslo_config.sphinxext' Sphinx
extension and can finally be put to rest.

[1] https://docs.openstack.org/oslo.config/latest/reference/sphinxext.html

Change-Id: Idae5ba656c3a0da8a26c6f1e63332365c8012c0b
tags/1.7.0
Stephen Finucane 2 years ago
parent
commit
d3c952064a
15 changed files with 10 additions and 1560 deletions
  1. +0
    -6
      README.rst
  2. +0
    -121
      autogenerate_config_docs/README.rst
  3. +0
    -18
      autogenerate_config_docs/__init__.py
  4. +0
    -273
      autogenerate_config_docs/autohelp-wrapper
  5. +0
    -696
      autogenerate_config_docs/autohelp.py
  6. +0
    -289
      autogenerate_config_docs/diff_branches.py
  7. +0
    -5
      autogenerate_config_docs/requirements.txt
  8. +0
    -45
      autogenerate_config_docs/templates/autohelp-category.rst.j2
  9. +0
    -40
      autogenerate_config_docs/templates/autohelp-group.rst.j2
  10. +0
    -61
      autogenerate_config_docs/templates/changes.rst.j2
  11. +0
    -1
      doc/source/autogenerate_config_docs.rst
  12. +0
    -1
      doc/source/index.rst
  13. +8
    -0
      releasenotes/notes/the-great-doc-tools-cleanup-1a79e2c200232489.yaml
  14. +0
    -1
      setup.cfg
  15. +2
    -3
      tox.ini

+ 0
- 6
README.rst View File

@@ -59,9 +59,3 @@ On openSUSE::
On Ubuntu::

$ apt-get install libxml2-dev libxslt-dev


Regenerating config option tables
---------------------------------

See :ref:`autogenerate_config_docs`.

+ 0
- 121
autogenerate_config_docs/README.rst View File

@@ -1,121 +0,0 @@
.. _autogenerate_config_docs:

autogenerate_config_docs
========================

Automatically generate configuration tables to document OpenStack.

Using the wrapper
-----------------

``autohelp-wrapper`` is the recommended tool to generate the configuration
tables. Don't bother using ``autohelp.py`` manually.

The ``autohelp-wrapper`` script installs a virtual environment and all the
needed dependencies, clones or updates the projects and manuals repositories,
then runs the ``autohelp.py`` script in the virtual environment.

New and updated flagmappings are generated in the ``openstack-manuals``
repository (``tools/autogenerate-config-flagmappings/`` directory).

Prior to running the following commands, you need to install several development
packages.

On Ubuntu:

.. code-block:: console

$ sudo apt-get install python-dev python-pip python-virtualenv \
libxml2-dev libxslt1-dev zlib1g-dev \
libmysqlclient-dev libpq-dev libffi-dev \
libsqlite3-dev libldap2-dev libsasl2-dev \
libjpeg-dev

On RHEL 7 and CentOS 7:

.. code-block:: console

$ sudo yum install https://www.rdoproject.org/repos/rdo-release.rpm
$ sudo yum update
$ sudo yum install python-devel python-pip python-virtualenv \
libxml2-devel libxslt-devel zlib-devel \
mariadb-devel postgresql-devel libffi-devel \
sqlite-devel openldap-devel cyrus-sasl-devel \
libjpeg-turbo-devel gcc git

.. note::
* libjpeg is needed for ironic

The workflow is:

.. code-block:: console

$ pip install -rrequirements.txt
$ ./autohelp-wrapper update
$ $EDITOR sources/openstack-manuals/tools/autogenerate-config-flagmappings/*.flagmappings
$ ./autohelp-wrapper rst
$ # check the results in sources/openstack-manuals

This will generate the tables for all the known projects.
Note for neutron project: If the driver/plugin resides outside the neutron
repository, then the driver/plugin has to be explicitly installed within the
virtual environment to generate the configuration options.

To generate the mappings and tables for a subset of projects, use the code
names as arguments:

.. code-block:: console

$ ./autohelp-wrapper update cinder heat
$ # edit the mappings files
$ ./autohelp-wrapper rst cinder heat


Flagmappings files
------------------

The tool uses flagmapping files to map options to custom categories. Flag
mapping files can be found in the ``tools/autogenerate-config-flagmappings``
folder of the openstack-manuals project. Not all projects use flagmapping
files, as those that do not will be disabled by the presence of a
``$project.disable`` file in that folder. For those that do, however, the files
use the following format::

OPTION_SECTION/OPTION_NAME group1 [group2, ...]

Groups need to be defined manually to organize the configuration tables.

The group values can only contain alphanumeric characters, _ and - (they will
be used as document IDs).

To make the table titles more user friendly, create or edit the PROJECT.headers
file in the manuals repository. Each line of this file is of the form:

::

GROUP A Nice Title

Working with branches
---------------------

``autohelp-wrapper`` works on the master branch by default, but you can tell it
to work on another branch:

.. code-block:: console

$ ./autohelp-wrapper -b stable/liberty update

.. note::
The ``-b`` switch doesn't apply to the ``openstack-manuals`` repository
which will be left untouched (no ``git branch``, no ``git update``).


Generate configuration difference
---------------------------------

To generate "New, updated, and deprecated options" for each service,
run ``diff_branches.py``. For example:

.. code-block:: console

$ ./diff_branches.py stable/liberty stable/mitaka nova

+ 0
- 18
autogenerate_config_docs/__init__.py View File

@@ -1,18 +0,0 @@
# Copyright 2012 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

import pbr.version


__version__ = pbr.version.VersionInfo('openstack-doc-tools').version_string()

+ 0
- 273
autogenerate_config_docs/autohelp-wrapper View File

@@ -1,273 +0,0 @@
#!/bin/bash
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

set -e

HERE=$(pwd)
VENVDIR=$HERE/venv
SOURCESDIR=$HERE/sources
MANUALSREPO=$SOURCESDIR/openstack-manuals
MAPPINGS_DIR=$MANUALSREPO/tools/autogenerate-config-flagmappings
AUTOHELP="python $HERE/autohelp.py"
GITBASE=${GITBASE:-git://git.openstack.org/openstack}
GITPROJ=${GITPROJ:-git://git.openstack.org/openstack}
PROJECTS="aodh ceilometer cinder glance heat ironic keystone manila \
murano neutron nova sahara senlin trove zaqar"
MANUALS_PROJECTS="openstack-manuals"
BRANCH=master
FAST=0
QUIET=0
CLONE_MANUALS=1

usage() {
echo "Wrapper for autohelp.py"
echo "Usage:"
echo " $(basename $0) [ OPTIONS ] dump|update|rst|setup [ project... ]"
echo
echo "Subcommands:"
echo " dump: Dumps the list of options with their attributes"
echo " update: Update or create the flagmapping files"
echo " rst: Generate the options tables in RST format"
echo " setup: Install the environment only"
echo
echo "Options:"
echo " -b BRANCH: Work on this branch (defaults to master)"
echo " -g GITPROJ: Use this location for the project git repos "
echo " (defaults to git://git.openstack.org/openstack)"
echo " -c: Recreate the virtual environment"
echo " -f: Work offline: Do not change environment or sources"
echo " -e PATH: Create the virtualenv in PATH"
echo " -v LEVEL: Verbose message (1 or 2)"
echo " (check various python modules imported or not)"
echo " -o OUTDIR: Path to output openstack-manuals directory "
echo " (defaults to ./sources/openstack-manuals)"
}

setup_venv() {
project=$1

if [ ! -e $VENVDIR/$project/bin/activate ]; then
mkdir -p $VENVDIR/$project
virtualenv $VENVDIR/$project
fi
activate_venv $project
}

activate_venv() {
project=$1

. $VENVDIR/$project/bin/activate
pip install --upgrade pip setuptools
}

get_project() {
project=$1
git_url=$GITPROJ

if [ ! -e $SOURCESDIR/$project ]; then
if [[ $MANUALS_PROJECTS =~ (^| )$project($| ) ]]; then
git_url=$GITBASE
fi
git clone $git_url/$project $SOURCESDIR/$project

if [ -e $MAPPINGS_DIR/$project.extra_repos ]; then
while read extra; do
git clone $git_url/$extra $SOURCESDIR/$extra
done < $MAPPINGS_DIR/$project.extra_repos
fi

else
if [ $project != openstack-manuals ]; then
(cd $SOURCESDIR/$project && git pull)
fi

if [ -e $MAPPINGS_DIR/$project.extra_repos ]; then
while read extra; do
(cd $SOURCESDIR/$extra && git pull)
done < $MAPPINGS_DIR/$project.extra_repos
fi
fi
}

setup_tools() {
pip install -rrequirements.txt
}

while getopts :b:g:e:o:v:cfq opt; do
case $opt in
b)
BRANCH=$OPTARG
;;
g)
GITPROJ=$OPTARG
;;
c)
rm -rf $VENVDIR
;;
e)
VENVDIR=$OPTARG
;;
o)
MANUALSREPO=$OPTARG
MAPPINGS_DIR=$OPTARG/tools/autogenerate-config-flagmappings
CLONE_MANUALS=0
;;
f)
FAST=1
;;
q)
QUIET=1
;;
v)
AUTOOPT="-v"
if [ $OPTARG = 2 ]; then
AUTOOPT="-vv"
fi
;;
\?)
usage
exit 1
;;
esac
done
shift $(($OPTIND - 1))

if [ $# -lt 1 ]; then
usage
exit 1
fi

ACTION=$1
shift

if [ $QUIET -eq 1 ]; then
exec 3>&1 >/dev/null
exec 4>&2 2>/dev/null
fi

case $ACTION in
update|rst|dump|setup) ;;
*)
usage
exit 1
;;
esac

if [ ! -e autohelp.py ]; then
echo "Execute this script in the autogenerate_config_docs directory."
exit 1
fi

[ $# != 0 ] && PROJECTS="$*"

RELEASE=$(echo $BRANCH | sed 's,^stable.,,')

if [ "$FAST" -eq 0 ] ; then
if [ "$CLONE_MANUALS" -eq 1 ] ; then
get_project openstack-manuals
fi

for project in $PROJECTS; do
setup_venv $project
setup_tools
if [ -e $MAPPINGS_DIR/$project.requirements ]; then
pip install -r $MAPPINGS_DIR/$project.requirements \
--allow-all-external
fi
get_project $project

(
pushd $SOURCESDIR/$project
module=$(echo $project | tr - _ )
find $module -name "*.pyc" -delete
GIT_CMD="git show-ref --verify --quiet refs/heads/$BRANCH"
if $GIT_CMD; then
git checkout $BRANCH
else
git checkout -b $BRANCH remotes/origin/$BRANCH
fi
pip install -rrequirements.txt
[ -e "test-requirements.txt" ] && \
pip install -rtest-requirements.txt
python setup.py install
popd

if [ -e $MAPPINGS_DIR/$project.extra_repos ]; then
while read extra; do
(
cd $SOURCESDIR/$extra
pip install -rrequirements.txt
[ -e "test-requirements.txt" ] && \
pip install -rtest-requirements.txt
python setup.py install
)
done < $MAPPINGS_DIR/$project.extra_repos
fi
)
done
fi

for project in $PROJECTS; do
echo "Working on $project..."
activate_venv $project
if [ "$ACTION" = "setup" ]; then
break
fi

if [ -e $MAPPINGS_DIR/$project.extra_repos ]; then
extra_flags=
while read extra; do
package=$(echo $extra | tr - _)
if [ $package = "networking_midonet" ]; then
package="midonet"
fi
if [ $package = "networking_hyperv" ]; then
package="hyperv"
fi
if [ $package = "networking_edge_vpn" ]; then
package="networking-edge-vpn"
fi
if [ $package = "networking_zvm" ]; then
package="neutron"
cp -r $SOURCESDIR/networking-zvm/neutron/plugins/zvm \
$SOURCESDIR/neutron/neutron/plugins
cp -r \
$SOURCESDIR/networking-zvm/neutron/plugins/ml2/drivers/zvm\
$SOURCESDIR/neutron/neutron/plugins/ml2/drivers
fi
extra_flags="$extra_flags -i $SOURCESDIR/$extra/$package"
done < $MAPPINGS_DIR/$project.extra_repos
fi

cd $MAPPINGS_DIR

case $ACTION in
update)
$AUTOHELP update $project -i $SOURCESDIR/$project/$project \
$extra_flags $AUTOOPT
mv $project.flagmappings.new $project.flagmappings
;;
rst)
$AUTOHELP rst $project -i $SOURCESDIR/$project/$project \
$extra_flags $AUTOOPT
;;
dump)
if [ $QUIET -eq 1 ]; then
exec 1>&3
exec 2>&4
fi
$AUTOHELP dump $project -i $SOURCESDIR/$project/$project \
$extra_flags $AUTOOPT
;;
esac
done

+ 0
- 696
autogenerate_config_docs/autohelp.py View File

@@ -1,696 +0,0 @@
#!/usr/bin/env python
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
# A collection of tools for working with flags from OpenStack
# packages and documentation.
#
# For an example of usage, run this program with the -h switch.
#

# Must import this before argparse
from oslo_config import cfg

import argparse
import importlib
import os
import pickle
import re
import sys

import jinja2
import stevedore

try:
from sqlalchemy import exc
except Exception:
pass

sys.path.insert(0, '.')
try:
from hooks import HOOKS # noqa
except ImportError:
pass


EXTENSIONS = ['oslo.cache',
'oslo.concurrency',
'oslo.db',
'oslo.log',
'oslo.messaging',
'oslo.middleware',
'oslo.policy',
'oslo.service']

_TYPE_DESCRIPTIONS = {
cfg.StrOpt: 'String',
cfg.BoolOpt: 'Boolean',
cfg.IntOpt: 'Integer',
cfg.FloatOpt: 'Floating point',
cfg.ListOpt: 'List',
cfg.DictOpt: 'Dict',
cfg.MultiStrOpt: 'Multi-valued',
cfg.IPOpt: 'IP address',
cfg.PortOpt: 'Port number',
cfg.HostnameOpt: 'Hostname',
cfg.URIOpt: 'URI',
cfg.HostAddressOpt: 'Host address',
cfg._ConfigFileOpt: 'List of filenames',
cfg._ConfigDirOpt: 'List of directory names',
}

register_re = re.compile(r'''^ +.*\.register_opts\((?P<opts>[^,)]+)'''
r'''(, (group=)?["'](?P<group>.*)["'])?\)''')


def import_modules(repo_location, package_name, verbose=0):
"""Import modules.

Loops through the repository, importing module by module to
populate the configuration object (cfg.CONF) created from Oslo.
"""

with open('ignore.list') as fd:
ignore_list = [l for l in fd.read().split('\n')
if l and (l[0] != '#')]

pkg_location = os.path.join(repo_location, package_name)
for root, dirs, files in os.walk(pkg_location):
skipdir = False
for excludedir in ('tests', 'locale',
os.path.join('db', 'migration'), 'transfer'):
if ((os.path.sep + excludedir + os.path.sep) in root or (
root.endswith(os.path.sep + excludedir))):
skipdir = True
break
if skipdir:
continue
for pyfile in files:
if pyfile.endswith('.py'):
abs_path = os.path.join(root, pyfile)
modfile = abs_path.split(repo_location, 1)[1]
modname = os.path.splitext(modfile)[0].split(os.path.sep)
modname = [m for m in modname if m != '']
modname = '.'.join(modname)
if modname.endswith('.__init__'):
modname = modname[:modname.rfind(".")]
if modname in ignore_list:
continue
try:
module = importlib.import_module(modname)
if verbose >= 1:
print("imported %s" % modname)
except ImportError as e:
"""
work around modules that don't like being imported in
this way FIXME This could probably be better, but does
not affect the configuration options found at this stage
"""
if verbose >= 2:
print("Failed to import: %s (%s)" % (modname, e))
continue
except cfg.DuplicateOptError as e:
"""
oslo.cfg doesn't allow redefinition of a config option, but
we don't mind. Don't fail if this happens.
"""
if verbose >= 2:
print(e)
continue
except cfg.NoSuchGroupError as e:
"""
If a group doesn't exist, we ignore the import.
"""
if verbose >= 2:
print(e)
continue
except exc.InvalidRequestError as e:
if verbose >= 2:
print(e)
continue
except Exception as e:
print("Impossible to import module %s" % modname)
raise

_register_runtime_opts(module, abs_path, verbose)
_run_hook(modname)

# All the components provide keystone token authentication, usually using a
# pipeline. Since the auth_token options can only be discovered at runtime
# in this configuration, we force their discovery by importing the module.
try:
import keystonemiddleware.auth_token # noqa
except cfg.DuplicateOptError:
pass


def _run_hook(modname):
try:
HOOKS[modname]()
except KeyError:
pass


def _register_runtime_opts(module, abs_path, verbose):
"""Handle options not registered on module import.

This function parses the .py files to discover calls to register_opts in
functions and methods. It then explicitly call cfg.register_opt on each
option to register (most of) them.
"""

with open(abs_path) as fd:
for line in fd:
m = register_re.search(line)
if not m:
continue

opts_var = m.group('opts')
opts_group = m.group('group')

# Get the object (an options list) from the opts_var string.
# This requires parsing the string which can be of the form
# 'foo.bar'. We treat each element as an attribute of the previous.
register = True
obj = module
for item in opts_var.split('.'):
try:
obj = getattr(obj, item)
except AttributeError:
# FIXME(gpocentek): AttributeError is raised when a part of
# the opts_var string is not an actual attribute. This will
# need more parsing tricks.
register = False
if verbose >= 2:
print("Ignoring %(obj)s in %(module)s" %
{'obj': opts_var, 'module': module})
break

if register and isinstance(obj, list):
for opt in obj:
if not isinstance(opt, cfg.Opt):
continue
try:
cfg.CONF.register_opt(opt, opts_group)
except cfg.DuplicateOptError:
# ignore options that have already been registered
pass


def _sanitize_default(opt):
"""Adapts unrealistic default values."""

# If the Oslo version is recent enough, we can use the 'sample_default'
# attribute
if (hasattr(opt, 'sample_default') and opt.sample_default is not None):
return str(opt.sample_default)

if ((type(opt).__name__ == "ListOpt") and (type(opt.default) == list)):
return ", ".join(str(item) for item in opt.default)

default = str(opt.default)

if default == os.uname()[1]:
return 'localhost'

if opt.name == 'bindir':
return '/usr/local/bin'

if opt.name == 'my_ip':
return '10.0.0.1'

if isinstance(opt, cfg.StrOpt) and default.strip() != default:
return '"%s"' % default

for pathelm in sys.path:
if pathelm in ('.', ''):
continue
if pathelm.endswith('/'):
pathelm = pathelm[:-1]
if pathelm in default:
default = re.sub(r'%s(/sources)?' % pathelm,
'/usr/lib/python/site-packages', default)

return default


def _get_overrides(package_name):
overrides_file = '%s.overrides' % package_name
if not os.path.exists(overrides_file):
return {}
overrides = {}
with open(overrides_file) as fd:
for line in fd:
if line == '#':
continue
try:
opt, sections = line.strip().split(' ', 1)
sections = [x.strip() for x in sections.split(' ')]
except ValueError:
continue

overrides[opt] = sections

return overrides


class OptionsCache(object):
def __init__(self, overrides={}, verbose=0):
self._verbose = verbose
self._opts_by_name = {}
self._opts_by_group = {}
self._opt_names = []
self._overrides = overrides

for optname in cfg.CONF._opts:
opt = cfg.CONF._opts[optname]['opt']
# We ignore some CLI opts by excluding SubCommandOpt objects
if not isinstance(opt, cfg.SubCommandOpt):
self._add_opt(optname, 'DEFAULT', opt)

for group in cfg.CONF._groups:
for optname in cfg.CONF._groups[group]._opts:
self._add_opt(group + '/' + optname, group,
cfg.CONF._groups[group]._opts[optname]['opt'])

self._opt_names.sort(OptionsCache._cmpopts)

def _add_opt(self, optname, group, opt):
if optname in self._opts_by_name:
if self._verbose >= 2:
print("Duplicate option name %s" % optname)
return

opt.default = _sanitize_default(opt)

def fill(optname, group, opt):
if optname in self._opts_by_name:
return
self._opts_by_name[optname] = (group, opt)
self._opt_names.append(optname)

if group not in self._opts_by_group:
self._opts_by_group[group] = []

self._opts_by_group[group].append(opt)

if optname in self._overrides:
for new_group in self._overrides[optname]:
if new_group == 'DEFAULT':
new_optname = opt.name
else:
new_optname = new_group + '/' + opt.name
fill(new_optname, new_group, opt)

else:
fill(optname, group, opt)

def __len__(self):
return len(self._opt_names)

def load_extension_options(self, module):
# Note that options loaded this way aren't added to _opts_by_module
loader = stevedore.named.NamedExtensionManager(
'oslo.config.opts',
names=(module,),
invoke_on_load=False
)
for ext in loader:
for group, opts in ext.plugin():
for opt in opts:
if group is None:
self._add_opt(opt.dest, 'DEFAULT', opt)
else:
self._add_opt(group + '/' + opt.dest, group, opt)

self._opt_names.sort(OptionsCache._cmpopts)

def maybe_load_extensions(self, repositories):
# Use the requirements.txt of the project to guess if an oslo module
# needs to be imported
needed_exts = set()
for repo in repositories:
base_path = os.path.dirname(repo)
for ext in EXTENSIONS:
requirements = os.path.join(base_path, 'requirements.txt')
with open(requirements) as fd:
for line in fd:
if line.startswith(ext):
needed_exts.add(ext)

for ext in needed_exts:
self.load_extension_options(ext)

def get_group_names(self):
return self._opts_by_group.keys()

def get_option_names(self):
return self._opt_names

def get_group(self, name):
return self._opts_by_group[name]

def get_option(self, name):
return self._opts_by_name[name]

def dump(self):
"""Dumps the list of options with their attributes.

This output is consumed by the diff_branches script.
"""
for name, (group, option) in self._opts_by_name.items():
deprecated_opts = [{'group': deprecated.group,
'name': deprecated.name}
for deprecated in option.deprecated_opts]
help_str = option.help.strip() if option.help else "None"
new_option = {
'default': option.default,
'help': help_str,
'deprecated_opts': deprecated_opts,
'type': option.__class__.__name__.split('.')[-1]
}
self._opts_by_name[name] = (group, new_option)
print(pickle.dumps(self._opts_by_name))

@staticmethod
def _cmpopts(x, y):
if '/' in x and '/' in y:
prex = x[:x.find('/')]
prey = y[:y.find('/')]
if prex != prey:
return cmp(prex, prey)
return cmp(x, y)
elif '/' in x:
return 1
elif '/' in y:
return -1
else:
return cmp(x, y)


def _use_categories(package_name):
return not os.path.isfile(package_name + '.disable')


def _get_options_by_cat(package_name):
options_by_cat = {}

with open(package_name + '.flagmappings') as f:
for line in f:
if not line.strip() or line.startswith('#'):
continue
opt, categories = line.split(' ', 1)
for category in categories.split():
options_by_cat.setdefault(category, []).append(opt)

return options_by_cat


def _get_category_names(package_name):
package_headers = package_name + '.headers'
category_names = {}
for headers_file in ('shared.headers', package_headers):
try:
with open(headers_file) as f:
for line in f:
if not line.strip() or line.startswith('#'):
continue
cat, nice_name = line.split(' ', 1)
category_names[cat] = nice_name.strip()
except IOError:
print("Cannot open %s (ignored)" % headers_file)

return category_names


def _format_opt(option):

def _remove_prefix(text, prefix):
if text.startswith(prefix):
return text[len(prefix):].lstrip(':')
return text

def _reflow_text(text):
text = re.sub(r'\n+\s*\* ', '$sentinal$* ', text)
text = text.replace('\n\n', '$sentinal$')
text = text.replace('\n', ' ')
text = ' '.join(text.split())
return text.split('$sentinal$')

def _strip_indentation(text):
return ' '.join([x.strip() for x in text.split('\n')]).strip()

help_text = option.help or "No help text available for this option."
help_text = _remove_prefix(help_text.strip(), 'DEPRECATED')
help_text = _reflow_text(help_text)

deprecated_text = (option.deprecated_reason or
'No deprecation reason provided for this option.')
deprecated_text = _strip_indentation(deprecated_text)

opt_type = _TYPE_DESCRIPTIONS.get(type(option), 'Unknown')

flags = []

if (option.deprecated_for_removal or
(option.help and option.help.startswith('DEPRECATED'))):
flags.append(('Deprecated', deprecated_text))
if option.mutable:
flags.append(('Mutable', 'This option can be changed without'
' restarting.'))

return {
'name': option.dest,
'type': opt_type,
'default': _sanitize_default(option),
'help': help_text,
'flags': flags
}


def write_files(package_name, options, target):
"""Write tables.

Some projects make use of flagmapping files, while others make use
of oslo.config's OptGroup to do the same in code. The function will
handle both, printing a list of options by either category or group.
"""
target = target or '../../doc/config-reference/source/tables'
if not os.path.isdir(target):
os.makedirs(target)

if _use_categories(package_name):
_write_files_by_category(package_name, options, target)
else:
_write_files_by_group(package_name, options, target)


def _write_files_by_category(package_name, options, target):
options_by_cat = _get_options_by_cat(package_name)
category_names = _get_category_names(package_name)

for cat in options_by_cat.keys():
env = {
'pkg': package_name,
'cat': cat,
'label': '-'.join([package_name, cat]),
'groups': [],
'items': [],
}

# Skip the options that is explicitly marked as disabled,
# which is used for common configuration options.
if cat == 'disable':
continue

if cat in category_names:
env['nice_cat'] = category_names[cat]
else:
env['nice_cat'] = cat
print("No nicename for %s" % cat)

curgroup = None
items = None
for optname in options_by_cat[cat]:
group, option = options.get_option(optname)

if group != curgroup:
if group is not None:
curgroup = group
env['groups'].append(group)
if items is not None:
env['items'].append(items)
items = []

items.append(_format_opt(option))

env['items'].append(items)

file_path = ("%(target)s/%(package_name)s-%(cat)s.rst" %
{'target': target, 'package_name': package_name,
'cat': cat})
tmpl_file = os.path.join(os.path.dirname(__file__),
'templates/autohelp-category.rst.j2')
_write_template(tmpl_file, file_path, env)


def _write_files_by_group(package_name, options, target):
for group in options.get_group_names():
env = {
'pkg': package_name,
'group': group,
'label': '-'.join([package_name, group]),
'items': [],
}

for option in options.get_group(group):
env['items'].append(_format_opt(option))

file_path = ("%(target)s/%(package_name)s-%(group)s.rst" %
{'target': target, 'package_name': package_name,
'group': group})
tmpl_file = os.path.join(os.path.dirname(__file__),
'templates/autohelp-group.rst.j2')
_write_template(tmpl_file, file_path, env)


def _write_template(template_path, output_path, env):
with open(template_path) as fd:
template = jinja2.Template(fd.read(), trim_blocks=True)
output = template.render(filename=output_path, **env)

with open(output_path, 'w') as fd:
fd.write(output)


def update_flagmappings(package_name, options, verbose=0):
"""Update flagmappings file.

Update a flagmappings file, adding or removing entries as needed.
This will create a new file $package_name.flagmappings.new with
category information merged from the existing $package_name.flagmappings.
"""
if not _use_categories:
print("This project does not use flagmappings. Nothing to update.")
return

original_flags = {}
try:
with open(package_name + '.flagmappings') as f:
for line in f:
try:
flag, category = line.split(' ', 1)
except ValueError:
flag = line.strip()
category = "Unknown"
original_flags.setdefault(flag, []).append(category.strip())
except IOError:
# If the flags file doesn't exist we'll create it
pass

updated_flags = []
for opt in options.get_option_names():
if len(original_flags.get(opt, [])) == 1:
updated_flags.append((opt, original_flags[opt][0]))
continue

updated_flags.append((opt, 'Unknown'))

with open(package_name + '.flagmappings.new', 'w') as f:
for flag, category in updated_flags:
f.write(flag + ' ' + category + '\n')

if verbose >= 1:
removed_flags = (set(original_flags.keys()) -
set([x[0] for x in updated_flags]))
added_flags = (set([x[0] for x in updated_flags]) -
set(original_flags.keys()))

print("\nRemoved Flags\n")
for line in sorted(removed_flags, OptionsCache._cmpopts):
print(line)

print("\nAdded Flags\n")
for line in sorted(added_flags, OptionsCache._cmpopts):
print(line)


def main():
parser = argparse.ArgumentParser(
description='Manage flag files, to aid in updating documentation.',
usage='%(prog)s <cmd> <package> [options]')
parser.add_argument('subcommand',
help='Action (update, rst, dump).',
choices=['update', 'rst', 'dump'])
parser.add_argument('package',
help='Name of the top-level package.')
parser.add_argument('-v', '--verbose',
action='count',
default=0,
dest='verbose',
required=False,)
parser.add_argument('-i', '--input',
dest='repos',
help='Path to a python package in which options '
'should be discoverd. Can be used multiple '
'times.',
required=False,
type=str,
action='append')
parser.add_argument('-o', '--output',
dest='target',
help='Directory or file in which data will be saved.\n'
'Defaults to ../../doc/common/tables/ '
'for "rst".\n'
'Defaults to stdout for "dump"',
required=False,
type=str,)
args = parser.parse_args()

if args.repos is None:
args.repos = ['./sources/%s/%s' % args.package]

for repository in args.repos:
package_name = os.path.basename(repository)
base_path = os.path.dirname(repository)

sys.path.insert(0, base_path)
try:
__import__(package_name)
except ImportError as e:
if args.verbose >= 1:
print(str(e))
print("Failed to import: %s (%s)" % (package_name, e))

import_modules(base_path, package_name, verbose=args.verbose)
sys.path.pop(0)

overrides = _get_overrides(package_name)
options = OptionsCache(overrides, verbose=args.verbose)
options.maybe_load_extensions(args.repos)

if args.verbose > 0:
print("%s options imported from package %s." % (len(options),
str(package_name)))

if args.subcommand == 'update':
update_flagmappings(args.package, options, verbose=args.verbose)

elif args.subcommand == 'rst':
write_files(args.package, options, args.target)

elif args.subcommand == 'dump':
options.dump()


if __name__ == "__main__":
main()

+ 0
- 289
autogenerate_config_docs/diff_branches.py View File

@@ -1,289 +0,0 @@
#!/usr/bin/env python
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
# A collection of tools for working with flags from OpenStack
# packages and documentation.
#
# For an example of usage, run this program with the -h switch.
#
import argparse
import os
import pickle
import subprocess
import sys

import jinja2


PROJECTS = ['aodh', 'ceilometer', 'cinder', 'glance', 'heat', 'ironic',
'keystone', 'manila', 'neutron', 'nova', 'sahara', 'trove']
MASTER_RELEASE = 'Ocata'
CODENAME_TITLE = {'aodh': 'Alarming',
'ceilometer': 'Telemetry',
'cinder': 'Block Storage',
'glance': 'Image service',
'heat': 'Orchestration',
'ironic': 'Bare Metal service',
'keystone': 'Identity service',
'manila': 'Shared File Systems service',
'murano': 'Application Catalog service',
'neutron': 'Networking',
'nova': 'Compute',
'sahara': 'Data Processing service',
'senlin': 'Clustering service',
'trove': 'Database service',
'zaqar': 'Message service'}


def setup_venv(projects, branch, novenvupdate):
"""Setup a virtual environment for `branch`."""
dirname = os.path.join('venv', branch.replace('/', '_'))
if novenvupdate and os.path.exists(dirname):
return
if not os.path.exists('venv'):
os.mkdir('venv')
args = ["./autohelp-wrapper", "-b", branch, "-e", dirname, "setup"]
args.extend(projects)
if subprocess.call(args) != 0:
print("Impossible to create the %s environment." % branch)
sys.exit(1)


def _get_packages(project, branch):
release = branch if '/' not in branch else branch.split('/')[1]
packages = [project]
try:
with open('extra_repos/%s-%s.txt' % (project, release)) as f:
packages.extend([p.strip() for p in f])
except IOError:
pass

return packages


def get_options(project, branch):
"""Get the list of known options for a project."""
print("Working on %(project)s (%(branch)s)" % {'project': project,
'branch': branch})
# And run autohelp script to get a serialized dict of the discovered
# options
dirname = os.path.join('venv', branch.replace('/', '_'))
args = ["./autohelp-wrapper", "-q", "-b", branch, "-e", dirname,
"dump", project]

path = os.environ.get("PATH")
bin_path = os.path.abspath(os.path.join(dirname, "bin"))
path = "%s:%s" % (bin_path, path)
serialized = subprocess.check_output(args,
env={'VIRTUAL_ENV': dirname,
'PATH': path})
ret = pickle.loads(serialized)
return ret


def _cmpopts(x, y):
"""Compare to option names.

The options can be of 2 forms: option_name or group/option_name. Options
without a group always comes first. Options are sorted alphabetically
inside a group.
"""
if '/' in x and '/' in y:
prex = x[:x.find('/')]
prey = y[:y.find('/')]
if prex != prey:
return cmp(prex, prey)
return cmp(x, y)
elif '/' in x:
return 1
elif '/' in y:
return -1
else:
return cmp(x, y)


def diff(old_list, new_list):
"""Compare the old and new lists of options."""
new_opts = []
new_defaults = []
deprecated_opts = []
for name, (group, option) in new_list.items():
# Find the new options
if name not in old_list.viewkeys():
new_opts.append(name)

# Find the options for which the default value has changed
elif option['default'] != old_list[name][1]['default']:
new_defaults.append(name)

# Find options that have been deprecated in the new release.
# If an option name is a key in the old_list dict, it means that it
# wasn't deprecated.

# Some options are deprecated, but not replaced with a new option.
# These options usually contain 'DEPRECATED' in their help string.
if 'DEPRECATED' in option['help']:
deprecated_opts.append((name, None))

for deprecated in option['deprecated_opts']:
# deprecated_opts is a list which always holds at least 1 invalid
# dict. Forget it.
if deprecated['name'] is None:
continue

if deprecated['group'] in [None, 'DEFAULT']:
full_name = deprecated['name']
else:
full_name = deprecated['group'] + '/' + deprecated['name']

if full_name in old_list.viewkeys():
deprecated_opts.append((full_name, name))

return new_opts, new_defaults, deprecated_opts


def format_option_name(name):
"""Return a formatted string for the option path."""
if name is None:
return "None"

try:
section, name = name.split('/')
except ValueError:
# name without a section ('log_dir')
return "[DEFAULT] %s" % name

return "[%s] %s" % (section, name)


def release_from_branch(branch):
if branch == 'master':
return MASTER_RELEASE
else:
return branch.replace('stable/', '').title()


def get_env(project, new_branch, old_list, new_list):
"""Generate the jinja2 environment for the defined branch and project."""
new_opts, new_defaults, deprecated_opts = diff(old_list, new_list)
release = release_from_branch(new_branch)

env = {
'release': release,
'project': project,
'codename': CODENAME_TITLE[project],
'new_opts': [],
'new_defaults': [],
'deprecated_opts': []
}

# New options
if new_opts:
for name in sorted(new_opts, _cmpopts):
opt = new_list[name][1]
name = format_option_name(name)
helptext = opt['help'].strip().replace('\n', ' ')
helptext = ' '.join(helptext.split())
cells = (("%(name)s = %(default)s" %
{'name': name,
'default': opt['default']}).strip(),
"(%(type)s) %(help)s" % {'type': opt['type'],
'help': helptext})
env['new_opts'].append(cells)

# New defaults
if new_defaults:
for name in sorted(new_defaults, _cmpopts):
old_default = old_list[name][1]['default']
new_default = new_list[name][1]['default']
if isinstance(old_default, list):
old_default = ", ".join(old_default)
if isinstance(new_default, list):
new_default = ", ".join(new_default)
name = format_option_name(name)
cells = (name, old_default, new_default)
env['new_defaults'].append(cells)

# Deprecated options
if deprecated_opts:
for deprecated, new in sorted(deprecated_opts, cmp=_cmpopts,
key=lambda tup: tup[0]):
deprecated = format_option_name(deprecated)
new = format_option_name(new)
env['deprecated_opts'].append((deprecated, new))

return env


def main():
parser = argparse.ArgumentParser(
description='Generate a summary of configuration option changes.',
usage='%(prog)s [options] <old_branch> <new_branch> [projects]')
parser.add_argument('old_branch',
help='Name of the old branch.')
parser.add_argument('new_branch',
help='Name of the new branch.')
parser.add_argument('projects',
help='List of projects to work on.',
nargs='*',
default=PROJECTS)
parser.add_argument('-i', '--input',
dest='sources',
help='Path to a folder containing the git '
'repositories.',
required=False,
default='./sources',
type=str,)
parser.add_argument('-o', '--output',
dest='target',
help='Directory or file in which data will be saved.\n'
'Defaults to "."',
required=False,
default='.',
type=str,)
parser.add_argument('-n', '--no-venv-update',
dest='novenvupdate',
help='Don\'t update the virtual envs.',
required=False,
action='store_true',
default=False,)
args = parser.parse_args()

setup_venv(args.projects, args.old_branch, args.novenvupdate)
setup_venv(args.projects, args.new_branch, args.novenvupdate)

for project in args.projects:
old_list = get_options(project, args.old_branch)
new_list = get_options(project, args.new_branch)

release = args.new_branch.replace('stable/', '')
env = get_env(project, release, old_list, new_list)
filename = ("%(project)s-conf-changes.rst" %
{'project': project})
tmpl_file = 'templates/changes.rst.j2'
if not os.path.exists(args.target):
os.makedirs(args.target)
dest = os.path.join(args.target, filename)

with open(tmpl_file) as fd:
template = jinja2.Template(fd.read(), trim_blocks=True)
output = template.render(**env)

with open(dest, 'w') as fd:
fd.write(output)

return 0


if __name__ == "__main__":
sys.exit(main())

+ 0
- 5
autogenerate_config_docs/requirements.txt View File

@@ -1,5 +0,0 @@
docutils
jinja2
lxml
oslo.config
oslo.i18n

+ 0
- 45
autogenerate_config_docs/templates/autohelp-category.rst.j2 View File

@@ -1,45 +0,0 @@
..
Warning: Do not edit this file. It is automatically generated from the
software project's code and your changes will be overwritten.

The tool to generate this file lives in openstack-doc-tools repository.

Please make any changes needed in the code, then run the
autogenerate-config-doc tool from the openstack-doc-tools repository, or
ask for help on the documentation mailing list, IRC channel or meeting.

.. _{{ label }}:

.. list-table:: Description of {{ nice_cat }} configuration options
:header-rows: 1
:class: config-ref-table

* - Configuration option = Default value
- Description
{% for group in groups %}

* - **[{{ group }}]**
-
{% for item in items[loop.index0] %}

{% if item['default'] is equalto '' %}
* - ``{{ item['name'] }}`` =
{% else %}
* - ``{{ item['name'] }}`` = ``{{ item['default'] }}``
{% endif %}
{% for paragraph in item['help'] %}

{% if loop.first %}
- ({{ item['type'] }}) {{ paragraph }}
{% else %}
{{ paragraph }}
{% endif %}
{% endfor %}
{% for flagname, flagdesc in item['flags'] %}

- **{{ flagname }}**

{{ flagdesc }}
{% endfor %}
{% endfor %}
{% endfor %}

+ 0
- 40
autogenerate_config_docs/templates/autohelp-group.rst.j2 View File

@@ -1,40 +0,0 @@
..
Warning: Do not edit this file. It is automatically generated from the
software project's code and your changes will be overwritten.

The tool to generate this file lives in openstack-doc-tools repository.

Please make any changes needed in the code, then run the
autogenerate-config-doc tool from the openstack-doc-tools repository, or
ask for help on the documentation mailing list, IRC channel or meeting.

.. _{{ label }}:

.. list-table:: Description of {{ group }} configuration options
:header-rows: 1
:class: config-ref-table

* - Configuration option = Default value
- Description
{% for item in items %}

{% if item['default'] is equalto '' %}
* - ``{{ item['name'] }}`` =
{% else %}
* - ``{{ item['name'] }}`` = ``{{ item['default'] }}``
{% endif %}
{% for paragraph in item['help'] %}

{% if loop.first %}
- ({{ item['type'] }}) {{ paragraph }}
{% else %}
{{ paragraph }}
{% endif %}
{% endfor %}
{% for flagname, flagdesc in item['flags'] %}

- **{{ flagname }}**

{{ flagdesc }}
{% endfor %}
{% endfor %}

+ 0
- 61
autogenerate_config_docs/templates/changes.rst.j2 View File

@@ -1,61 +0,0 @@
New, updated, and deprecated options in {{ release }} for {{ codename }}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{{ '~' * release|length }}~~~~~{{ '~' * codename|length }}

..
Warning: Do not edit this file. It is automatically generated and your
changes will be overwritten. The tool to do so lives in the
openstack-doc-tools repository.

{% if new_opts %}
.. list-table:: New options
:header-rows: 1
:class: config-ref-table

* - Option = default value
- (Type) Help string
{% for cells in new_opts %}
* - ``{{ cells[0] }}``
- {{ cells[1] }}
{% endfor %}
{% endif %}

{% if new_defaults %}
.. list-table:: New default values
:header-rows: 1
:class: config-ref-table

* - Option
- Previous default value
- New default value
{% for cells in new_defaults %}
* - ``{{ cells[0] }}``
{% if cells[1] is equalto '' %}
-
{% else %}
- ``{{ cells[1] }}``
{% endif %}
{% if cells[2] is equalto '' %}
-
{% else %}
- ``{{ cells[2] }}``
{% endif %}
{% endfor %}
{% endif %}

{% if deprecated_opts %}
.. list-table:: Deprecated options
:header-rows: 1
:class: config-ref-table

* - Deprecated option
- New Option
{% for cells in deprecated_opts %}
* - ``{{ cells[0] }}``
- ``{{ cells[1] }}``
{% endfor %}
{% endif %}

{% if not new_opts and not new_defaults and not deprecated_opts %}
There are no new, updated, and deprecated options
in {{ release }} for {{ codename }}.
{% endif %}

+ 0
- 1
doc/source/autogenerate_config_docs.rst View File

@@ -1 +0,0 @@
.. include:: ../../autogenerate_config_docs/README.rst

+ 0
- 1
doc/source/index.rst View File

@@ -10,7 +10,6 @@ Contents:
doc-tools-readme
installation
usage
autogenerate_config_docs
man/openstack-doc-test
sitemap-readme
release_notes


+ 8
- 0
releasenotes/notes/the-great-doc-tools-cleanup-1a79e2c200232489.yaml View File

@@ -0,0 +1,8 @@
---
upgrade:
- |
The `autogenerate_config_docs` set of tools has been removed. These were
always buggy and hard to maintain, and they have been superseded by the
``oslo_config.sphinxext`` `Sphinx extension`__.

__ https://docs.openstack.org/oslo.config/latest/reference/sphinxext.html

+ 0
- 1
setup.cfg View File

@@ -22,7 +22,6 @@ classifier =
[files]
packages =
os_doc_tools
autogenerate_config_docs
data_files =
share/openstack-doc-tools/sitemap = sitemap/*
share/openstack-doc-tools/cleanup = cleanup/*


+ 2
- 3
tox.ini View File

@@ -25,8 +25,7 @@ commands =
doc8 -e txt -e rst doc/source/ HACKING.rst
# Run bashate during pep8 runs to ensure violations are caught by
# the check and gate queues.
bashate autogenerate_config_docs/autohelp-wrapper \
bin/doc-tools-check-languages \
bashate bin/doc-tools-check-languages \
cleanup/remove_trailing_whitespaces.sh

[testenv:pylint]
@@ -58,7 +57,7 @@ usedevelop = False
[flake8]
show-source = True
builtins = _
exclude=.venv,.git,.tox,dist,*lib/python*,*egg,build,*autogenerate_config_docs/venv,*autogenerate_config_docs/sources,doc/source/conf.py
exclude=.venv,.git,.tox,dist,*lib/python*,*egg,build,doc/source/conf.py
# 28 is currently the most complex thing we have
max-complexity=29
ignore = H101

Loading…
Cancel
Save