Remove 'autogenerate_config_docs'
This tool has been superseded by the 'oslo_config.sphinxext' Sphinx extension and can finally be put to rest. [1] https://docs.openstack.org/oslo.config/latest/reference/sphinxext.html Change-Id: Idae5ba656c3a0da8a26c6f1e63332365c8012c0b
This commit is contained in:
Stephen Finucane
parent
3dd0834d7a
commit
d3c952064a
@ -59,9 +59,3 @@ On openSUSE::
|
||||
On Ubuntu::
|
||||
|
||||
$ apt-get install libxml2-dev libxslt-dev
|
||||
|
||||
|
||||
Regenerating config option tables
|
||||
---------------------------------
|
||||
|
||||
See :ref:`autogenerate_config_docs`.
|
||||
|
||||
@ -1,121 +0,0 @@
|
||||
.. _autogenerate_config_docs:
|
||||
|
||||
autogenerate_config_docs
|
||||
========================
|
||||
|
||||
Automatically generate configuration tables to document OpenStack.
|
||||
|
||||
Using the wrapper
|
||||
-----------------
|
||||
|
||||
``autohelp-wrapper`` is the recommended tool to generate the configuration
|
||||
tables. Don't bother using ``autohelp.py`` manually.
|
||||
|
||||
The ``autohelp-wrapper`` script installs a virtual environment and all the
|
||||
needed dependencies, clones or updates the projects and manuals repositories,
|
||||
then runs the ``autohelp.py`` script in the virtual environment.
|
||||
|
||||
New and updated flagmappings are generated in the ``openstack-manuals``
|
||||
repository (``tools/autogenerate-config-flagmappings/`` directory).
|
||||
|
||||
Prior to running the following commands, you need to install several development
|
||||
packages.
|
||||
|
||||
On Ubuntu:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ sudo apt-get install python-dev python-pip python-virtualenv \
|
||||
libxml2-dev libxslt1-dev zlib1g-dev \
|
||||
libmysqlclient-dev libpq-dev libffi-dev \
|
||||
libsqlite3-dev libldap2-dev libsasl2-dev \
|
||||
libjpeg-dev
|
||||
|
||||
On RHEL 7 and CentOS 7:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ sudo yum install https://www.rdoproject.org/repos/rdo-release.rpm
|
||||
$ sudo yum update
|
||||
$ sudo yum install python-devel python-pip python-virtualenv \
|
||||
libxml2-devel libxslt-devel zlib-devel \
|
||||
mariadb-devel postgresql-devel libffi-devel \
|
||||
sqlite-devel openldap-devel cyrus-sasl-devel \
|
||||
libjpeg-turbo-devel gcc git
|
||||
|
||||
.. note::
|
||||
* libjpeg is needed for ironic
|
||||
|
||||
The workflow is:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ pip install -rrequirements.txt
|
||||
$ ./autohelp-wrapper update
|
||||
$ $EDITOR sources/openstack-manuals/tools/autogenerate-config-flagmappings/*.flagmappings
|
||||
$ ./autohelp-wrapper rst
|
||||
$ # check the results in sources/openstack-manuals
|
||||
|
||||
This will generate the tables for all the known projects.
|
||||
Note for neutron project: If the driver/plugin resides outside the neutron
|
||||
repository, then the driver/plugin has to be explicitly installed within the
|
||||
virtual environment to generate the configuration options.
|
||||
|
||||
To generate the mappings and tables for a subset of projects, use the code
|
||||
names as arguments:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ ./autohelp-wrapper update cinder heat
|
||||
$ # edit the mappings files
|
||||
$ ./autohelp-wrapper rst cinder heat
|
||||
|
||||
|
||||
Flagmappings files
|
||||
------------------
|
||||
|
||||
The tool uses flagmapping files to map options to custom categories. Flag
|
||||
mapping files can be found in the ``tools/autogenerate-config-flagmappings``
|
||||
folder of the openstack-manuals project. Not all projects use flagmapping
|
||||
files, as those that do not will be disabled by the presence of a
|
||||
``$project.disable`` file in that folder. For those that do, however, the files
|
||||
use the following format::
|
||||
|
||||
OPTION_SECTION/OPTION_NAME group1 [group2, ...]
|
||||
|
||||
Groups need to be defined manually to organize the configuration tables.
|
||||
|
||||
The group values can only contain alphanumeric characters, _ and - (they will
|
||||
be used as document IDs).
|
||||
|
||||
To make the table titles more user friendly, create or edit the PROJECT.headers
|
||||
file in the manuals repository. Each line of this file is of the form:
|
||||
|
||||
::
|
||||
|
||||
GROUP A Nice Title
|
||||
|
||||
Working with branches
|
||||
---------------------
|
||||
|
||||
``autohelp-wrapper`` works on the master branch by default, but you can tell it
|
||||
to work on another branch:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ ./autohelp-wrapper -b stable/liberty update
|
||||
|
||||
.. note::
|
||||
The ``-b`` switch doesn't apply to the ``openstack-manuals`` repository
|
||||
which will be left untouched (no ``git branch``, no ``git update``).
|
||||
|
||||
|
||||
Generate configuration difference
|
||||
---------------------------------
|
||||
|
||||
To generate "New, updated, and deprecated options" for each service,
|
||||
run ``diff_branches.py``. For example:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ ./diff_branches.py stable/liberty stable/mitaka nova
|
||||
@ -1,18 +0,0 @@
|
||||
# Copyright 2012 OpenStack Foundation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import pbr.version
|
||||
|
||||
|
||||
__version__ = pbr.version.VersionInfo('openstack-doc-tools').version_string()
|
||||
@ -1,273 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
set -e
|
||||
|
||||
HERE=$(pwd)
|
||||
VENVDIR=$HERE/venv
|
||||
SOURCESDIR=$HERE/sources
|
||||
MANUALSREPO=$SOURCESDIR/openstack-manuals
|
||||
MAPPINGS_DIR=$MANUALSREPO/tools/autogenerate-config-flagmappings
|
||||
AUTOHELP="python $HERE/autohelp.py"
|
||||
GITBASE=${GITBASE:-git://git.openstack.org/openstack}
|
||||
GITPROJ=${GITPROJ:-git://git.openstack.org/openstack}
|
||||
PROJECTS="aodh ceilometer cinder glance heat ironic keystone manila \
|
||||
murano neutron nova sahara senlin trove zaqar"
|
||||
MANUALS_PROJECTS="openstack-manuals"
|
||||
BRANCH=master
|
||||
FAST=0
|
||||
QUIET=0
|
||||
CLONE_MANUALS=1
|
||||
|
||||
usage() {
|
||||
echo "Wrapper for autohelp.py"
|
||||
echo "Usage:"
|
||||
echo " $(basename $0) [ OPTIONS ] dump|update|rst|setup [ project... ]"
|
||||
echo
|
||||
echo "Subcommands:"
|
||||
echo " dump: Dumps the list of options with their attributes"
|
||||
echo " update: Update or create the flagmapping files"
|
||||
echo " rst: Generate the options tables in RST format"
|
||||
echo " setup: Install the environment only"
|
||||
echo
|
||||
echo "Options:"
|
||||
echo " -b BRANCH: Work on this branch (defaults to master)"
|
||||
echo " -g GITPROJ: Use this location for the project git repos "
|
||||
echo " (defaults to git://git.openstack.org/openstack)"
|
||||
echo " -c: Recreate the virtual environment"
|
||||
echo " -f: Work offline: Do not change environment or sources"
|
||||
echo " -e PATH: Create the virtualenv in PATH"
|
||||
echo " -v LEVEL: Verbose message (1 or 2)"
|
||||
echo " (check various python modules imported or not)"
|
||||
echo " -o OUTDIR: Path to output openstack-manuals directory "
|
||||
echo " (defaults to ./sources/openstack-manuals)"
|
||||
}
|
||||
|
||||
setup_venv() {
|
||||
project=$1
|
||||
|
||||
if [ ! -e $VENVDIR/$project/bin/activate ]; then
|
||||
mkdir -p $VENVDIR/$project
|
||||
virtualenv $VENVDIR/$project
|
||||
fi
|
||||
activate_venv $project
|
||||
}
|
||||
|
||||
activate_venv() {
|
||||
project=$1
|
||||
|
||||
. $VENVDIR/$project/bin/activate
|
||||
pip install --upgrade pip setuptools
|
||||
}
|
||||
|
||||
get_project() {
|
||||
project=$1
|
||||
git_url=$GITPROJ
|
||||
|
||||
if [ ! -e $SOURCESDIR/$project ]; then
|
||||
if [[ $MANUALS_PROJECTS =~ (^| )$project($| ) ]]; then
|
||||
git_url=$GITBASE
|
||||
fi
|
||||
git clone $git_url/$project $SOURCESDIR/$project
|
||||
|
||||
if [ -e $MAPPINGS_DIR/$project.extra_repos ]; then
|
||||
while read extra; do
|
||||
git clone $git_url/$extra $SOURCESDIR/$extra
|
||||
done < $MAPPINGS_DIR/$project.extra_repos
|
||||
fi
|
||||
|
||||
else
|
||||
if [ $project != openstack-manuals ]; then
|
||||
(cd $SOURCESDIR/$project && git pull)
|
||||
fi
|
||||
|
||||
if [ -e $MAPPINGS_DIR/$project.extra_repos ]; then
|
||||
while read extra; do
|
||||
(cd $SOURCESDIR/$extra && git pull)
|
||||
done < $MAPPINGS_DIR/$project.extra_repos
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
setup_tools() {
|
||||
pip install -rrequirements.txt
|
||||
}
|
||||
|
||||
while getopts :b:g:e:o:v:cfq opt; do
|
||||
case $opt in
|
||||
b)
|
||||
BRANCH=$OPTARG
|
||||
;;
|
||||
g)
|
||||
GITPROJ=$OPTARG
|
||||
;;
|
||||
c)
|
||||
rm -rf $VENVDIR
|
||||
;;
|
||||
e)
|
||||
VENVDIR=$OPTARG
|
||||
;;
|
||||
o)
|
||||
MANUALSREPO=$OPTARG
|
||||
MAPPINGS_DIR=$OPTARG/tools/autogenerate-config-flagmappings
|
||||
CLONE_MANUALS=0
|
||||
;;
|
||||
f)
|
||||
FAST=1
|
||||
;;
|
||||
q)
|
||||
QUIET=1
|
||||
;;
|
||||
v)
|
||||
AUTOOPT="-v"
|
||||
if [ $OPTARG = 2 ]; then
|
||||
AUTOOPT="-vv"
|
||||
fi
|
||||
;;
|
||||
\?)
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
shift $(($OPTIND - 1))
|
||||
|
||||
if [ $# -lt 1 ]; then
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ACTION=$1
|
||||
shift
|
||||
|
||||
if [ $QUIET -eq 1 ]; then
|
||||
exec 3>&1 >/dev/null
|
||||
exec 4>&2 2>/dev/null
|
||||
fi
|
||||
|
||||
case $ACTION in
|
||||
update|rst|dump|setup) ;;
|
||||
*)
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
if [ ! -e autohelp.py ]; then
|
||||
echo "Execute this script in the autogenerate_config_docs directory."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
[ $# != 0 ] && PROJECTS="$*"
|
||||
|
||||
RELEASE=$(echo $BRANCH | sed 's,^stable.,,')
|
||||
|
||||
if [ "$FAST" -eq 0 ] ; then
|
||||
if [ "$CLONE_MANUALS" -eq 1 ] ; then
|
||||
get_project openstack-manuals
|
||||
fi
|
||||
|
||||
for project in $PROJECTS; do
|
||||
setup_venv $project
|
||||
setup_tools
|
||||
if [ -e $MAPPINGS_DIR/$project.requirements ]; then
|
||||
pip install -r $MAPPINGS_DIR/$project.requirements \
|
||||
--allow-all-external
|
||||
fi
|
||||
get_project $project
|
||||
|
||||
(
|
||||
pushd $SOURCESDIR/$project
|
||||
module=$(echo $project | tr - _ )
|
||||
find $module -name "*.pyc" -delete
|
||||
GIT_CMD="git show-ref --verify --quiet refs/heads/$BRANCH"
|
||||
if $GIT_CMD; then
|
||||
git checkout $BRANCH
|
||||
else
|
||||
git checkout -b $BRANCH remotes/origin/$BRANCH
|
||||
fi
|
||||
pip install -rrequirements.txt
|
||||
[ -e "test-requirements.txt" ] && \
|
||||
pip install -rtest-requirements.txt
|
||||
python setup.py install
|
||||
popd
|
||||
|
||||
if [ -e $MAPPINGS_DIR/$project.extra_repos ]; then
|
||||
while read extra; do
|
||||
(
|
||||
cd $SOURCESDIR/$extra
|
||||
pip install -rrequirements.txt
|
||||
[ -e "test-requirements.txt" ] && \
|
||||
pip install -rtest-requirements.txt
|
||||
python setup.py install
|
||||
)
|
||||
done < $MAPPINGS_DIR/$project.extra_repos
|
||||
fi
|
||||
)
|
||||
done
|
||||
fi
|
||||
|
||||
for project in $PROJECTS; do
|
||||
echo "Working on $project..."
|
||||
activate_venv $project
|
||||
if [ "$ACTION" = "setup" ]; then
|
||||
break
|
||||
fi
|
||||
|
||||
if [ -e $MAPPINGS_DIR/$project.extra_repos ]; then
|
||||
extra_flags=
|
||||
while read extra; do
|
||||
package=$(echo $extra | tr - _)
|
||||
if [ $package = "networking_midonet" ]; then
|
||||
package="midonet"
|
||||
fi
|
||||
if [ $package = "networking_hyperv" ]; then
|
||||
package="hyperv"
|
||||
fi
|
||||
if [ $package = "networking_edge_vpn" ]; then
|
||||
package="networking-edge-vpn"
|
||||
fi
|
||||
if [ $package = "networking_zvm" ]; then
|
||||
package="neutron"
|
||||
cp -r $SOURCESDIR/networking-zvm/neutron/plugins/zvm \
|
||||
$SOURCESDIR/neutron/neutron/plugins
|
||||
cp -r \
|
||||
$SOURCESDIR/networking-zvm/neutron/plugins/ml2/drivers/zvm\
|
||||
$SOURCESDIR/neutron/neutron/plugins/ml2/drivers
|
||||
fi
|
||||
extra_flags="$extra_flags -i $SOURCESDIR/$extra/$package"
|
||||
done < $MAPPINGS_DIR/$project.extra_repos
|
||||
fi
|
||||
|
||||
cd $MAPPINGS_DIR
|
||||
|
||||
case $ACTION in
|
||||
update)
|
||||
$AUTOHELP update $project -i $SOURCESDIR/$project/$project \
|
||||
$extra_flags $AUTOOPT
|
||||
mv $project.flagmappings.new $project.flagmappings
|
||||
;;
|
||||
rst)
|
||||
$AUTOHELP rst $project -i $SOURCESDIR/$project/$project \
|
||||
$extra_flags $AUTOOPT
|
||||
;;
|
||||
dump)
|
||||
if [ $QUIET -eq 1 ]; then
|
||||
exec 1>&3
|
||||
exec 2>&4
|
||||
fi
|
||||
$AUTOHELP dump $project -i $SOURCESDIR/$project/$project \
|
||||
$extra_flags $AUTOOPT
|
||||
;;
|
||||
esac
|
||||
done
|
||||
@ -1,696 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
# A collection of tools for working with flags from OpenStack
|
||||
# packages and documentation.
|
||||
#
|
||||
# For an example of usage, run this program with the -h switch.
|
||||
#
|
||||
|
||||
# Must import this before argparse
|
||||
from oslo_config import cfg
|
||||
|
||||
import argparse
|
||||
import importlib
|
||||
import os
|
||||
import pickle
|
||||
import re
|
||||
import sys
|
||||
|
||||
import jinja2
|
||||
import stevedore
|
||||
|
||||
try:
|
||||
from sqlalchemy import exc
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
sys.path.insert(0, '.')
|
||||
try:
|
||||
from hooks import HOOKS # noqa
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
|
||||
EXTENSIONS = ['oslo.cache',
|
||||
'oslo.concurrency',
|
||||
'oslo.db',
|
||||
'oslo.log',
|
||||
'oslo.messaging',
|
||||
'oslo.middleware',
|
||||
'oslo.policy',
|
||||
'oslo.service']
|
||||
|
||||
_TYPE_DESCRIPTIONS = {
|
||||
cfg.StrOpt: 'String',
|
||||
cfg.BoolOpt: 'Boolean',
|
||||
cfg.IntOpt: 'Integer',
|
||||
cfg.FloatOpt: 'Floating point',
|
||||
cfg.ListOpt: 'List',
|
||||
cfg.DictOpt: 'Dict',
|
||||
cfg.MultiStrOpt: 'Multi-valued',
|
||||
cfg.IPOpt: 'IP address',
|
||||
cfg.PortOpt: 'Port number',
|
||||
cfg.HostnameOpt: 'Hostname',
|
||||
cfg.URIOpt: 'URI',
|
||||
cfg.HostAddressOpt: 'Host address',
|
||||
cfg._ConfigFileOpt: 'List of filenames',
|
||||
cfg._ConfigDirOpt: 'List of directory names',
|
||||
}
|
||||
|
||||
register_re = re.compile(r'''^ +.*\.register_opts\((?P<opts>[^,)]+)'''
|
||||
r'''(, (group=)?["'](?P<group>.*)["'])?\)''')
|
||||
|
||||
|
||||
def import_modules(repo_location, package_name, verbose=0):
|
||||
"""Import modules.
|
||||
|
||||
Loops through the repository, importing module by module to
|
||||
populate the configuration object (cfg.CONF) created from Oslo.
|
||||
"""
|
||||
|
||||
with open('ignore.list') as fd:
|
||||
ignore_list = [l for l in fd.read().split('\n')
|
||||
if l and (l[0] != '#')]
|
||||
|
||||
pkg_location = os.path.join(repo_location, package_name)
|
||||
for root, dirs, files in os.walk(pkg_location):
|
||||
skipdir = False
|
||||
for excludedir in ('tests', 'locale',
|
||||
os.path.join('db', 'migration'), 'transfer'):
|
||||
if ((os.path.sep + excludedir + os.path.sep) in root or (
|
||||
root.endswith(os.path.sep + excludedir))):
|
||||
skipdir = True
|
||||
break
|
||||
if skipdir:
|
||||
continue
|
||||
for pyfile in files:
|
||||
if pyfile.endswith('.py'):
|
||||
abs_path = os.path.join(root, pyfile)
|
||||
modfile = abs_path.split(repo_location, 1)[1]
|
||||
modname = os.path.splitext(modfile)[0].split(os.path.sep)
|
||||
modname = [m for m in modname if m != '']
|
||||
modname = '.'.join(modname)
|
||||
if modname.endswith('.__init__'):
|
||||
modname = modname[:modname.rfind(".")]
|
||||
if modname in ignore_list:
|
||||
continue
|
||||
try:
|
||||
module = importlib.import_module(modname)
|
||||
if verbose >= 1:
|
||||
print("imported %s" % modname)
|
||||
except ImportError as e:
|
||||
"""
|
||||
work around modules that don't like being imported in
|
||||
this way FIXME This could probably be better, but does
|
||||
not affect the configuration options found at this stage
|
||||
"""
|
||||
if verbose >= 2:
|
||||
print("Failed to import: %s (%s)" % (modname, e))
|
||||
continue
|
||||
except cfg.DuplicateOptError as e:
|
||||
"""
|
||||
oslo.cfg doesn't allow redefinition of a config option, but
|
||||
we don't mind. Don't fail if this happens.
|
||||
"""
|
||||
if verbose >= 2:
|
||||
print(e)
|
||||
continue
|
||||
except cfg.NoSuchGroupError as e:
|
||||
"""
|
||||
If a group doesn't exist, we ignore the import.
|
||||
"""
|
||||
if verbose >= 2:
|
||||
print(e)
|
||||
continue
|
||||
except exc.InvalidRequestError as e:
|
||||
if verbose >= 2:
|
||||
print(e)
|
||||
continue
|
||||
except Exception as e:
|
||||
print("Impossible to import module %s" % modname)
|
||||
raise
|
||||
|
||||
_register_runtime_opts(module, abs_path, verbose)
|
||||
_run_hook(modname)
|
||||
|
||||
# All the components provide keystone token authentication, usually using a
|
||||
# pipeline. Since the auth_token options can only be discovered at runtime
|
||||
# in this configuration, we force their discovery by importing the module.
|
||||
try:
|
||||
import keystonemiddleware.auth_token # noqa
|
||||
except cfg.DuplicateOptError:
|
||||
pass
|
||||
|
||||
|
||||
def _run_hook(modname):
|
||||
try:
|
||||
HOOKS[modname]()
|
||||
except KeyError:
|
||||
pass
|
||||
|
||||
|
||||
def _register_runtime_opts(module, abs_path, verbose):
|
||||
"""Handle options not registered on module import.
|
||||
|
||||
This function parses the .py files to discover calls to register_opts in
|
||||
functions and methods. It then explicitly call cfg.register_opt on each
|
||||
option to register (most of) them.
|
||||
"""
|
||||
|
||||
with open(abs_path) as fd:
|
||||
for line in fd:
|
||||
m = register_re.search(line)
|
||||
if not m:
|
||||
continue
|
||||
|
||||
opts_var = m.group('opts')
|
||||
opts_group = m.group('group')
|
||||
|
||||
# Get the object (an options list) from the opts_var string.
|
||||
# This requires parsing the string which can be of the form
|
||||
# 'foo.bar'. We treat each element as an attribute of the previous.
|
||||
register = True
|
||||
obj = module
|
||||
for item in opts_var.split('.'):
|
||||
try:
|
||||
obj = getattr(obj, item)
|
||||
except AttributeError:
|
||||
# FIXME(gpocentek): AttributeError is raised when a part of
|
||||
# the opts_var string is not an actual attribute. This will
|
||||
# need more parsing tricks.
|
||||
register = False
|
||||
if verbose >= 2:
|
||||
print("Ignoring %(obj)s in %(module)s" %
|
||||
{'obj': opts_var, 'module': module})
|
||||
break
|
||||
|
||||
if register and isinstance(obj, list):
|
||||
for opt in obj:
|
||||
if not isinstance(opt, cfg.Opt):
|
||||
continue
|
||||
try:
|
||||
cfg.CONF.register_opt(opt, opts_group)
|
||||
except cfg.DuplicateOptError:
|
||||
# ignore options that have already been registered
|
||||
pass
|
||||
|
||||
|
||||
def _sanitize_default(opt):
|
||||
"""Adapts unrealistic default values."""
|
||||
|
||||
# If the Oslo version is recent enough, we can use the 'sample_default'
|
||||
# attribute
|
||||
if (hasattr(opt, 'sample_default') and opt.sample_default is not None):
|
||||
return str(opt.sample_default)
|
||||
|
||||
if ((type(opt).__name__ == "ListOpt") and (type(opt.default) == list)):
|
||||
return ", ".join(str(item) for item in opt.default)
|
||||
|
||||
default = str(opt.default)
|
||||
|
||||
if default == os.uname()[1]:
|
||||
return 'localhost'
|
||||
|
||||
if opt.name == 'bindir':
|
||||
return '/usr/local/bin'
|
||||
|
||||
if opt.name == 'my_ip':
|
||||
return '10.0.0.1'
|
||||
|
||||
if isinstance(opt, cfg.StrOpt) and default.strip() != default:
|
||||
return '"%s"' % default
|
||||
|
||||
for pathelm in sys.path:
|
||||
if pathelm in ('.', ''):
|
||||
continue
|
||||
if pathelm.endswith('/'):
|
||||
pathelm = pathelm[:-1]
|
||||
if pathelm in default:
|
||||
default = re.sub(r'%s(/sources)?' % pathelm,
|
||||
'/usr/lib/python/site-packages', default)
|
||||
|
||||
return default
|
||||
|
||||
|
||||
def _get_overrides(package_name):
|
||||
overrides_file = '%s.overrides' % package_name
|
||||
if not os.path.exists(overrides_file):
|
||||
return {}
|
||||
overrides = {}
|
||||
with open(overrides_file) as fd:
|
||||
for line in fd:
|
||||
if line == '#':
|
||||
continue
|
||||
try:
|
||||
opt, sections = line.strip().split(' ', 1)
|
||||
sections = [x.strip() for x in sections.split(' ')]
|
||||
except ValueError:
|
||||
continue
|
||||
|
||||
overrides[opt] = sections
|
||||
|
||||
return overrides
|
||||
|
||||
|
||||
class OptionsCache(object):
|
||||
def __init__(self, overrides={}, verbose=0):
|
||||
self._verbose = verbose
|
||||
self._opts_by_name = {}
|
||||
self._opts_by_group = {}
|
||||
self._opt_names = []
|
||||
self._overrides = overrides
|
||||
|
||||
for optname in cfg.CONF._opts:
|
||||
opt = cfg.CONF._opts[optname]['opt']
|
||||
# We ignore some CLI opts by excluding SubCommandOpt objects
|
||||
if not isinstance(opt, cfg.SubCommandOpt):
|
||||
self._add_opt(optname, 'DEFAULT', opt)
|
||||
|
||||
for group in cfg.CONF._groups:
|
||||
for optname in cfg.CONF._groups[group]._opts:
|
||||
self._add_opt(group + '/' + optname, group,
|
||||
cfg.CONF._groups[group]._opts[optname]['opt'])
|
||||
|
||||
self._opt_names.sort(OptionsCache._cmpopts)
|
||||
|
||||
def _add_opt(self, optname, group, opt):
|
||||
if optname in self._opts_by_name:
|
||||
if self._verbose >= 2:
|
||||
print("Duplicate option name %s" % optname)
|
||||
return
|
||||
|
||||
opt.default = _sanitize_default(opt)
|
||||
|
||||
def fill(optname, group, opt):
|
||||
if optname in self._opts_by_name:
|
||||
return
|
||||
self._opts_by_name[optname] = (group, opt)
|
||||
self._opt_names.append(optname)
|
||||
|
||||
if group not in self._opts_by_group:
|
||||
self._opts_by_group[group] = []
|
||||
|
||||
self._opts_by_group[group].append(opt)
|
||||
|
||||
if optname in self._overrides:
|
||||
for new_group in self._overrides[optname]:
|
||||
if new_group == 'DEFAULT':
|
||||
new_optname = opt.name
|
||||
else:
|
||||
new_optname = new_group + '/' + opt.name
|
||||
fill(new_optname, new_group, opt)
|
||||
|
||||
else:
|
||||
fill(optname, group, opt)
|
||||
|
||||
def __len__(self):
|
||||
return len(self._opt_names)
|
||||
|
||||
def load_extension_options(self, module):
|
||||
# Note that options loaded this way aren't added to _opts_by_module
|
||||
loader = stevedore.named.NamedExtensionManager(
|
||||
'oslo.config.opts',
|
||||
names=(module,),
|
||||
invoke_on_load=False
|
||||
)
|
||||
for ext in loader:
|
||||
for group, opts in ext.plugin():
|
||||
for opt in opts:
|
||||
if group is None:
|
||||
self._add_opt(opt.dest, 'DEFAULT', opt)
|
||||
else:
|
||||
self._add_opt(group + '/' + opt.dest, group, opt)
|
||||
|
||||
self._opt_names.sort(OptionsCache._cmpopts)
|
||||
|
||||
def maybe_load_extensions(self, repositories):
|
||||
# Use the requirements.txt of the project to guess if an oslo module
|
||||
# needs to be imported
|
||||
needed_exts = set()
|
||||
for repo in repositories:
|
||||
base_path = os.path.dirname(repo)
|
||||
for ext in EXTENSIONS:
|
||||
requirements = os.path.join(base_path, 'requirements.txt')
|
||||
with open(requirements) as fd:
|
||||
for line in fd:
|
||||
if line.startswith(ext):
|
||||
needed_exts.add(ext)
|
||||
|
||||
for ext in needed_exts:
|
||||
self.load_extension_options(ext)
|
||||
|
||||
def get_group_names(self):
|
||||
return self._opts_by_group.keys()
|
||||
|
||||
def get_option_names(self):
|
||||
return self._opt_names
|
||||
|
||||
def get_group(self, name):
|
||||
return self._opts_by_group[name]
|
||||
|
||||
def get_option(self, name):
|
||||
return self._opts_by_name[name]
|
||||
|
||||
def dump(self):
|
||||
"""Dumps the list of options with their attributes.
|
||||
|
||||
This output is consumed by the diff_branches script.
|
||||
"""
|
||||
for name, (group, option) in self._opts_by_name.items():
|
||||
deprecated_opts = [{'group': deprecated.group,
|
||||
'name': deprecated.name}
|
||||
for deprecated in option.deprecated_opts]
|
||||
help_str = option.help.strip() if option.help else "None"
|
||||
new_option = {
|
||||
'default': option.default,
|
||||
'help': help_str,
|
||||
'deprecated_opts': deprecated_opts,
|
||||
'type': option.__class__.__name__.split('.')[-1]
|
||||
}
|
||||
self._opts_by_name[name] = (group, new_option)
|
||||
print(pickle.dumps(self._opts_by_name))
|
||||
|
||||
@staticmethod
|
||||
def _cmpopts(x, y):
|
||||
if '/' in x and '/' in y:
|
||||
prex = x[:x.find('/')]
|
||||
prey = y[:y.find('/')]
|
||||
if prex != prey:
|
||||
return cmp(prex, prey)
|
||||
return cmp(x, y)
|
||||
elif '/' in x:
|
||||
return 1
|
||||
elif '/' in y:
|
||||
return -1
|
||||
else:
|
||||
return cmp(x, y)
|
||||
|
||||
|
||||
def _use_categories(package_name):
|
||||
return not os.path.isfile(package_name + '.disable')
|
||||
|
||||
|
||||
def _get_options_by_cat(package_name):
|
||||
options_by_cat = {}
|
||||
|
||||
with open(package_name + '.flagmappings') as f:
|
||||
for line in f:
|
||||
if not line.strip() or line.startswith('#'):
|
||||
continue
|
||||
opt, categories = line.split(' ', 1)
|
||||
for category in categories.split():
|
||||
options_by_cat.setdefault(category, []).append(opt)
|
||||
|
||||
return options_by_cat
|
||||
|
||||
|
||||
def _get_category_names(package_name):
|
||||
package_headers = package_name + '.headers'
|
||||
category_names = {}
|
||||
for headers_file in ('shared.headers', package_headers):
|
||||
try:
|
||||
with open(headers_file) as f:
|
||||
for line in f:
|
||||
if not line.strip() or line.startswith('#'):
|
||||
continue
|
||||
cat, nice_name = line.split(' ', 1)
|
||||
category_names[cat] = nice_name.strip()
|
||||
except IOError:
|
||||
print("Cannot open %s (ignored)" % headers_file)
|
||||
|
||||
return category_names
|
||||
|
||||
|
||||
def _format_opt(option):
|
||||
|
||||
def _remove_prefix(text, prefix):
|
||||
if text.startswith(prefix):
|
||||
return text[len(prefix):].lstrip(':')
|
||||
return text
|
||||
|
||||
def _reflow_text(text):
|
||||
text = re.sub(r'\n+\s*\* ', '$sentinal$* ', text)
|
||||
text = text.replace('\n\n', '$sentinal$')
|
||||
text = text.replace('\n', ' ')
|
||||
text = ' '.join(text.split())
|
||||
return text.split('$sentinal$')
|
||||
|
||||
def _strip_indentation(text):
|
||||
return ' '.join([x.strip() for x in text.split('\n')]).strip()
|
||||
|
||||
help_text = option.help or "No help text available for this option."
|
||||
help_text = _remove_prefix(help_text.strip(), 'DEPRECATED')
|
||||
help_text = _reflow_text(help_text)
|
||||
|
||||
deprecated_text = (option.deprecated_reason or
|
||||
'No deprecation reason provided for this option.')
|
||||
deprecated_text = _strip_indentation(deprecated_text)
|
||||
|
||||
opt_type = _TYPE_DESCRIPTIONS.get(type(option), 'Unknown')
|
||||
|
||||
flags = []
|
||||
|
||||
if (option.deprecated_for_removal or
|
||||
(option.help and option.help.startswith('DEPRECATED'))):
|
||||
flags.append(('Deprecated', deprecated_text))
|
||||
if option.mutable:
|
||||
flags.append(('Mutable', 'This option can be changed without'
|
||||
' restarting.'))
|
||||
|
||||
return {
|
||||
'name': option.dest,
|
||||
'type': opt_type,
|
||||
'default': _sanitize_default(option),
|
||||
'help': help_text,
|
||||
'flags': flags
|
||||
}
|
||||
|
||||
|
||||
def write_files(package_name, options, target):
|
||||
"""Write tables.
|
||||
|
||||
Some projects make use of flagmapping files, while others make use
|
||||
of oslo.config's OptGroup to do the same in code. The function will
|
||||
handle both, printing a list of options by either category or group.
|
||||
"""
|
||||
target = target or '../../doc/config-reference/source/tables'
|
||||
if not os.path.isdir(target):
|
||||
os.makedirs(target)
|
||||
|
||||
if _use_categories(package_name):
|
||||
_write_files_by_category(package_name, options, target)
|
||||
else:
|
||||
_write_files_by_group(package_name, options, target)
|
||||
|
||||
|
||||
def _write_files_by_category(package_name, options, target):
|
||||
options_by_cat = _get_options_by_cat(package_name)
|
||||
category_names = _get_category_names(package_name)
|
||||
|
||||
for cat in options_by_cat.keys():
|
||||
env = {
|
||||
'pkg': package_name,
|
||||
'cat': cat,
|
||||
'label': '-'.join([package_name, cat]),
|
||||
'groups': [],
|
||||
'items': [],
|
||||
}
|
||||
|
||||
# Skip the options that is explicitly marked as disabled,
|
||||
# which is used for common configuration options.
|
||||
if cat == 'disable':
|
||||
continue
|
||||
|
||||
if cat in category_names:
|
||||
env['nice_cat'] = category_names[cat]
|
||||
else:
|
||||
env['nice_cat'] = cat
|
||||
print("No nicename for %s" % cat)
|
||||
|
||||
curgroup = None
|
||||
items = None
|
||||
for optname in options_by_cat[cat]:
|
||||
group, option = options.get_option(optname)
|
||||
|
||||
if group != curgroup:
|
||||
if group is not None:
|
||||
curgroup = group
|
||||
env['groups'].append(group)
|
||||
if items is not None:
|
||||
env['items'].append(items)
|
||||
items = []
|
||||
|
||||
items.append(_format_opt(option))
|
||||
|
||||
env['items'].append(items)
|
||||
|
||||
file_path = ("%(target)s/%(package_name)s-%(cat)s.rst" %
|
||||
{'target': target, 'package_name': package_name,
|
||||
'cat': cat})
|
||||
tmpl_file = os.path.join(os.path.dirname(__file__),
|
||||
'templates/autohelp-category.rst.j2')
|
||||
_write_template(tmpl_file, file_path, env)
|
||||
|
||||
|
||||
def _write_files_by_group(package_name, options, target):
|
||||
for group in options.get_group_names():
|
||||
env = {
|
||||
'pkg': package_name,
|
||||
'group': group,
|
||||
'label': '-'.join([package_name, group]),
|
||||
'items': [],
|
||||
}
|
||||
|
||||
for option in options.get_group(group):
|
||||
env['items'].append(_format_opt(option))
|
||||
|
||||
file_path = ("%(target)s/%(package_name)s-%(group)s.rst" %
|
||||
{'target': target, 'package_name': package_name,
|
||||
'group': group})
|
||||
tmpl_file = os.path.join(os.path.dirname(__file__),
|
||||
'templates/autohelp-group.rst.j2')
|
||||
_write_template(tmpl_file, file_path, env)
|
||||
|
||||
|
||||
def _write_template(template_path, output_path, env):
|
||||
with open(template_path) as fd:
|
||||
template = jinja2.Template(fd.read(), trim_blocks=True)
|
||||
output = template.render(filename=output_path, **env)
|
||||
|
||||
with open(output_path, 'w') as fd:
|
||||
fd.write(output)
|
||||
|
||||
|
||||
def update_flagmappings(package_name, options, verbose=0):
|
||||
"""Update flagmappings file.
|
||||
|
||||
Update a flagmappings file, adding or removing entries as needed.
|
||||
This will create a new file $package_name.flagmappings.new with
|
||||
category information merged from the existing $package_name.flagmappings.
|
||||
"""
|
||||
if not _use_categories:
|
||||
print("This project does not use flagmappings. Nothing to update.")
|
||||
return
|
||||
|
||||
original_flags = {}
|
||||
try:
|
||||
with open(package_name + '.flagmappings') as f:
|
||||
for line in f:
|
||||
try:
|
||||
flag, category = line.split(' ', 1)
|
||||
except ValueError:
|
||||
flag = line.strip()
|
||||
category = "Unknown"
|
||||
original_flags.setdefault(flag, []).append(category.strip())
|
||||
except IOError:
|
||||
# If the flags file doesn't exist we'll create it
|
||||
pass
|
||||
|
||||
updated_flags = []
|
||||
for opt in options.get_option_names():
|
||||
if len(original_flags.get(opt, [])) == 1:
|
||||
updated_flags.append((opt, original_flags[opt][0]))
|
||||
continue
|
||||
|
||||
updated_flags.append((opt, 'Unknown'))
|
||||
|
||||
with open(package_name + '.flagmappings.new', 'w') as f:
|
||||
for flag, category in updated_flags:
|
||||
f.write(flag + ' ' + category + '\n')
|
||||
|
||||
if verbose >= 1:
|
||||
removed_flags = (set(original_flags.keys()) -
|
||||
set([x[0] for x in updated_flags]))
|
||||
added_flags = (set([x[0] for x in updated_flags]) -
|
||||
set(original_flags.keys()))
|
||||
|
||||
print("\nRemoved Flags\n")
|
||||
for line in sorted(removed_flags, OptionsCache._cmpopts):
|
||||
print(line)
|
||||
|
||||
print("\nAdded Flags\n")
|
||||
for line in sorted(added_flags, OptionsCache._cmpopts):
|
||||
print(line)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Manage flag files, to aid in updating documentation.',
|
||||
usage='%(prog)s <cmd> <package> [options]')
|
||||
parser.add_argument('subcommand',
|
||||
help='Action (update, rst, dump).',
|
||||
choices=['update', 'rst', 'dump'])
|
||||
parser.add_argument('package',
|
||||
help='Name of the top-level package.')
|
||||
parser.add_argument('-v', '--verbose',
|
||||
action='count',
|
||||
default=0,
|
||||
dest='verbose',
|
||||
required=False,)
|
||||
parser.add_argument('-i', '--input',
|
||||
dest='repos',
|
||||
help='Path to a python package in which options '
|
||||
'should be discoverd. Can be used multiple '
|
||||
'times.',
|
||||
required=False,
|
||||
type=str,
|
||||
action='append')
|
||||
parser.add_argument('-o', '--output',
|
||||
dest='target',
|
||||
help='Directory or file in which data will be saved.\n'
|
||||
'Defaults to ../../doc/common/tables/ '
|
||||
'for "rst".\n'
|
||||
'Defaults to stdout for "dump"',
|
||||
required=False,
|
||||
type=str,)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.repos is None:
|
||||
args.repos = ['./sources/%s/%s' % args.package]
|
||||
|
||||
for repository in args.repos:
|
||||
package_name = os.path.basename(repository)
|
||||
base_path = os.path.dirname(repository)
|
||||
|
||||
sys.path.insert(0, base_path)
|
||||
try:
|
||||
__import__(package_name)
|
||||
except ImportError as e:
|
||||
if args.verbose >= 1:
|
||||
print(str(e))
|
||||
print("Failed to import: %s (%s)" % (package_name, e))
|
||||
|
||||
import_modules(base_path, package_name, verbose=args.verbose)
|
||||
sys.path.pop(0)
|
||||
|
||||
overrides = _get_overrides(package_name)
|
||||
options = OptionsCache(overrides, verbose=args.verbose)
|
||||
options.maybe_load_extensions(args.repos)
|
||||
|
||||
if args.verbose > 0:
|
||||
print("%s options imported from package %s." % (len(options),
|
||||
str(package_name)))
|
||||
|
||||
if args.subcommand == 'update':
|
||||
update_flagmappings(args.package, options, verbose=args.verbose)
|
||||
|
||||
elif args.subcommand == 'rst':
|
||||
write_files(args.package, options, args.target)
|
||||
|
||||
elif args.subcommand == 'dump':
|
||||
options.dump()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -1,289 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
# A collection of tools for working with flags from OpenStack
|
||||
# packages and documentation.
|
||||
#
|
||||
# For an example of usage, run this program with the -h switch.
|
||||
#
|
||||
import argparse
|
||||
import os
|
||||
import pickle
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import jinja2
|
||||
|
||||
|
||||
PROJECTS = ['aodh', 'ceilometer', 'cinder', 'glance', 'heat', 'ironic',
|
||||
'keystone', 'manila', 'neutron', 'nova', 'sahara', 'trove']
|
||||
MASTER_RELEASE = 'Ocata'
|
||||
CODENAME_TITLE = {'aodh': 'Alarming',
|
||||
'ceilometer': 'Telemetry',
|
||||
'cinder': 'Block Storage',
|
||||
'glance': 'Image service',
|
||||
'heat': 'Orchestration',
|
||||
'ironic': 'Bare Metal service',
|
||||
'keystone': 'Identity service',
|
||||
'manila': 'Shared File Systems service',
|
||||
'murano': 'Application Catalog service',
|
||||
'neutron': 'Networking',
|
||||
'nova': 'Compute',
|
||||
'sahara': 'Data Processing service',
|
||||
'senlin': 'Clustering service',
|
||||
'trove': 'Database service',
|
||||
'zaqar': 'Message service'}
|
||||
|
||||
|
||||
def setup_venv(projects, branch, novenvupdate):
|
||||
"""Setup a virtual environment for `branch`."""
|
||||
dirname = os.path.join('venv', branch.replace('/', '_'))
|
||||
if novenvupdate and os.path.exists(dirname):
|
||||
return
|
||||
if not os.path.exists('venv'):
|
||||
os.mkdir('venv')
|
||||
args = ["./autohelp-wrapper", "-b", branch, "-e", dirname, "setup"]
|
||||
args.extend(projects)
|
||||
if subprocess.call(args) != 0:
|
||||
print("Impossible to create the %s environment." % branch)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def _get_packages(project, branch):
|
||||
release = branch if '/' not in branch else branch.split('/')[1]
|
||||
packages = [project]
|
||||
try:
|
||||
with open('extra_repos/%s-%s.txt' % (project, release)) as f:
|
||||
packages.extend([p.strip() for p in f])
|
||||
except IOError:
|
||||
pass
|
||||
|
||||
return packages
|
||||
|
||||
|
||||
def get_options(project, branch):
|
||||
"""Get the list of known options for a project."""
|
||||
print("Working on %(project)s (%(branch)s)" % {'project': project,
|
||||
'branch': branch})
|
||||
# And run autohelp script to get a serialized dict of the discovered
|
||||
# options
|
||||
dirname = os.path.join('venv', branch.replace('/', '_'))
|
||||
args = ["./autohelp-wrapper", "-q", "-b", branch, "-e", dirname,
|
||||
"dump", project]
|
||||
|
||||
path = os.environ.get("PATH")
|
||||
bin_path = os.path.abspath(os.path.join(dirname, "bin"))
|
||||
path = "%s:%s" % (bin_path, path)
|
||||
serialized = subprocess.check_output(args,
|
||||
env={'VIRTUAL_ENV': dirname,
|
||||
'PATH': path})
|
||||
ret = pickle.loads(serialized)
|
||||
return ret
|
||||
|
||||
|
||||
def _cmpopts(x, y):
|
||||
"""Compare to option names.
|
||||
|
||||
The options can be of 2 forms: option_name or group/option_name. Options
|
||||
without a group always comes first. Options are sorted alphabetically
|
||||
inside a group.
|
||||
"""
|
||||
if '/' in x and '/' in y:
|
||||
prex = x[:x.find('/')]
|
||||
prey = y[:y.find('/')]
|
||||
if prex != prey:
|
||||
return cmp(prex, prey)
|
||||
return cmp(x, y)
|
||||
elif '/' in x:
|
||||
return 1
|
||||
elif '/' in y:
|
||||
return -1
|
||||
else:
|
||||
return cmp(x, y)
|
||||
|
||||
|
||||
def diff(old_list, new_list):
|
||||
"""Compare the old and new lists of options."""
|
||||
new_opts = []
|
||||
new_defaults = []
|
||||
deprecated_opts = []
|
||||
for name, (group, option) in new_list.items():
|
||||
# Find the new options
|
||||
if name not in old_list.viewkeys():
|
||||
new_opts.append(name)
|
||||
|
||||
# Find the options for which the default value has changed
|
||||
elif option['default'] != old_list[name][1]['default']:
|
||||
new_defaults.append(name)
|
||||
|
||||
# Find options that have been deprecated in the new release.
|
||||
# If an option name is a key in the old_list dict, it means that it
|
||||
# wasn't deprecated.
|
||||
|
||||
# Some options are deprecated, but not replaced with a new option.
|
||||
# These options usually contain 'DEPRECATED' in their help string.
|
||||
if 'DEPRECATED' in option['help']:
|
||||
deprecated_opts.append((name, None))
|
||||
|
||||
for deprecated in option['deprecated_opts']:
|
||||
# deprecated_opts is a list which always holds at least 1 invalid
|
||||
# dict. Forget it.
|
||||
if deprecated['name'] is None:
|
||||
continue
|
||||
|
||||
if deprecated['group'] in [None, 'DEFAULT']:
|
||||
full_name = deprecated['name']
|
||||
else:
|
||||
full_name = deprecated['group'] + '/' + deprecated['name']
|
||||
|
||||
if full_name in old_list.viewkeys():
|
||||
deprecated_opts.append((full_name, name))
|
||||
|
||||
return new_opts, new_defaults, deprecated_opts
|
||||
|
||||
|
||||
def format_option_name(name):
|
||||
"""Return a formatted string for the option path."""
|
||||
if name is None:
|
||||
return "None"
|
||||
|
||||
try:
|
||||
section, name = name.split('/')
|
||||
except ValueError:
|
||||
# name without a section ('log_dir')
|
||||
return "[DEFAULT] %s" % name
|
||||
|
||||
return "[%s] %s" % (section, name)
|
||||
|
||||
|
||||
def release_from_branch(branch):
|
||||
if branch == 'master':
|
||||
return MASTER_RELEASE
|
||||
else:
|
||||
return branch.replace('stable/', '').title()
|
||||
|
||||
|
||||
def get_env(project, new_branch, old_list, new_list):
|
||||
"""Generate the jinja2 environment for the defined branch and project."""
|
||||
new_opts, new_defaults, deprecated_opts = diff(old_list, new_list)
|
||||
release = release_from_branch(new_branch)
|
||||
|
||||
env = {
|
||||
'release': release,
|
||||
'project': project,
|
||||
'codename': CODENAME_TITLE[project],
|
||||
'new_opts': [],
|
||||
'new_defaults': [],
|
||||
'deprecated_opts': []
|
||||
}
|
||||
|
||||
# New options
|
||||
if new_opts:
|
||||
for name in sorted(new_opts, _cmpopts):
|
||||
opt = new_list[name][1]
|
||||
name = format_option_name(name)
|
||||
helptext = opt['help'].strip().replace('\n', ' ')
|
||||
helptext = ' '.join(helptext.split())
|
||||
cells = (("%(name)s = %(default)s" %
|
||||
{'name': name,
|
||||
'default': opt['default']}).strip(),
|
||||
"(%(type)s) %(help)s" % {'type': opt['type'],
|
||||
'help': helptext})
|
||||
env['new_opts'].append(cells)
|
||||
|
||||
# New defaults
|
||||
if new_defaults:
|
||||
for name in sorted(new_defaults, _cmpopts):
|
||||
old_default = old_list[name][1]['default']
|
||||
new_default = new_list[name][1]['default']
|
||||
if isinstance(old_default, list):
|
||||
old_default = ", ".join(old_default)
|
||||
if isinstance(new_default, list):
|
||||
new_default = ", ".join(new_default)
|
||||
name = format_option_name(name)
|
||||
cells = (name, old_default, new_default)
|
||||
env['new_defaults'].append(cells)
|
||||
|
||||
# Deprecated options
|
||||
if deprecated_opts:
|
||||
for deprecated, new in sorted(deprecated_opts, cmp=_cmpopts,
|
||||
key=lambda tup: tup[0]):
|
||||
deprecated = format_option_name(deprecated)
|
||||
new = format_option_name(new)
|
||||
env['deprecated_opts'].append((deprecated, new))
|
||||
|
||||
return env
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Generate a summary of configuration option changes.',
|
||||
usage='%(prog)s [options] <old_branch> <new_branch> [projects]')
|
||||
parser.add_argument('old_branch',
|
||||
help='Name of the old branch.')
|
||||
parser.add_argument('new_branch',
|
||||
help='Name of the new branch.')
|
||||
parser.add_argument('projects',
|
||||
help='List of projects to work on.',
|
||||
nargs='*',
|
||||
default=PROJECTS)
|
||||
parser.add_argument('-i', '--input',
|
||||
dest='sources',
|
||||
help='Path to a folder containing the git '
|
||||
'repositories.',
|
||||
required=False,
|
||||
default='./sources',
|
||||
type=str,)
|
||||
parser.add_argument('-o', '--output',
|
||||
dest='target',
|
||||
help='Directory or file in which data will be saved.\n'
|
||||
'Defaults to "."',
|
||||
required=False,
|
||||
default='.',
|
||||
type=str,)
|
||||
parser.add_argument('-n', '--no-venv-update',
|
||||
dest='novenvupdate',
|
||||
help='Don\'t update the virtual envs.',
|
||||
required=False,
|
||||
action='store_true',
|
||||
default=False,)
|
||||
args = parser.parse_args()
|
||||
|
||||
setup_venv(args.projects, args.old_branch, args.novenvupdate)
|
||||
setup_venv(args.projects, args.new_branch, args.novenvupdate)
|
||||
|
||||
for project in args.projects:
|
||||
old_list = get_options(project, args.old_branch)
|
||||
new_list = get_options(project, args.new_branch)
|
||||
|
||||
release = args.new_branch.replace('stable/', '')
|
||||
env = get_env(project, release, old_list, new_list)
|
||||
filename = ("%(project)s-conf-changes.rst" %
|
||||
{'project': project})
|
||||
tmpl_file = 'templates/changes.rst.j2'
|
||||
if not os.path.exists(args.target):
|
||||
os.makedirs(args.target)
|
||||
dest = os.path.join(args.target, filename)
|
||||
|
||||
with open(tmpl_file) as fd:
|
||||
template = jinja2.Template(fd.read(), trim_blocks=True)
|
||||
output = template.render(**env)
|
||||
|
||||
with open(dest, 'w') as fd:
|
||||
fd.write(output)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@ -1,5 +0,0 @@
|
||||
docutils
|
||||
jinja2
|
||||
lxml
|
||||
oslo.config
|
||||
oslo.i18n
|
||||
@ -1,45 +0,0 @@
|
||||
..
|
||||
Warning: Do not edit this file. It is automatically generated from the
|
||||
software project's code and your changes will be overwritten.
|
||||
|
||||
The tool to generate this file lives in openstack-doc-tools repository.
|
||||
|
||||
Please make any changes needed in the code, then run the
|
||||
autogenerate-config-doc tool from the openstack-doc-tools repository, or
|
||||
ask for help on the documentation mailing list, IRC channel or meeting.
|
||||
|
||||
.. _{{ label }}:
|
||||
|
||||
.. list-table:: Description of {{ nice_cat }} configuration options
|
||||
:header-rows: 1
|
||||
:class: config-ref-table
|
||||
|
||||
* - Configuration option = Default value
|
||||
- Description
|
||||
{% for group in groups %}
|
||||
|
||||
* - **[{{ group }}]**
|
||||
-
|
||||
{% for item in items[loop.index0] %}
|
||||
|
||||
{% if item['default'] is equalto '' %}
|
||||
* - ``{{ item['name'] }}`` =
|
||||
{% else %}
|
||||
* - ``{{ item['name'] }}`` = ``{{ item['default'] }}``
|
||||
{% endif %}
|
||||
{% for paragraph in item['help'] %}
|
||||
|
||||
{% if loop.first %}
|
||||
- ({{ item['type'] }}) {{ paragraph }}
|
||||
{% else %}
|
||||
{{ paragraph }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{% for flagname, flagdesc in item['flags'] %}
|
||||
|
||||
- **{{ flagname }}**
|
||||
|
||||
{{ flagdesc }}
|
||||
{% endfor %}
|
||||
{% endfor %}
|
||||
{% endfor %}
|
||||
@ -1,40 +0,0 @@
|
||||
..
|
||||
Warning: Do not edit this file. It is automatically generated from the
|
||||
software project's code and your changes will be overwritten.
|
||||
|
||||
The tool to generate this file lives in openstack-doc-tools repository.
|
||||
|
||||
Please make any changes needed in the code, then run the
|
||||
autogenerate-config-doc tool from the openstack-doc-tools repository, or
|
||||
ask for help on the documentation mailing list, IRC channel or meeting.
|
||||
|
||||
.. _{{ label }}:
|
||||
|
||||
.. list-table:: Description of {{ group }} configuration options
|
||||
:header-rows: 1
|
||||
:class: config-ref-table
|
||||
|
||||
* - Configuration option = Default value
|
||||
- Description
|
||||
{% for item in items %}
|
||||
|
||||
{% if item['default'] is equalto '' %}
|
||||
* - ``{{ item['name'] }}`` =
|
||||
{% else %}
|
||||
* - ``{{ item['name'] }}`` = ``{{ item['default'] }}``
|
||||
{% endif %}
|
||||
{% for paragraph in item['help'] %}
|
||||
|
||||
{% if loop.first %}
|
||||
- ({{ item['type'] }}) {{ paragraph }}
|
||||
{% else %}
|
||||
{{ paragraph }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{% for flagname, flagdesc in item['flags'] %}
|
||||
|
||||
- **{{ flagname }}**
|
||||
|
||||
{{ flagdesc }}
|
||||
{% endfor %}
|
||||
{% endfor %}
|
||||
@ -1,61 +0,0 @@
|
||||
New, updated, and deprecated options in {{ release }} for {{ codename }}
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{{ '~' * release|length }}~~~~~{{ '~' * codename|length }}
|
||||
|
||||
..
|
||||
Warning: Do not edit this file. It is automatically generated and your
|
||||
changes will be overwritten. The tool to do so lives in the
|
||||
openstack-doc-tools repository.
|
||||
|
||||
{% if new_opts %}
|
||||
.. list-table:: New options
|
||||
:header-rows: 1
|
||||
:class: config-ref-table
|
||||
|
||||
* - Option = default value
|
||||
- (Type) Help string
|
||||
{% for cells in new_opts %}
|
||||
* - ``{{ cells[0] }}``
|
||||
- {{ cells[1] }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if new_defaults %}
|
||||
.. list-table:: New default values
|
||||
:header-rows: 1
|
||||
:class: config-ref-table
|
||||
|
||||
* - Option
|
||||
- Previous default value
|
||||
- New default value
|
||||
{% for cells in new_defaults %}
|
||||
* - ``{{ cells[0] }}``
|
||||
{% if cells[1] is equalto '' %}
|
||||
-
|
||||
{% else %}
|
||||
- ``{{ cells[1] }}``
|
||||
{% endif %}
|
||||
{% if cells[2] is equalto '' %}
|
||||
-
|
||||
{% else %}
|
||||
- ``{{ cells[2] }}``
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if deprecated_opts %}
|
||||
.. list-table:: Deprecated options
|
||||
:header-rows: 1
|
||||
:class: config-ref-table
|
||||
|
||||
* - Deprecated option
|
||||
- New Option
|
||||
{% for cells in deprecated_opts %}
|
||||
* - ``{{ cells[0] }}``
|
||||
- ``{{ cells[1] }}``
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if not new_opts and not new_defaults and not deprecated_opts %}
|
||||
There are no new, updated, and deprecated options
|
||||
in {{ release }} for {{ codename }}.
|
||||
{% endif %}
|
||||
@ -1 +0,0 @@
|
||||
.. include:: ../../autogenerate_config_docs/README.rst
|
||||
@ -10,7 +10,6 @@ Contents:
|
||||
doc-tools-readme
|
||||
installation
|
||||
usage
|
||||
autogenerate_config_docs
|
||||
man/openstack-doc-test
|
||||
sitemap-readme
|
||||
release_notes
|
||||
|
||||
@ -0,0 +1,8 @@
|
||||
---
|
||||
upgrade:
|
||||
- |
|
||||
The `autogenerate_config_docs` set of tools has been removed. These were
|
||||
always buggy and hard to maintain, and they have been superseded by the
|
||||
``oslo_config.sphinxext`` `Sphinx extension`__.
|
||||
|
||||
__ https://docs.openstack.org/oslo.config/latest/reference/sphinxext.html
|
||||
@ -22,7 +22,6 @@ classifier =
|
||||
[files]
|
||||
packages =
|
||||
os_doc_tools
|
||||
autogenerate_config_docs
|
||||
data_files =
|
||||
share/openstack-doc-tools/sitemap = sitemap/*
|
||||
share/openstack-doc-tools/cleanup = cleanup/*
|
||||
|
||||
5
tox.ini
5
tox.ini
@ -25,8 +25,7 @@ commands =
|
||||
doc8 -e txt -e rst doc/source/ HACKING.rst
|
||||
# Run bashate during pep8 runs to ensure violations are caught by
|
||||
# the check and gate queues.
|
||||
bashate autogenerate_config_docs/autohelp-wrapper \
|
||||
bin/doc-tools-check-languages \
|
||||
bashate bin/doc-tools-check-languages \
|
||||
cleanup/remove_trailing_whitespaces.sh
|
||||
|
||||
[testenv:pylint]
|
||||
@ -58,7 +57,7 @@ usedevelop = False
|
||||
[flake8]
|
||||
show-source = True
|
||||
builtins = _
|
||||
exclude=.venv,.git,.tox,dist,*lib/python*,*egg,build,*autogenerate_config_docs/venv,*autogenerate_config_docs/sources,doc/source/conf.py
|
||||
exclude=.venv,.git,.tox,dist,*lib/python*,*egg,build,doc/source/conf.py
|
||||
# 28 is currently the most complex thing we have
|
||||
max-complexity=29
|
||||
ignore = H101
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user