diff --git a/README.rst b/README.rst index 08b8167e..77844d23 100644 --- a/README.rst +++ b/README.rst @@ -59,9 +59,3 @@ On openSUSE:: On Ubuntu:: $ apt-get install libxml2-dev libxslt-dev - - -Regenerating config option tables ---------------------------------- - -See :ref:`autogenerate_config_docs`. diff --git a/autogenerate_config_docs/README.rst b/autogenerate_config_docs/README.rst deleted file mode 100644 index 201d2644..00000000 --- a/autogenerate_config_docs/README.rst +++ /dev/null @@ -1,121 +0,0 @@ -.. _autogenerate_config_docs: - -autogenerate_config_docs -======================== - -Automatically generate configuration tables to document OpenStack. - -Using the wrapper ------------------ - -``autohelp-wrapper`` is the recommended tool to generate the configuration -tables. Don't bother using ``autohelp.py`` manually. - -The ``autohelp-wrapper`` script installs a virtual environment and all the -needed dependencies, clones or updates the projects and manuals repositories, -then runs the ``autohelp.py`` script in the virtual environment. - -New and updated flagmappings are generated in the ``openstack-manuals`` -repository (``tools/autogenerate-config-flagmappings/`` directory). - -Prior to running the following commands, you need to install several development -packages. - -On Ubuntu: - -.. code-block:: console - - $ sudo apt-get install python-dev python-pip python-virtualenv \ - libxml2-dev libxslt1-dev zlib1g-dev \ - libmysqlclient-dev libpq-dev libffi-dev \ - libsqlite3-dev libldap2-dev libsasl2-dev \ - libjpeg-dev - -On RHEL 7 and CentOS 7: - -.. code-block:: console - - $ sudo yum install https://www.rdoproject.org/repos/rdo-release.rpm - $ sudo yum update - $ sudo yum install python-devel python-pip python-virtualenv \ - libxml2-devel libxslt-devel zlib-devel \ - mariadb-devel postgresql-devel libffi-devel \ - sqlite-devel openldap-devel cyrus-sasl-devel \ - libjpeg-turbo-devel gcc git - -.. note:: - * libjpeg is needed for ironic - -The workflow is: - -.. code-block:: console - - $ pip install -rrequirements.txt - $ ./autohelp-wrapper update - $ $EDITOR sources/openstack-manuals/tools/autogenerate-config-flagmappings/*.flagmappings - $ ./autohelp-wrapper rst - $ # check the results in sources/openstack-manuals - -This will generate the tables for all the known projects. -Note for neutron project: If the driver/plugin resides outside the neutron -repository, then the driver/plugin has to be explicitly installed within the -virtual environment to generate the configuration options. - -To generate the mappings and tables for a subset of projects, use the code -names as arguments: - -.. code-block:: console - - $ ./autohelp-wrapper update cinder heat - $ # edit the mappings files - $ ./autohelp-wrapper rst cinder heat - - -Flagmappings files ------------------- - -The tool uses flagmapping files to map options to custom categories. Flag -mapping files can be found in the ``tools/autogenerate-config-flagmappings`` -folder of the openstack-manuals project. Not all projects use flagmapping -files, as those that do not will be disabled by the presence of a -``$project.disable`` file in that folder. For those that do, however, the files -use the following format:: - - OPTION_SECTION/OPTION_NAME group1 [group2, ...] - -Groups need to be defined manually to organize the configuration tables. - -The group values can only contain alphanumeric characters, _ and - (they will -be used as document IDs). - -To make the table titles more user friendly, create or edit the PROJECT.headers -file in the manuals repository. Each line of this file is of the form: - -:: - - GROUP A Nice Title - -Working with branches ---------------------- - -``autohelp-wrapper`` works on the master branch by default, but you can tell it -to work on another branch: - -.. code-block:: console - - $ ./autohelp-wrapper -b stable/liberty update - -.. note:: - The ``-b`` switch doesn't apply to the ``openstack-manuals`` repository - which will be left untouched (no ``git branch``, no ``git update``). - - -Generate configuration difference ---------------------------------- - -To generate "New, updated, and deprecated options" for each service, -run ``diff_branches.py``. For example: - -.. code-block:: console - - $ ./diff_branches.py stable/liberty stable/mitaka nova diff --git a/autogenerate_config_docs/__init__.py b/autogenerate_config_docs/__init__.py deleted file mode 100644 index 52601d6a..00000000 --- a/autogenerate_config_docs/__init__.py +++ /dev/null @@ -1,18 +0,0 @@ -# Copyright 2012 OpenStack Foundation -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. - -import pbr.version - - -__version__ = pbr.version.VersionInfo('openstack-doc-tools').version_string() diff --git a/autogenerate_config_docs/autohelp-wrapper b/autogenerate_config_docs/autohelp-wrapper deleted file mode 100755 index da643005..00000000 --- a/autogenerate_config_docs/autohelp-wrapper +++ /dev/null @@ -1,273 +0,0 @@ -#!/bin/bash -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. - -set -e - -HERE=$(pwd) -VENVDIR=$HERE/venv -SOURCESDIR=$HERE/sources -MANUALSREPO=$SOURCESDIR/openstack-manuals -MAPPINGS_DIR=$MANUALSREPO/tools/autogenerate-config-flagmappings -AUTOHELP="python $HERE/autohelp.py" -GITBASE=${GITBASE:-git://git.openstack.org/openstack} -GITPROJ=${GITPROJ:-git://git.openstack.org/openstack} -PROJECTS="aodh ceilometer cinder glance heat ironic keystone manila \ - murano neutron nova sahara senlin trove zaqar" -MANUALS_PROJECTS="openstack-manuals" -BRANCH=master -FAST=0 -QUIET=0 -CLONE_MANUALS=1 - -usage() { - echo "Wrapper for autohelp.py" - echo "Usage:" - echo " $(basename $0) [ OPTIONS ] dump|update|rst|setup [ project... ]" - echo - echo "Subcommands:" - echo " dump: Dumps the list of options with their attributes" - echo " update: Update or create the flagmapping files" - echo " rst: Generate the options tables in RST format" - echo " setup: Install the environment only" - echo - echo "Options:" - echo " -b BRANCH: Work on this branch (defaults to master)" - echo " -g GITPROJ: Use this location for the project git repos " - echo " (defaults to git://git.openstack.org/openstack)" - echo " -c: Recreate the virtual environment" - echo " -f: Work offline: Do not change environment or sources" - echo " -e PATH: Create the virtualenv in PATH" - echo " -v LEVEL: Verbose message (1 or 2)" - echo " (check various python modules imported or not)" - echo " -o OUTDIR: Path to output openstack-manuals directory " - echo " (defaults to ./sources/openstack-manuals)" -} - -setup_venv() { - project=$1 - - if [ ! -e $VENVDIR/$project/bin/activate ]; then - mkdir -p $VENVDIR/$project - virtualenv $VENVDIR/$project - fi - activate_venv $project -} - -activate_venv() { - project=$1 - - . $VENVDIR/$project/bin/activate - pip install --upgrade pip setuptools -} - -get_project() { - project=$1 - git_url=$GITPROJ - - if [ ! -e $SOURCESDIR/$project ]; then - if [[ $MANUALS_PROJECTS =~ (^| )$project($| ) ]]; then - git_url=$GITBASE - fi - git clone $git_url/$project $SOURCESDIR/$project - - if [ -e $MAPPINGS_DIR/$project.extra_repos ]; then - while read extra; do - git clone $git_url/$extra $SOURCESDIR/$extra - done < $MAPPINGS_DIR/$project.extra_repos - fi - - else - if [ $project != openstack-manuals ]; then - (cd $SOURCESDIR/$project && git pull) - fi - - if [ -e $MAPPINGS_DIR/$project.extra_repos ]; then - while read extra; do - (cd $SOURCESDIR/$extra && git pull) - done < $MAPPINGS_DIR/$project.extra_repos - fi - fi -} - -setup_tools() { - pip install -rrequirements.txt -} - -while getopts :b:g:e:o:v:cfq opt; do - case $opt in - b) - BRANCH=$OPTARG - ;; - g) - GITPROJ=$OPTARG - ;; - c) - rm -rf $VENVDIR - ;; - e) - VENVDIR=$OPTARG - ;; - o) - MANUALSREPO=$OPTARG - MAPPINGS_DIR=$OPTARG/tools/autogenerate-config-flagmappings - CLONE_MANUALS=0 - ;; - f) - FAST=1 - ;; - q) - QUIET=1 - ;; - v) - AUTOOPT="-v" - if [ $OPTARG = 2 ]; then - AUTOOPT="-vv" - fi - ;; - \?) - usage - exit 1 - ;; - esac -done -shift $(($OPTIND - 1)) - -if [ $# -lt 1 ]; then - usage - exit 1 -fi - -ACTION=$1 -shift - -if [ $QUIET -eq 1 ]; then - exec 3>&1 >/dev/null - exec 4>&2 2>/dev/null -fi - -case $ACTION in - update|rst|dump|setup) ;; - *) - usage - exit 1 - ;; -esac - -if [ ! -e autohelp.py ]; then - echo "Execute this script in the autogenerate_config_docs directory." - exit 1 -fi - -[ $# != 0 ] && PROJECTS="$*" - -RELEASE=$(echo $BRANCH | sed 's,^stable.,,') - -if [ "$FAST" -eq 0 ] ; then - if [ "$CLONE_MANUALS" -eq 1 ] ; then - get_project openstack-manuals - fi - - for project in $PROJECTS; do - setup_venv $project - setup_tools - if [ -e $MAPPINGS_DIR/$project.requirements ]; then - pip install -r $MAPPINGS_DIR/$project.requirements \ - --allow-all-external - fi - get_project $project - - ( - pushd $SOURCESDIR/$project - module=$(echo $project | tr - _ ) - find $module -name "*.pyc" -delete - GIT_CMD="git show-ref --verify --quiet refs/heads/$BRANCH" - if $GIT_CMD; then - git checkout $BRANCH - else - git checkout -b $BRANCH remotes/origin/$BRANCH - fi - pip install -rrequirements.txt - [ -e "test-requirements.txt" ] && \ - pip install -rtest-requirements.txt - python setup.py install - popd - - if [ -e $MAPPINGS_DIR/$project.extra_repos ]; then - while read extra; do - ( - cd $SOURCESDIR/$extra - pip install -rrequirements.txt - [ -e "test-requirements.txt" ] && \ - pip install -rtest-requirements.txt - python setup.py install - ) - done < $MAPPINGS_DIR/$project.extra_repos - fi - ) - done -fi - -for project in $PROJECTS; do - echo "Working on $project..." - activate_venv $project - if [ "$ACTION" = "setup" ]; then - break - fi - - if [ -e $MAPPINGS_DIR/$project.extra_repos ]; then - extra_flags= - while read extra; do - package=$(echo $extra | tr - _) - if [ $package = "networking_midonet" ]; then - package="midonet" - fi - if [ $package = "networking_hyperv" ]; then - package="hyperv" - fi - if [ $package = "networking_edge_vpn" ]; then - package="networking-edge-vpn" - fi - if [ $package = "networking_zvm" ]; then - package="neutron" - cp -r $SOURCESDIR/networking-zvm/neutron/plugins/zvm \ - $SOURCESDIR/neutron/neutron/plugins - cp -r \ - $SOURCESDIR/networking-zvm/neutron/plugins/ml2/drivers/zvm\ - $SOURCESDIR/neutron/neutron/plugins/ml2/drivers - fi - extra_flags="$extra_flags -i $SOURCESDIR/$extra/$package" - done < $MAPPINGS_DIR/$project.extra_repos - fi - - cd $MAPPINGS_DIR - - case $ACTION in - update) - $AUTOHELP update $project -i $SOURCESDIR/$project/$project \ - $extra_flags $AUTOOPT - mv $project.flagmappings.new $project.flagmappings - ;; - rst) - $AUTOHELP rst $project -i $SOURCESDIR/$project/$project \ - $extra_flags $AUTOOPT - ;; - dump) - if [ $QUIET -eq 1 ]; then - exec 1>&3 - exec 2>&4 - fi - $AUTOHELP dump $project -i $SOURCESDIR/$project/$project \ - $extra_flags $AUTOOPT - ;; - esac -done diff --git a/autogenerate_config_docs/autohelp.py b/autogenerate_config_docs/autohelp.py deleted file mode 100755 index 66d7cfd5..00000000 --- a/autogenerate_config_docs/autohelp.py +++ /dev/null @@ -1,696 +0,0 @@ -#!/usr/bin/env python -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. -# -# A collection of tools for working with flags from OpenStack -# packages and documentation. -# -# For an example of usage, run this program with the -h switch. -# - -# Must import this before argparse -from oslo_config import cfg - -import argparse -import importlib -import os -import pickle -import re -import sys - -import jinja2 -import stevedore - -try: - from sqlalchemy import exc -except Exception: - pass - -sys.path.insert(0, '.') -try: - from hooks import HOOKS # noqa -except ImportError: - pass - - -EXTENSIONS = ['oslo.cache', - 'oslo.concurrency', - 'oslo.db', - 'oslo.log', - 'oslo.messaging', - 'oslo.middleware', - 'oslo.policy', - 'oslo.service'] - -_TYPE_DESCRIPTIONS = { - cfg.StrOpt: 'String', - cfg.BoolOpt: 'Boolean', - cfg.IntOpt: 'Integer', - cfg.FloatOpt: 'Floating point', - cfg.ListOpt: 'List', - cfg.DictOpt: 'Dict', - cfg.MultiStrOpt: 'Multi-valued', - cfg.IPOpt: 'IP address', - cfg.PortOpt: 'Port number', - cfg.HostnameOpt: 'Hostname', - cfg.URIOpt: 'URI', - cfg.HostAddressOpt: 'Host address', - cfg._ConfigFileOpt: 'List of filenames', - cfg._ConfigDirOpt: 'List of directory names', -} - -register_re = re.compile(r'''^ +.*\.register_opts\((?P[^,)]+)''' - r'''(, (group=)?["'](?P.*)["'])?\)''') - - -def import_modules(repo_location, package_name, verbose=0): - """Import modules. - - Loops through the repository, importing module by module to - populate the configuration object (cfg.CONF) created from Oslo. - """ - - with open('ignore.list') as fd: - ignore_list = [l for l in fd.read().split('\n') - if l and (l[0] != '#')] - - pkg_location = os.path.join(repo_location, package_name) - for root, dirs, files in os.walk(pkg_location): - skipdir = False - for excludedir in ('tests', 'locale', - os.path.join('db', 'migration'), 'transfer'): - if ((os.path.sep + excludedir + os.path.sep) in root or ( - root.endswith(os.path.sep + excludedir))): - skipdir = True - break - if skipdir: - continue - for pyfile in files: - if pyfile.endswith('.py'): - abs_path = os.path.join(root, pyfile) - modfile = abs_path.split(repo_location, 1)[1] - modname = os.path.splitext(modfile)[0].split(os.path.sep) - modname = [m for m in modname if m != ''] - modname = '.'.join(modname) - if modname.endswith('.__init__'): - modname = modname[:modname.rfind(".")] - if modname in ignore_list: - continue - try: - module = importlib.import_module(modname) - if verbose >= 1: - print("imported %s" % modname) - except ImportError as e: - """ - work around modules that don't like being imported in - this way FIXME This could probably be better, but does - not affect the configuration options found at this stage - """ - if verbose >= 2: - print("Failed to import: %s (%s)" % (modname, e)) - continue - except cfg.DuplicateOptError as e: - """ - oslo.cfg doesn't allow redefinition of a config option, but - we don't mind. Don't fail if this happens. - """ - if verbose >= 2: - print(e) - continue - except cfg.NoSuchGroupError as e: - """ - If a group doesn't exist, we ignore the import. - """ - if verbose >= 2: - print(e) - continue - except exc.InvalidRequestError as e: - if verbose >= 2: - print(e) - continue - except Exception as e: - print("Impossible to import module %s" % modname) - raise - - _register_runtime_opts(module, abs_path, verbose) - _run_hook(modname) - - # All the components provide keystone token authentication, usually using a - # pipeline. Since the auth_token options can only be discovered at runtime - # in this configuration, we force their discovery by importing the module. - try: - import keystonemiddleware.auth_token # noqa - except cfg.DuplicateOptError: - pass - - -def _run_hook(modname): - try: - HOOKS[modname]() - except KeyError: - pass - - -def _register_runtime_opts(module, abs_path, verbose): - """Handle options not registered on module import. - - This function parses the .py files to discover calls to register_opts in - functions and methods. It then explicitly call cfg.register_opt on each - option to register (most of) them. - """ - - with open(abs_path) as fd: - for line in fd: - m = register_re.search(line) - if not m: - continue - - opts_var = m.group('opts') - opts_group = m.group('group') - - # Get the object (an options list) from the opts_var string. - # This requires parsing the string which can be of the form - # 'foo.bar'. We treat each element as an attribute of the previous. - register = True - obj = module - for item in opts_var.split('.'): - try: - obj = getattr(obj, item) - except AttributeError: - # FIXME(gpocentek): AttributeError is raised when a part of - # the opts_var string is not an actual attribute. This will - # need more parsing tricks. - register = False - if verbose >= 2: - print("Ignoring %(obj)s in %(module)s" % - {'obj': opts_var, 'module': module}) - break - - if register and isinstance(obj, list): - for opt in obj: - if not isinstance(opt, cfg.Opt): - continue - try: - cfg.CONF.register_opt(opt, opts_group) - except cfg.DuplicateOptError: - # ignore options that have already been registered - pass - - -def _sanitize_default(opt): - """Adapts unrealistic default values.""" - - # If the Oslo version is recent enough, we can use the 'sample_default' - # attribute - if (hasattr(opt, 'sample_default') and opt.sample_default is not None): - return str(opt.sample_default) - - if ((type(opt).__name__ == "ListOpt") and (type(opt.default) == list)): - return ", ".join(str(item) for item in opt.default) - - default = str(opt.default) - - if default == os.uname()[1]: - return 'localhost' - - if opt.name == 'bindir': - return '/usr/local/bin' - - if opt.name == 'my_ip': - return '10.0.0.1' - - if isinstance(opt, cfg.StrOpt) and default.strip() != default: - return '"%s"' % default - - for pathelm in sys.path: - if pathelm in ('.', ''): - continue - if pathelm.endswith('/'): - pathelm = pathelm[:-1] - if pathelm in default: - default = re.sub(r'%s(/sources)?' % pathelm, - '/usr/lib/python/site-packages', default) - - return default - - -def _get_overrides(package_name): - overrides_file = '%s.overrides' % package_name - if not os.path.exists(overrides_file): - return {} - overrides = {} - with open(overrides_file) as fd: - for line in fd: - if line == '#': - continue - try: - opt, sections = line.strip().split(' ', 1) - sections = [x.strip() for x in sections.split(' ')] - except ValueError: - continue - - overrides[opt] = sections - - return overrides - - -class OptionsCache(object): - def __init__(self, overrides={}, verbose=0): - self._verbose = verbose - self._opts_by_name = {} - self._opts_by_group = {} - self._opt_names = [] - self._overrides = overrides - - for optname in cfg.CONF._opts: - opt = cfg.CONF._opts[optname]['opt'] - # We ignore some CLI opts by excluding SubCommandOpt objects - if not isinstance(opt, cfg.SubCommandOpt): - self._add_opt(optname, 'DEFAULT', opt) - - for group in cfg.CONF._groups: - for optname in cfg.CONF._groups[group]._opts: - self._add_opt(group + '/' + optname, group, - cfg.CONF._groups[group]._opts[optname]['opt']) - - self._opt_names.sort(OptionsCache._cmpopts) - - def _add_opt(self, optname, group, opt): - if optname in self._opts_by_name: - if self._verbose >= 2: - print("Duplicate option name %s" % optname) - return - - opt.default = _sanitize_default(opt) - - def fill(optname, group, opt): - if optname in self._opts_by_name: - return - self._opts_by_name[optname] = (group, opt) - self._opt_names.append(optname) - - if group not in self._opts_by_group: - self._opts_by_group[group] = [] - - self._opts_by_group[group].append(opt) - - if optname in self._overrides: - for new_group in self._overrides[optname]: - if new_group == 'DEFAULT': - new_optname = opt.name - else: - new_optname = new_group + '/' + opt.name - fill(new_optname, new_group, opt) - - else: - fill(optname, group, opt) - - def __len__(self): - return len(self._opt_names) - - def load_extension_options(self, module): - # Note that options loaded this way aren't added to _opts_by_module - loader = stevedore.named.NamedExtensionManager( - 'oslo.config.opts', - names=(module,), - invoke_on_load=False - ) - for ext in loader: - for group, opts in ext.plugin(): - for opt in opts: - if group is None: - self._add_opt(opt.dest, 'DEFAULT', opt) - else: - self._add_opt(group + '/' + opt.dest, group, opt) - - self._opt_names.sort(OptionsCache._cmpopts) - - def maybe_load_extensions(self, repositories): - # Use the requirements.txt of the project to guess if an oslo module - # needs to be imported - needed_exts = set() - for repo in repositories: - base_path = os.path.dirname(repo) - for ext in EXTENSIONS: - requirements = os.path.join(base_path, 'requirements.txt') - with open(requirements) as fd: - for line in fd: - if line.startswith(ext): - needed_exts.add(ext) - - for ext in needed_exts: - self.load_extension_options(ext) - - def get_group_names(self): - return self._opts_by_group.keys() - - def get_option_names(self): - return self._opt_names - - def get_group(self, name): - return self._opts_by_group[name] - - def get_option(self, name): - return self._opts_by_name[name] - - def dump(self): - """Dumps the list of options with their attributes. - - This output is consumed by the diff_branches script. - """ - for name, (group, option) in self._opts_by_name.items(): - deprecated_opts = [{'group': deprecated.group, - 'name': deprecated.name} - for deprecated in option.deprecated_opts] - help_str = option.help.strip() if option.help else "None" - new_option = { - 'default': option.default, - 'help': help_str, - 'deprecated_opts': deprecated_opts, - 'type': option.__class__.__name__.split('.')[-1] - } - self._opts_by_name[name] = (group, new_option) - print(pickle.dumps(self._opts_by_name)) - - @staticmethod - def _cmpopts(x, y): - if '/' in x and '/' in y: - prex = x[:x.find('/')] - prey = y[:y.find('/')] - if prex != prey: - return cmp(prex, prey) - return cmp(x, y) - elif '/' in x: - return 1 - elif '/' in y: - return -1 - else: - return cmp(x, y) - - -def _use_categories(package_name): - return not os.path.isfile(package_name + '.disable') - - -def _get_options_by_cat(package_name): - options_by_cat = {} - - with open(package_name + '.flagmappings') as f: - for line in f: - if not line.strip() or line.startswith('#'): - continue - opt, categories = line.split(' ', 1) - for category in categories.split(): - options_by_cat.setdefault(category, []).append(opt) - - return options_by_cat - - -def _get_category_names(package_name): - package_headers = package_name + '.headers' - category_names = {} - for headers_file in ('shared.headers', package_headers): - try: - with open(headers_file) as f: - for line in f: - if not line.strip() or line.startswith('#'): - continue - cat, nice_name = line.split(' ', 1) - category_names[cat] = nice_name.strip() - except IOError: - print("Cannot open %s (ignored)" % headers_file) - - return category_names - - -def _format_opt(option): - - def _remove_prefix(text, prefix): - if text.startswith(prefix): - return text[len(prefix):].lstrip(':') - return text - - def _reflow_text(text): - text = re.sub(r'\n+\s*\* ', '$sentinal$* ', text) - text = text.replace('\n\n', '$sentinal$') - text = text.replace('\n', ' ') - text = ' '.join(text.split()) - return text.split('$sentinal$') - - def _strip_indentation(text): - return ' '.join([x.strip() for x in text.split('\n')]).strip() - - help_text = option.help or "No help text available for this option." - help_text = _remove_prefix(help_text.strip(), 'DEPRECATED') - help_text = _reflow_text(help_text) - - deprecated_text = (option.deprecated_reason or - 'No deprecation reason provided for this option.') - deprecated_text = _strip_indentation(deprecated_text) - - opt_type = _TYPE_DESCRIPTIONS.get(type(option), 'Unknown') - - flags = [] - - if (option.deprecated_for_removal or - (option.help and option.help.startswith('DEPRECATED'))): - flags.append(('Deprecated', deprecated_text)) - if option.mutable: - flags.append(('Mutable', 'This option can be changed without' - ' restarting.')) - - return { - 'name': option.dest, - 'type': opt_type, - 'default': _sanitize_default(option), - 'help': help_text, - 'flags': flags - } - - -def write_files(package_name, options, target): - """Write tables. - - Some projects make use of flagmapping files, while others make use - of oslo.config's OptGroup to do the same in code. The function will - handle both, printing a list of options by either category or group. - """ - target = target or '../../doc/config-reference/source/tables' - if not os.path.isdir(target): - os.makedirs(target) - - if _use_categories(package_name): - _write_files_by_category(package_name, options, target) - else: - _write_files_by_group(package_name, options, target) - - -def _write_files_by_category(package_name, options, target): - options_by_cat = _get_options_by_cat(package_name) - category_names = _get_category_names(package_name) - - for cat in options_by_cat.keys(): - env = { - 'pkg': package_name, - 'cat': cat, - 'label': '-'.join([package_name, cat]), - 'groups': [], - 'items': [], - } - - # Skip the options that is explicitly marked as disabled, - # which is used for common configuration options. - if cat == 'disable': - continue - - if cat in category_names: - env['nice_cat'] = category_names[cat] - else: - env['nice_cat'] = cat - print("No nicename for %s" % cat) - - curgroup = None - items = None - for optname in options_by_cat[cat]: - group, option = options.get_option(optname) - - if group != curgroup: - if group is not None: - curgroup = group - env['groups'].append(group) - if items is not None: - env['items'].append(items) - items = [] - - items.append(_format_opt(option)) - - env['items'].append(items) - - file_path = ("%(target)s/%(package_name)s-%(cat)s.rst" % - {'target': target, 'package_name': package_name, - 'cat': cat}) - tmpl_file = os.path.join(os.path.dirname(__file__), - 'templates/autohelp-category.rst.j2') - _write_template(tmpl_file, file_path, env) - - -def _write_files_by_group(package_name, options, target): - for group in options.get_group_names(): - env = { - 'pkg': package_name, - 'group': group, - 'label': '-'.join([package_name, group]), - 'items': [], - } - - for option in options.get_group(group): - env['items'].append(_format_opt(option)) - - file_path = ("%(target)s/%(package_name)s-%(group)s.rst" % - {'target': target, 'package_name': package_name, - 'group': group}) - tmpl_file = os.path.join(os.path.dirname(__file__), - 'templates/autohelp-group.rst.j2') - _write_template(tmpl_file, file_path, env) - - -def _write_template(template_path, output_path, env): - with open(template_path) as fd: - template = jinja2.Template(fd.read(), trim_blocks=True) - output = template.render(filename=output_path, **env) - - with open(output_path, 'w') as fd: - fd.write(output) - - -def update_flagmappings(package_name, options, verbose=0): - """Update flagmappings file. - - Update a flagmappings file, adding or removing entries as needed. - This will create a new file $package_name.flagmappings.new with - category information merged from the existing $package_name.flagmappings. - """ - if not _use_categories: - print("This project does not use flagmappings. Nothing to update.") - return - - original_flags = {} - try: - with open(package_name + '.flagmappings') as f: - for line in f: - try: - flag, category = line.split(' ', 1) - except ValueError: - flag = line.strip() - category = "Unknown" - original_flags.setdefault(flag, []).append(category.strip()) - except IOError: - # If the flags file doesn't exist we'll create it - pass - - updated_flags = [] - for opt in options.get_option_names(): - if len(original_flags.get(opt, [])) == 1: - updated_flags.append((opt, original_flags[opt][0])) - continue - - updated_flags.append((opt, 'Unknown')) - - with open(package_name + '.flagmappings.new', 'w') as f: - for flag, category in updated_flags: - f.write(flag + ' ' + category + '\n') - - if verbose >= 1: - removed_flags = (set(original_flags.keys()) - - set([x[0] for x in updated_flags])) - added_flags = (set([x[0] for x in updated_flags]) - - set(original_flags.keys())) - - print("\nRemoved Flags\n") - for line in sorted(removed_flags, OptionsCache._cmpopts): - print(line) - - print("\nAdded Flags\n") - for line in sorted(added_flags, OptionsCache._cmpopts): - print(line) - - -def main(): - parser = argparse.ArgumentParser( - description='Manage flag files, to aid in updating documentation.', - usage='%(prog)s [options]') - parser.add_argument('subcommand', - help='Action (update, rst, dump).', - choices=['update', 'rst', 'dump']) - parser.add_argument('package', - help='Name of the top-level package.') - parser.add_argument('-v', '--verbose', - action='count', - default=0, - dest='verbose', - required=False,) - parser.add_argument('-i', '--input', - dest='repos', - help='Path to a python package in which options ' - 'should be discoverd. Can be used multiple ' - 'times.', - required=False, - type=str, - action='append') - parser.add_argument('-o', '--output', - dest='target', - help='Directory or file in which data will be saved.\n' - 'Defaults to ../../doc/common/tables/ ' - 'for "rst".\n' - 'Defaults to stdout for "dump"', - required=False, - type=str,) - args = parser.parse_args() - - if args.repos is None: - args.repos = ['./sources/%s/%s' % args.package] - - for repository in args.repos: - package_name = os.path.basename(repository) - base_path = os.path.dirname(repository) - - sys.path.insert(0, base_path) - try: - __import__(package_name) - except ImportError as e: - if args.verbose >= 1: - print(str(e)) - print("Failed to import: %s (%s)" % (package_name, e)) - - import_modules(base_path, package_name, verbose=args.verbose) - sys.path.pop(0) - - overrides = _get_overrides(package_name) - options = OptionsCache(overrides, verbose=args.verbose) - options.maybe_load_extensions(args.repos) - - if args.verbose > 0: - print("%s options imported from package %s." % (len(options), - str(package_name))) - - if args.subcommand == 'update': - update_flagmappings(args.package, options, verbose=args.verbose) - - elif args.subcommand == 'rst': - write_files(args.package, options, args.target) - - elif args.subcommand == 'dump': - options.dump() - - -if __name__ == "__main__": - main() diff --git a/autogenerate_config_docs/diff_branches.py b/autogenerate_config_docs/diff_branches.py deleted file mode 100755 index 5b8ee082..00000000 --- a/autogenerate_config_docs/diff_branches.py +++ /dev/null @@ -1,289 +0,0 @@ -#!/usr/bin/env python -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. -# -# A collection of tools for working with flags from OpenStack -# packages and documentation. -# -# For an example of usage, run this program with the -h switch. -# -import argparse -import os -import pickle -import subprocess -import sys - -import jinja2 - - -PROJECTS = ['aodh', 'ceilometer', 'cinder', 'glance', 'heat', 'ironic', - 'keystone', 'manila', 'neutron', 'nova', 'sahara', 'trove'] -MASTER_RELEASE = 'Ocata' -CODENAME_TITLE = {'aodh': 'Alarming', - 'ceilometer': 'Telemetry', - 'cinder': 'Block Storage', - 'glance': 'Image service', - 'heat': 'Orchestration', - 'ironic': 'Bare Metal service', - 'keystone': 'Identity service', - 'manila': 'Shared File Systems service', - 'murano': 'Application Catalog service', - 'neutron': 'Networking', - 'nova': 'Compute', - 'sahara': 'Data Processing service', - 'senlin': 'Clustering service', - 'trove': 'Database service', - 'zaqar': 'Message service'} - - -def setup_venv(projects, branch, novenvupdate): - """Setup a virtual environment for `branch`.""" - dirname = os.path.join('venv', branch.replace('/', '_')) - if novenvupdate and os.path.exists(dirname): - return - if not os.path.exists('venv'): - os.mkdir('venv') - args = ["./autohelp-wrapper", "-b", branch, "-e", dirname, "setup"] - args.extend(projects) - if subprocess.call(args) != 0: - print("Impossible to create the %s environment." % branch) - sys.exit(1) - - -def _get_packages(project, branch): - release = branch if '/' not in branch else branch.split('/')[1] - packages = [project] - try: - with open('extra_repos/%s-%s.txt' % (project, release)) as f: - packages.extend([p.strip() for p in f]) - except IOError: - pass - - return packages - - -def get_options(project, branch): - """Get the list of known options for a project.""" - print("Working on %(project)s (%(branch)s)" % {'project': project, - 'branch': branch}) - # And run autohelp script to get a serialized dict of the discovered - # options - dirname = os.path.join('venv', branch.replace('/', '_')) - args = ["./autohelp-wrapper", "-q", "-b", branch, "-e", dirname, - "dump", project] - - path = os.environ.get("PATH") - bin_path = os.path.abspath(os.path.join(dirname, "bin")) - path = "%s:%s" % (bin_path, path) - serialized = subprocess.check_output(args, - env={'VIRTUAL_ENV': dirname, - 'PATH': path}) - ret = pickle.loads(serialized) - return ret - - -def _cmpopts(x, y): - """Compare to option names. - - The options can be of 2 forms: option_name or group/option_name. Options - without a group always comes first. Options are sorted alphabetically - inside a group. - """ - if '/' in x and '/' in y: - prex = x[:x.find('/')] - prey = y[:y.find('/')] - if prex != prey: - return cmp(prex, prey) - return cmp(x, y) - elif '/' in x: - return 1 - elif '/' in y: - return -1 - else: - return cmp(x, y) - - -def diff(old_list, new_list): - """Compare the old and new lists of options.""" - new_opts = [] - new_defaults = [] - deprecated_opts = [] - for name, (group, option) in new_list.items(): - # Find the new options - if name not in old_list.viewkeys(): - new_opts.append(name) - - # Find the options for which the default value has changed - elif option['default'] != old_list[name][1]['default']: - new_defaults.append(name) - - # Find options that have been deprecated in the new release. - # If an option name is a key in the old_list dict, it means that it - # wasn't deprecated. - - # Some options are deprecated, but not replaced with a new option. - # These options usually contain 'DEPRECATED' in their help string. - if 'DEPRECATED' in option['help']: - deprecated_opts.append((name, None)) - - for deprecated in option['deprecated_opts']: - # deprecated_opts is a list which always holds at least 1 invalid - # dict. Forget it. - if deprecated['name'] is None: - continue - - if deprecated['group'] in [None, 'DEFAULT']: - full_name = deprecated['name'] - else: - full_name = deprecated['group'] + '/' + deprecated['name'] - - if full_name in old_list.viewkeys(): - deprecated_opts.append((full_name, name)) - - return new_opts, new_defaults, deprecated_opts - - -def format_option_name(name): - """Return a formatted string for the option path.""" - if name is None: - return "None" - - try: - section, name = name.split('/') - except ValueError: - # name without a section ('log_dir') - return "[DEFAULT] %s" % name - - return "[%s] %s" % (section, name) - - -def release_from_branch(branch): - if branch == 'master': - return MASTER_RELEASE - else: - return branch.replace('stable/', '').title() - - -def get_env(project, new_branch, old_list, new_list): - """Generate the jinja2 environment for the defined branch and project.""" - new_opts, new_defaults, deprecated_opts = diff(old_list, new_list) - release = release_from_branch(new_branch) - - env = { - 'release': release, - 'project': project, - 'codename': CODENAME_TITLE[project], - 'new_opts': [], - 'new_defaults': [], - 'deprecated_opts': [] - } - - # New options - if new_opts: - for name in sorted(new_opts, _cmpopts): - opt = new_list[name][1] - name = format_option_name(name) - helptext = opt['help'].strip().replace('\n', ' ') - helptext = ' '.join(helptext.split()) - cells = (("%(name)s = %(default)s" % - {'name': name, - 'default': opt['default']}).strip(), - "(%(type)s) %(help)s" % {'type': opt['type'], - 'help': helptext}) - env['new_opts'].append(cells) - - # New defaults - if new_defaults: - for name in sorted(new_defaults, _cmpopts): - old_default = old_list[name][1]['default'] - new_default = new_list[name][1]['default'] - if isinstance(old_default, list): - old_default = ", ".join(old_default) - if isinstance(new_default, list): - new_default = ", ".join(new_default) - name = format_option_name(name) - cells = (name, old_default, new_default) - env['new_defaults'].append(cells) - - # Deprecated options - if deprecated_opts: - for deprecated, new in sorted(deprecated_opts, cmp=_cmpopts, - key=lambda tup: tup[0]): - deprecated = format_option_name(deprecated) - new = format_option_name(new) - env['deprecated_opts'].append((deprecated, new)) - - return env - - -def main(): - parser = argparse.ArgumentParser( - description='Generate a summary of configuration option changes.', - usage='%(prog)s [options] [projects]') - parser.add_argument('old_branch', - help='Name of the old branch.') - parser.add_argument('new_branch', - help='Name of the new branch.') - parser.add_argument('projects', - help='List of projects to work on.', - nargs='*', - default=PROJECTS) - parser.add_argument('-i', '--input', - dest='sources', - help='Path to a folder containing the git ' - 'repositories.', - required=False, - default='./sources', - type=str,) - parser.add_argument('-o', '--output', - dest='target', - help='Directory or file in which data will be saved.\n' - 'Defaults to "."', - required=False, - default='.', - type=str,) - parser.add_argument('-n', '--no-venv-update', - dest='novenvupdate', - help='Don\'t update the virtual envs.', - required=False, - action='store_true', - default=False,) - args = parser.parse_args() - - setup_venv(args.projects, args.old_branch, args.novenvupdate) - setup_venv(args.projects, args.new_branch, args.novenvupdate) - - for project in args.projects: - old_list = get_options(project, args.old_branch) - new_list = get_options(project, args.new_branch) - - release = args.new_branch.replace('stable/', '') - env = get_env(project, release, old_list, new_list) - filename = ("%(project)s-conf-changes.rst" % - {'project': project}) - tmpl_file = 'templates/changes.rst.j2' - if not os.path.exists(args.target): - os.makedirs(args.target) - dest = os.path.join(args.target, filename) - - with open(tmpl_file) as fd: - template = jinja2.Template(fd.read(), trim_blocks=True) - output = template.render(**env) - - with open(dest, 'w') as fd: - fd.write(output) - - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/autogenerate_config_docs/requirements.txt b/autogenerate_config_docs/requirements.txt deleted file mode 100644 index 39abf122..00000000 --- a/autogenerate_config_docs/requirements.txt +++ /dev/null @@ -1,5 +0,0 @@ -docutils -jinja2 -lxml -oslo.config -oslo.i18n diff --git a/autogenerate_config_docs/templates/autohelp-category.rst.j2 b/autogenerate_config_docs/templates/autohelp-category.rst.j2 deleted file mode 100644 index ee8859c4..00000000 --- a/autogenerate_config_docs/templates/autohelp-category.rst.j2 +++ /dev/null @@ -1,45 +0,0 @@ -.. - Warning: Do not edit this file. It is automatically generated from the - software project's code and your changes will be overwritten. - - The tool to generate this file lives in openstack-doc-tools repository. - - Please make any changes needed in the code, then run the - autogenerate-config-doc tool from the openstack-doc-tools repository, or - ask for help on the documentation mailing list, IRC channel or meeting. - -.. _{{ label }}: - -.. list-table:: Description of {{ nice_cat }} configuration options - :header-rows: 1 - :class: config-ref-table - - * - Configuration option = Default value - - Description -{% for group in groups %} - - * - **[{{ group }}]** - - -{% for item in items[loop.index0] %} - -{% if item['default'] is equalto '' %} - * - ``{{ item['name'] }}`` = -{% else %} - * - ``{{ item['name'] }}`` = ``{{ item['default'] }}`` -{% endif %} -{% for paragraph in item['help'] %} - -{% if loop.first %} - - ({{ item['type'] }}) {{ paragraph }} -{% else %} - {{ paragraph }} -{% endif %} -{% endfor %} -{% for flagname, flagdesc in item['flags'] %} - - - **{{ flagname }}** - - {{ flagdesc }} -{% endfor %} -{% endfor %} -{% endfor %} diff --git a/autogenerate_config_docs/templates/autohelp-group.rst.j2 b/autogenerate_config_docs/templates/autohelp-group.rst.j2 deleted file mode 100644 index ea1504bb..00000000 --- a/autogenerate_config_docs/templates/autohelp-group.rst.j2 +++ /dev/null @@ -1,40 +0,0 @@ -.. - Warning: Do not edit this file. It is automatically generated from the - software project's code and your changes will be overwritten. - - The tool to generate this file lives in openstack-doc-tools repository. - - Please make any changes needed in the code, then run the - autogenerate-config-doc tool from the openstack-doc-tools repository, or - ask for help on the documentation mailing list, IRC channel or meeting. - -.. _{{ label }}: - -.. list-table:: Description of {{ group }} configuration options - :header-rows: 1 - :class: config-ref-table - - * - Configuration option = Default value - - Description -{% for item in items %} - -{% if item['default'] is equalto '' %} - * - ``{{ item['name'] }}`` = -{% else %} - * - ``{{ item['name'] }}`` = ``{{ item['default'] }}`` -{% endif %} -{% for paragraph in item['help'] %} - -{% if loop.first %} - - ({{ item['type'] }}) {{ paragraph }} -{% else %} - {{ paragraph }} -{% endif %} -{% endfor %} -{% for flagname, flagdesc in item['flags'] %} - - - **{{ flagname }}** - - {{ flagdesc }} -{% endfor %} -{% endfor %} diff --git a/autogenerate_config_docs/templates/changes.rst.j2 b/autogenerate_config_docs/templates/changes.rst.j2 deleted file mode 100644 index 23b4f995..00000000 --- a/autogenerate_config_docs/templates/changes.rst.j2 +++ /dev/null @@ -1,61 +0,0 @@ -New, updated, and deprecated options in {{ release }} for {{ codename }} -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{{ '~' * release|length }}~~~~~{{ '~' * codename|length }} - -.. - Warning: Do not edit this file. It is automatically generated and your - changes will be overwritten. The tool to do so lives in the - openstack-doc-tools repository. - -{% if new_opts %} -.. list-table:: New options - :header-rows: 1 - :class: config-ref-table - - * - Option = default value - - (Type) Help string -{% for cells in new_opts %} - * - ``{{ cells[0] }}`` - - {{ cells[1] }} -{% endfor %} -{% endif %} - -{% if new_defaults %} -.. list-table:: New default values - :header-rows: 1 - :class: config-ref-table - - * - Option - - Previous default value - - New default value -{% for cells in new_defaults %} - * - ``{{ cells[0] }}`` -{% if cells[1] is equalto '' %} - - -{% else %} - - ``{{ cells[1] }}`` -{% endif %} -{% if cells[2] is equalto '' %} - - -{% else %} - - ``{{ cells[2] }}`` -{% endif %} -{% endfor %} -{% endif %} - -{% if deprecated_opts %} -.. list-table:: Deprecated options - :header-rows: 1 - :class: config-ref-table - - * - Deprecated option - - New Option -{% for cells in deprecated_opts %} - * - ``{{ cells[0] }}`` - - ``{{ cells[1] }}`` -{% endfor %} -{% endif %} - -{% if not new_opts and not new_defaults and not deprecated_opts %} -There are no new, updated, and deprecated options -in {{ release }} for {{ codename }}. -{% endif %} diff --git a/doc/source/autogenerate_config_docs.rst b/doc/source/autogenerate_config_docs.rst deleted file mode 100644 index 0bc5eb9b..00000000 --- a/doc/source/autogenerate_config_docs.rst +++ /dev/null @@ -1 +0,0 @@ -.. include:: ../../autogenerate_config_docs/README.rst diff --git a/doc/source/index.rst b/doc/source/index.rst index 748a0adb..557eb0bb 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -10,7 +10,6 @@ Contents: doc-tools-readme installation usage - autogenerate_config_docs man/openstack-doc-test sitemap-readme release_notes diff --git a/releasenotes/notes/the-great-doc-tools-cleanup-1a79e2c200232489.yaml b/releasenotes/notes/the-great-doc-tools-cleanup-1a79e2c200232489.yaml new file mode 100644 index 00000000..f5b33aec --- /dev/null +++ b/releasenotes/notes/the-great-doc-tools-cleanup-1a79e2c200232489.yaml @@ -0,0 +1,8 @@ +--- +upgrade: + - | + The `autogenerate_config_docs` set of tools has been removed. These were + always buggy and hard to maintain, and they have been superseded by the + ``oslo_config.sphinxext`` `Sphinx extension`__. + + __ https://docs.openstack.org/oslo.config/latest/reference/sphinxext.html diff --git a/setup.cfg b/setup.cfg index 4a65ccb0..86695c27 100644 --- a/setup.cfg +++ b/setup.cfg @@ -22,7 +22,6 @@ classifier = [files] packages = os_doc_tools - autogenerate_config_docs data_files = share/openstack-doc-tools/sitemap = sitemap/* share/openstack-doc-tools/cleanup = cleanup/* diff --git a/tox.ini b/tox.ini index 706b94c1..13168756 100644 --- a/tox.ini +++ b/tox.ini @@ -25,8 +25,7 @@ commands = doc8 -e txt -e rst doc/source/ HACKING.rst # Run bashate during pep8 runs to ensure violations are caught by # the check and gate queues. - bashate autogenerate_config_docs/autohelp-wrapper \ - bin/doc-tools-check-languages \ + bashate bin/doc-tools-check-languages \ cleanup/remove_trailing_whitespaces.sh [testenv:pylint] @@ -58,7 +57,7 @@ usedevelop = False [flake8] show-source = True builtins = _ -exclude=.venv,.git,.tox,dist,*lib/python*,*egg,build,*autogenerate_config_docs/venv,*autogenerate_config_docs/sources,doc/source/conf.py +exclude=.venv,.git,.tox,dist,*lib/python*,*egg,build,doc/source/conf.py # 28 is currently the most complex thing we have max-complexity=29 ignore = H101