openstack/pbr
Code Issues Proposed changes

Remove sphinx doc building integration

Browse Source 
Setuptools and sphinx [0] have removed support for building docs through
setup.py. This has broken imports which break our own local doc builds.
Lets just remove it all (including the tests) so that we are compatible
with modern setuptools and sphinx.

An alternative approach would be to do conditional imports and continue
to support this for old setuptools and sphinx. But that feels like a
dead end that will be difficult to test. Probably better to rip this
bandaid off.

[0] https://www.sphinx-doc.org/en/master/changes.html#id7

Change-Id: I65038caceb192f495288414079ca0f07ce6318bb
This commit is contained in:
Clark Boylan
2023-05-30 16:01:33 -07:00
parent 98c84b5f87
commit 47c5afe79a
7 changed files with 6 additions and 597 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
doc/source/user/features.rst
View File

@@ -292,23 +292,11 @@ Setup Commands
This feature has been superseded by the `sphinxcontrib-apidoc`__ (for
generation of API documentation) and :ref:`pbr.sphinxext` (for configuration
of versioning via package metadata) extensions. It will be removed in a
future release.
of versioning via package metadata) extensions. It has been removed in
version 6.0.
__ https://pypi.org/project/sphinxcontrib-apidoc/
Sphinx can produce auto documentation indexes based on signatures and
docstrings of your project but you have to give it index files to tell it to
*autodoc* each module: that's kind of repetitive and boring. *pbr* will scan
your project, find all of your modules, and generate all of the stub files for
you.
In addition, Sphinx documentation setups are altered to have several pieces of
information that are known to ``setup.py`` injected into the Sphinx config.
See the :ref:`pbr-setup-cfg` section of the configuration file for
details on configuring your project for *autodoc*.
``test``
~~~~~~~~

51
doc/source/user/using.rst
View File

@@ -117,7 +117,6 @@ However, *pbr* does provide a number of additional sections:
In addition, there are some modifications to other sections:
- ``metadata``
- ``build_sphinx``
For all other sections, you should refer to either the `setuptools`_
documentation or the documentation of the package that provides the section,
@@ -354,54 +353,8 @@ The ``pbr`` section controls *pbr*-specific options and behaviours.
This feature has been superseded by the `sphinxcontrib-apidoc`_ (for
generation of API documentation) and :ref:`pbr.sphinxext` (for configuration
of versioning via package metadata) extensions. It will be removed in a
future release.
The ``build_sphinx`` section is a version of the ``build_sphinx`` *setuptools*
plugin provided with Sphinx. This plugin extends the original plugin to add the
following:
- Automatic generation of module documentation using the ``sphinx-apidoc`` tool
- Automatic configuration of the ``project``, ``version`` and ``release``
settings using information from *pbr* itself
- Support for multiple builders using the ``builders`` configuration option
.. note::
Only applies to Sphinx < 1.6. See documentation on ``builders`` below.
The version of ``build_sphinx`` provided by *pbr* provides a single additional
option.
``builders``
A comma separated list of builders to run. For example, to build both HTML
and man page documentation, you would define the following in your
``setup.cfg``:
.. code-block:: ini
[build_sphinx]
builders = html,man
source-dir = doc/source
build-dir = doc/build
all-files = 1
warning-is-error = 1
.. deprecated:: 3.2.0
Sphinx 1.6+ adds support for specifying multiple builders in the default
``builder`` option. You should use this option instead. Refer to the
`Sphinx documentation`_ for more information.
For information on the remaining options, refer to the `Sphinx documentation`_.
In addition, the ``autodoc_index_modules``, ``autodoc_tree_index_modules``,
``autodoc_exclude_modules`` and ``autodoc_tree_excludes`` options :ref:`in the
pbr section <pbr-setup-cfg>` will affect the output of the automatic module
documentation generation.
.. _Sphinx documentation: http://www.sphinx-doc.org/en/stable/setuptools.html
of versioning via package metadata) extensions. It has been removed in
version 6.0.
Requirements
------------

292
pbr/builddoc.py
View File

@@ -1,292 +0,0 @@
# Copyright 2011 OpenStack Foundation
# Copyright 2012-2013 Hewlett-Packard Development Company, L.P.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from distutils import log
import fnmatch
import os
import sys
try:
import cStringIO
except ImportError:
import io as cStringIO
try:
import sphinx
# NOTE(dhellmann): Newer versions of Sphinx have moved the apidoc
# module into sphinx.ext and the API is slightly different (the
# function expects sys.argv[1:] instead of sys.argv[:]. So, figure
# out where we can import it from and set a flag so we can invoke
# it properly. See this change in sphinx for details:
# https://github.com/sphinx-doc/sphinx/commit/87630c8ae8bff8c0e23187676e6343d8903003a6
try:
from sphinx.ext import apidoc
apidoc_use_padding = False
except ImportError:
from sphinx import apidoc
apidoc_use_padding = True
from sphinx import application
from sphinx import setup_command
except Exception as e:
# NOTE(dhellmann): During the installation of docutils, setuptools
# tries to import pbr code to find the egg_info.writer hooks. That
# imports this module, which imports sphinx, which imports
# docutils, which is being installed. Because docutils uses 2to3
# to convert its code during installation under python 3, the
# import fails, but it fails with an error other than ImportError
# (today it's a NameError on StandardError, an exception base
# class). Convert the exception type here so it can be caught in
# packaging.py where we try to determine if we can import and use
# sphinx by importing this module. See bug #1403510 for details.
raise ImportError(str(e))
from pbr import git
from pbr import options
from pbr import version
_deprecated_options = ['autodoc_tree_index_modules', 'autodoc_index_modules',
'autodoc_tree_excludes', 'autodoc_exclude_modules']
_deprecated_envs = ['AUTODOC_TREE_INDEX_MODULES', 'AUTODOC_INDEX_MODULES']
_rst_template = """%(heading)s
%(underline)s
.. automodule:: %(module)s
:members:
:undoc-members:
:show-inheritance:
"""
def _find_modules(arg, dirname, files):
for filename in files:
if filename.endswith('.py') and filename != '__init__.py':
arg["%s.%s" % (dirname.replace('/', '.'),
filename[:-3])] = True
class LocalBuildDoc(setup_command.BuildDoc):
builders = ['html']
command_name = 'build_sphinx'
sphinx_initialized = False
def _get_source_dir(self):
option_dict = self.distribution.get_option_dict('build_sphinx')
pbr_option_dict = self.distribution.get_option_dict('pbr')
_, api_doc_dir = pbr_option_dict.get('api_doc_dir', (None, 'api'))
if 'source_dir' in option_dict:
source_dir = os.path.join(option_dict['source_dir'][1],
api_doc_dir)
else:
source_dir = 'doc/source/' + api_doc_dir
if not os.path.exists(source_dir):
os.makedirs(source_dir)
return source_dir
def generate_autoindex(self, excluded_modules=None):
log.info("[pbr] Autodocumenting from %s"
% os.path.abspath(os.curdir))
modules = {}
source_dir = self._get_source_dir()
for pkg in self.distribution.packages:
if '.' not in pkg:
for dirpath, dirnames, files in os.walk(pkg):
_find_modules(modules, dirpath, files)
def include(module):
return not any(fnmatch.fnmatch(module, pat)
for pat in excluded_modules)
module_list = sorted(mod for mod in modules.keys() if include(mod))
autoindex_filename = os.path.join(source_dir, 'autoindex.rst')
with open(autoindex_filename, 'w') as autoindex:
autoindex.write(""".. toctree::
:maxdepth: 1
""")
for module in module_list:
output_filename = os.path.join(source_dir,
"%s.rst" % module)
heading = "The :mod:`%s` Module" % module
underline = "=" * len(heading)
values = dict(module=module, heading=heading,
underline=underline)
log.info("[pbr] Generating %s"
% output_filename)
with open(output_filename, 'w') as output_file:
output_file.write(_rst_template % values)
autoindex.write(" %s.rst\n" % module)
def _sphinx_tree(self):
source_dir = self._get_source_dir()
cmd = ['-H', 'Modules', '-o', source_dir, '.']
if apidoc_use_padding:
cmd.insert(0, 'apidoc')
apidoc.main(cmd + self.autodoc_tree_excludes)
def _sphinx_run(self):
if not self.verbose:
status_stream = cStringIO.StringIO()
else:
status_stream = sys.stdout
confoverrides = {}
if self.project:
confoverrides['project'] = self.project
if self.version:
confoverrides['version'] = self.version
if self.release:
confoverrides['release'] = self.release
if self.today:
confoverrides['today'] = self.today
if self.sphinx_initialized:
confoverrides['suppress_warnings'] = [
'app.add_directive', 'app.add_role',
'app.add_generic_role', 'app.add_node',
'image.nonlocal_uri',
]
app = application.Sphinx(
self.source_dir, self.config_dir,
self.builder_target_dir, self.doctree_dir,
self.builder, confoverrides, status_stream,
freshenv=self.fresh_env, warningiserror=self.warning_is_error)
self.sphinx_initialized = True
try:
app.build(force_all=self.all_files)
except Exception as err:
from docutils import utils
if isinstance(err, utils.SystemMessage):
sys.stder.write('reST markup error:\n')
sys.stderr.write(err.args[0].encode('ascii',
'backslashreplace'))
sys.stderr.write('\n')
else:
raise
if self.link_index:
src = app.config.master_doc + app.builder.out_suffix
dst = app.builder.get_outfilename('index')
os.symlink(src, dst)
def run(self):
option_dict = self.distribution.get_option_dict('pbr')
# TODO(stephenfin): Remove this (and the entire file) when 5.0 is
# released
warn_opts = set(option_dict.keys()).intersection(_deprecated_options)
warn_env = list(filter(lambda x: os.getenv(x), _deprecated_envs))
if warn_opts or warn_env:
msg = ('The autodoc and autodoc_tree features are deprecated in '
'4.2 and will be removed in a future release. You should '
'use the sphinxcontrib-apidoc Sphinx extension instead. '
'Refer to the pbr documentation for more information.')
if warn_opts:
msg += ' Deprecated options: %s' % list(warn_opts)
if warn_env:
msg += ' Deprecated environment variables: %s' % warn_env
log.warn(msg)
if git._git_is_installed():
git.write_git_changelog(option_dict=option_dict)
git.generate_authors(option_dict=option_dict)
tree_index = options.get_boolean_option(option_dict,
'autodoc_tree_index_modules',
'AUTODOC_TREE_INDEX_MODULES')
auto_index = options.get_boolean_option(option_dict,
'autodoc_index_modules',
'AUTODOC_INDEX_MODULES')
if not os.getenv('SPHINX_DEBUG'):
# NOTE(afazekas): These options can be used together,
# but they do a very similar thing in a different way
if tree_index:
self._sphinx_tree()
if auto_index:
self.generate_autoindex(
set(option_dict.get(
"autodoc_exclude_modules",
[None, ""])[1].split()))
self.finalize_options()
is_multibuilder_sphinx = version.SemanticVersion.from_pip_string(
sphinx.__version__) >= version.SemanticVersion(1, 6)
# TODO(stephenfin): Remove support for Sphinx < 1.6 in 4.0
if not is_multibuilder_sphinx:
log.warn('[pbr] Support for Sphinx < 1.6 will be dropped in '
'pbr 4.0. Upgrade to Sphinx 1.6+')
# TODO(stephenfin): Remove this at the next MAJOR version bump
if self.builders != ['html']:
log.warn("[pbr] Sphinx 1.6 added native support for "
"specifying multiple builders in the "
"'[sphinx_build] builder' configuration option, "
"found in 'setup.cfg'. As a result, the "
"'[sphinx_build] builders' option has been "
"deprecated and will be removed in pbr 4.0. Migrate "
"to the 'builder' configuration option.")
if is_multibuilder_sphinx:
self.builder = self.builders
if is_multibuilder_sphinx:
# Sphinx >= 1.6
return setup_command.BuildDoc.run(self)
# Sphinx < 1.6
for builder in self.builders:
self.builder = builder
self.finalize_options()
self._sphinx_run()
def initialize_options(self):
# Not a new style class, super keyword does not work.
setup_command.BuildDoc.initialize_options(self)
# NOTE(dstanek): exclude setup.py from the autodoc tree index
# builds because all projects will have an issue with it
self.autodoc_tree_excludes = ['setup.py']
def finalize_options(self):
from pbr import util
# Not a new style class, super keyword does not work.
setup_command.BuildDoc.finalize_options(self)
# Handle builder option from command line - override cfg
option_dict = self.distribution.get_option_dict('build_sphinx')
if 'command line' in option_dict.get('builder', [[]])[0]:
self.builders = option_dict['builder'][1]
# Allow builders to be configurable - as a comma separated list.
if not isinstance(self.builders, list) and self.builders:
self.builders = self.builders.split(',')
self.project = self.distribution.get_name()
self.version = self.distribution.get_version()
self.release = self.distribution.get_version()
# NOTE(dstanek): check for autodoc tree exclusion overrides
# in the setup.cfg
opt = 'autodoc_tree_excludes'
option_dict = self.distribution.get_option_dict('pbr')
if opt in option_dict:
self.autodoc_tree_excludes = util.split_multiline(
option_dict[opt][1])
# handle Sphinx < 1.5.0
if not hasattr(self, 'warning_is_error'):
self.warning_is_error = False

3
pbr/hooks/commands.py
View File

@@ -47,9 +47,6 @@ class CommandsConfig(base.BaseConfig):
if os.name != 'nt':
easy_install.get_script_args = packaging.override_get_script_args
if packaging.have_sphinx():
self.add_command('pbr.builddoc.LocalBuildDoc')
if os.path.exists('.testr.conf') and packaging.have_testr():
# There is a .testr.conf file. We want to use it.
self.add_command('pbr.packaging.TestrTest')

12
pbr/packaging.py
View File

@@ -659,19 +659,11 @@ class LocalSDist(sdist.sdist):
sdist.sdist.make_distribution(self)
try:
from pbr import builddoc
_have_sphinx = True
# Import the symbols from their new home so the package API stays
# compatible.
LocalBuildDoc = builddoc.LocalBuildDoc
except ImportError:
_have_sphinx = False
LocalBuildDoc = None
LocalBuildDoc = None
def have_sphinx():
return _have_sphinx
return False
def _get_increment_kwargs(git_dir, tag):

4
pbr/tests/test_core.py
View File

@@ -77,10 +77,6 @@ class TestCore(base.BaseTestCase):
stdout, _, _ = self.run_setup('--keywords')
assert stdout == 'packaging, distutils, setuptools'
def test_setup_py_build_sphinx(self):
stdout, _, return_code = self.run_setup('build_sphinx')
self.assertEqual(0, return_code)
def test_sdist_extra_files(self):
"""Test that the extra files are correctly added."""

225
pbr/tests/test_setup.py
View File

@@ -29,7 +29,6 @@ import fixtures
from pbr import git
from pbr import options
from pbr import packaging
from pbr.tests import base
@@ -221,227 +220,3 @@ class GitLogsTest(base.BaseTestCase):
self.assertIn(author_old, authors)
self.assertIn(author_new, authors)
self.assertIn(co_author, authors)
class _SphinxConfig(object):
man_pages = ['foo']
class BaseSphinxTest(base.BaseTestCase):
def setUp(self):
super(BaseSphinxTest, self).setUp()
# setup_command requires the Sphinx instance to have some
# attributes that aren't set normally with the way we use the
# class (because we replace the constructor). Add default
# values directly to the class definition.
import sphinx.application
sphinx.application.Sphinx.messagelog = []
sphinx.application.Sphinx.statuscode = 0
self.useFixture(fixtures.MonkeyPatch(
"sphinx.application.Sphinx.__init__", lambda *a, **kw: None))
self.useFixture(fixtures.MonkeyPatch(
"sphinx.application.Sphinx.build", lambda *a, **kw: None))
self.useFixture(fixtures.MonkeyPatch(
"sphinx.application.Sphinx.config", _SphinxConfig))
self.useFixture(fixtures.MonkeyPatch(
"sphinx.config.Config.init_values", lambda *a: None))
self.useFixture(fixtures.MonkeyPatch(
"sphinx.config.Config.__init__", lambda *a: None))
from distutils import dist
self.distr = dist.Distribution()
self.distr.packages = ("fake_package",)
self.distr.command_options["build_sphinx"] = {
"source_dir": ["a", "."]}
pkg_fixture = fixtures.PythonPackage(
"fake_package", [("fake_module.py", b""),
("another_fake_module_for_testing.py", b""),
("fake_private_module.py", b"")])
self.useFixture(pkg_fixture)
self.useFixture(base.DiveDir(pkg_fixture.base))
self.distr.command_options["pbr"] = {}
if hasattr(self, "excludes"):
self.distr.command_options["pbr"]["autodoc_exclude_modules"] = (
'setup.cfg',
"fake_package.fake_private_module\n"
"fake_package.another_fake_*\n"
"fake_package.unknown_module")
if hasattr(self, 'has_opt') and self.has_opt:
options = self.distr.command_options["pbr"]
options["autodoc_index_modules"] = ('setup.cfg', self.autodoc)
class BuildSphinxTest(BaseSphinxTest):
scenarios = [
('true_autodoc_caps',
dict(has_opt=True, autodoc='True', has_autodoc=True)),
('true_autodoc_caps_with_excludes',
dict(has_opt=True, autodoc='True', has_autodoc=True,
excludes="fake_package.fake_private_module\n"
"fake_package.another_fake_*\n"
"fake_package.unknown_module")),
('true_autodoc_lower',
dict(has_opt=True, autodoc='true', has_autodoc=True)),
('false_autodoc',
dict(has_opt=True, autodoc='False', has_autodoc=False)),
('no_autodoc',
dict(has_opt=False, autodoc='False', has_autodoc=False)),
]
def test_build_doc(self):
build_doc = packaging.LocalBuildDoc(self.distr)
build_doc.run()
self.assertTrue(
os.path.exists("api/autoindex.rst") == self.has_autodoc)
self.assertTrue(
os.path.exists(
"api/fake_package.fake_module.rst") == self.has_autodoc)
if not self.has_autodoc or hasattr(self, "excludes"):
assertion = self.assertFalse
else:
assertion = self.assertTrue
assertion(
os.path.exists(
"api/fake_package.fake_private_module.rst"))
assertion(
os.path.exists(
"api/fake_package.another_fake_module_for_testing.rst"))
def test_builders_config(self):
build_doc = packaging.LocalBuildDoc(self.distr)
build_doc.finalize_options()
self.assertEqual(1, len(build_doc.builders))
self.assertIn('html', build_doc.builders)
build_doc = packaging.LocalBuildDoc(self.distr)
build_doc.builders = ''
build_doc.finalize_options()
self.assertEqual('', build_doc.builders)
build_doc = packaging.LocalBuildDoc(self.distr)
build_doc.builders = 'man'
build_doc.finalize_options()
self.assertEqual(1, len(build_doc.builders))
self.assertIn('man', build_doc.builders)
build_doc = packaging.LocalBuildDoc(self.distr)
build_doc.builders = 'html,man,doctest'
build_doc.finalize_options()
self.assertIn('html', build_doc.builders)
self.assertIn('man', build_doc.builders)
self.assertIn('doctest', build_doc.builders)
def test_cmd_builder_override(self):
if self.has_opt:
self.distr.command_options["pbr"] = {
"autodoc_index_modules": ('setup.cfg', self.autodoc)
}
self.distr.command_options["build_sphinx"]["builder"] = (
"command line", "non-existing-builder")
build_doc = packaging.LocalBuildDoc(self.distr)
self.assertNotIn('non-existing-builder', build_doc.builders)
self.assertIn('html', build_doc.builders)
# process command line options which should override config
build_doc.finalize_options()
self.assertIn('non-existing-builder', build_doc.builders)
self.assertNotIn('html', build_doc.builders)
def test_cmd_builder_override_multiple_builders(self):
if self.has_opt:
self.distr.command_options["pbr"] = {
"autodoc_index_modules": ('setup.cfg', self.autodoc)
}
self.distr.command_options["build_sphinx"]["builder"] = (
"command line", "builder1,builder2")
build_doc = packaging.LocalBuildDoc(self.distr)
build_doc.finalize_options()
self.assertEqual(["builder1", "builder2"], build_doc.builders)
class APIAutoDocTest(base.BaseTestCase):
def setUp(self):
super(APIAutoDocTest, self).setUp()
# setup_command requires the Sphinx instance to have some
# attributes that aren't set normally with the way we use the
# class (because we replace the constructor). Add default
# values directly to the class definition.
import sphinx.application
sphinx.application.Sphinx.messagelog = []
sphinx.application.Sphinx.statuscode = 0
self.useFixture(fixtures.MonkeyPatch(
"sphinx.application.Sphinx.__init__", lambda *a, **kw: None))
self.useFixture(fixtures.MonkeyPatch(
"sphinx.application.Sphinx.build", lambda *a, **kw: None))
self.useFixture(fixtures.MonkeyPatch(
"sphinx.application.Sphinx.config", _SphinxConfig))
self.useFixture(fixtures.MonkeyPatch(
"sphinx.config.Config.init_values", lambda *a: None))
self.useFixture(fixtures.MonkeyPatch(
"sphinx.config.Config.__init__", lambda *a: None))
from distutils import dist
self.distr = dist.Distribution()
self.distr.packages = ("fake_package",)
self.distr.command_options["build_sphinx"] = {
"source_dir": ["a", "."]}
self.sphinx_options = self.distr.command_options["build_sphinx"]
pkg_fixture = fixtures.PythonPackage(
"fake_package", [("fake_module.py", b""),
("another_fake_module_for_testing.py", b""),
("fake_private_module.py", b"")])
self.useFixture(pkg_fixture)
self.useFixture(base.DiveDir(pkg_fixture.base))
self.pbr_options = self.distr.command_options.setdefault('pbr', {})
self.pbr_options["autodoc_index_modules"] = ('setup.cfg', 'True')
def test_default_api_build_dir(self):
build_doc = packaging.LocalBuildDoc(self.distr)
build_doc.run()
print('PBR OPTIONS:', self.pbr_options)
print('DISTR OPTIONS:', self.distr.command_options)
self.assertTrue(os.path.exists("api/autoindex.rst"))
self.assertTrue(os.path.exists("api/fake_package.fake_module.rst"))
self.assertTrue(
os.path.exists(
"api/fake_package.fake_private_module.rst"))
self.assertTrue(
os.path.exists(
"api/fake_package.another_fake_module_for_testing.rst"))
def test_different_api_build_dir(self):
# Options have to come out of the settings dict as a tuple
# showing the source and the value.
self.pbr_options['api_doc_dir'] = (None, 'contributor/api')
build_doc = packaging.LocalBuildDoc(self.distr)
build_doc.run()
print('PBR OPTIONS:', self.pbr_options)
print('DISTR OPTIONS:', self.distr.command_options)
self.assertTrue(os.path.exists("contributor/api/autoindex.rst"))
self.assertTrue(
os.path.exists("contributor/api/fake_package.fake_module.rst"))
self.assertTrue(
os.path.exists(
"contributor/api/fake_package.fake_private_module.rst"))