1052 Commits

Author SHA1 Message Date
Zuul
4e0e72d480 Merge "Add support for looking in environment for config" 2018-11-02 19:58:06 +00:00
Zuul
76fd7231cb Merge "Document when sample_default is in use" 2018-11-02 16:24:50 +00:00
Chris Dent
ea8a0f6a8b Add support for looking in environment for config
An _environment source is added that looks in os.environ for
values.

Using the environment is on by default, but can be shut down
by setting `use_env` to False when __call__ is called.

The enviroment is inspected before any other sources
of config data but the value is used after command line
arguments and before config file options.

This is done by checking both the command line and config
files and then inspecting the location of the result. If
it is command_line, we use it. If not, we use the environment
value (if any). If there's no environment value, the config
file value is used. If checking the command line and config
file results in a KeyError, the environment value is used,
if set.

The names of the environment variables follow the rules
described in oslo_config.sources._environment.

A new exception has been added: ConfigSourceValueError, this
is the superclass of the existing ConfigFileValueError. The
code in _do_get has been updated to only use
ConfigFileValueError when it is in fact a file from whence a
ValueError came.

Documentation has been updated and a rlease note created to
indicate the new functionality.

Change-Id: I3245c40ebdcc96f8e3b2dc0bab3b4aa71d07ad15
2018-10-17 20:49:05 +01:00
Sean McGinnis
c09b80440a Update sphinx extension logging
Sphinx 1.6 deprecated using the application object to perform logging
and it will be removed in the upcoming 2.0 release. This updates our
extensions to use the recommended sphinx.util.logging instead.

Change-Id: Id7da5a90b804420ece0699e4986b382e6d8bc8f2
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2018-10-16 14:32:33 -05:00
ZhongShengping
2298afed75 Clean up .gitignore references to personal tools
Developers run all sorts of different tools within Git repositories,
any of which can leave their own special trashfiles all over the
place. We can't every hope to catalog them all, so better to
recommend developers simply configure a global core.excludesfile to
filter the irrelevant files which tend to get created by their
personal choice of tools.

To this end, remove the long-standing sections for "Mr Developer"
and "Editors" since their mere existence here sends the signal that
we welcome (and have time to review) additions for any old tool
someone ever might happen to try. Also add a comment block
explaining this, for clarity.

We can, and should of course, continue to list files created by the
tools recommended by our workflow (test frameworks called from tox,
documentation and packaging builds, et cetera).

This change is a port of I1b41efac219fca44e2548fc36633724d0ecfc0cb
from the openstack-dev/oslo-cookiecutter repository.

Change-Id: I5baba0f92e4272ba7e28f22ab6264651794ebf50
2018-10-15 11:56:20 +08:00
Zuul
d2ee0bf406 Merge "Unit tests to illustrate positional argument bug" 2018-10-12 17:54:08 +00:00
Zuul
83d2881b43 Merge "avoid trailing space in sphinxext log output" 2018-10-10 18:14:22 +00:00
Ilya Shakhat
aa195b8eb7 Close a previously opened file in config generator
Proper handling of file lifecycle to avoid file-handler leakage.

The patch should cope with complains from Tempest:
/home/zuul/.../oslo_config/generator.py:770: ResourceWarning:
unclosed file <_io.TextIOWrapper name='/.../etc/tempest.conf.sample' mode='w' encoding='UTF-8'>
  generate(conf)

Also this patch should address unstable Tempest test case
that relies on presence of config file on file system -
tempest.tests.cmd.test_verify_tempest_config.TestDiscovery.test_get_config_file()
(http://git.openstack.org/cgit/openstack/tempest/tree/tempest/tests/cmd/test_verify_tempest_config.py#n590)

Change-Id: I02a4c6a9a6422075365afe6e29a877f3f8dfad32
2018-10-09 16:12:47 +02:00
Doug Hellmann
4b4c3ca6bb avoid trailing space in sphinxext log output
Fix nit from parent patch.

Change-Id: I25b8ea98a91ec0b10787110584cb8cf293bcc404
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-10-09 11:56:04 +01:00
Zuul
ae7dbd939f Merge "sphinxext: Handle multi-line group descriptions" 2018-10-08 17:26:39 +00:00
Zuul
03e8202b90 Merge "sphinxext: Print output to logs, not to file" 2018-10-08 17:26:38 +00:00
Zuul
7f45e34c85 Merge "Always build universal wheels" 2018-10-08 12:02:42 +00:00
Zuul
8449a0444c Merge "Add release note and documentation for config validator" 2018-10-08 11:40:07 +00:00
Zuul
561da1eb96 Merge "Add support for machine-readable files to validator" 2018-10-08 11:40:06 +00:00
Zuul
f8e20de73e Merge "Simple config validator" 2018-10-08 11:33:04 +00:00
Dolph Mathews
b5f76a2ce8 Unit tests to illustrate positional argument bug
This patch does not provide a fix, but instead serves to illustrate
several use cases where positional arguments do not behave correctly on
the CLI.

Change-Id: Ibdb05066b95a285f6618c861eb4d38465dbf0d02
Related-Bug: 1676989
2018-10-08 11:17:49 +01:00
Zuul
7625e9531c Merge "generator: Pass conf to _OptFormatter" 2018-10-05 14:35:31 +00:00
Hervé Beraud
3ed560af13 Always build universal wheels
All of our projects are pure python, so we can build
universal wheels by default.

Change-Id: Ic5e5089a8ff62660392a92cfc9c0aa29d4d32e0a
2018-10-05 13:52:20 +02:00
Corey Bryant
7ad146c487 python3: Ensure ConfigOpts __iter__ uses list(d.keys())
d.keys() no longer returns a copy of the keys in Python 3.
This is part of PEP 3106.

Change-Id: Ie4897109c87d6dc2952c2eb35b83fe368fb31803
Closes-Bug: #1796163
2018-10-04 15:35:54 -04:00
Zuul
b12783ea0f Merge "move some documentation out of the source files" 2018-10-04 04:09:03 +00:00
Ben Nemec
8cc60344e6 Document when sample_default is in use
Because sample_default makes the generator silently ignore the
actual default value, it is possible for it to cause confusion for
users. This change adds a message to the generated opt help to let
them know that the documented default may not be the actual default.

Change-Id: Ib48d5dd1d18546b5464e9e096c47d1e3d67be57c
Closes-Bug: 1795729
2018-10-02 19:30:06 +00:00
Ben Nemec
1f17b823e0 Add release note and documentation for config validator
Change-Id: Id23d7bbb38c6780621f09b2e24c0cc88831bdb1a
2018-09-26 19:49:43 +00:00
Ben Nemec
e3cd230c06 Add support for machine-readable files to validator
In some cases it may be easier to generate a machine-readable sample
config file and use that to validate a config file instead of using
the config generator directly.

Change-Id: I7afe65c4abcbf8cdf1f5299ed3826c7205074e28
2018-09-26 19:49:43 +00:00
Ben Nemec
9362ccfa1a Simple config validator
This is a very basic config validator that just reads the provided
config file and compares it with the machine readable sample config
data from the config generator to find any options that are not
defined in the project. It will also warn about deprecated options
in use in the provided file.

Change-Id: I410b16ffcfb069140bd51dffaba7d56eada1a7e9
2018-09-26 19:49:14 +00:00
Chuck Short
7072a7552d Replace assertRaisesRegexp with assertRaisesRegex
This replaces the deprecated (in python 3.2) unittest.TestCase method
assertRaisesRegexp() with assertRaisesRegex().

Change-Id: Iba92ef7a76d9392750a95e3bd603f364de4ec890
Signed-off-by: Chuck Short <chucks@redhat.com>
6.5.1 6.5.0
2018-08-27 09:48:29 -04:00
Stephen Finucane
43c6ba77c9 sphinxext: Handle multi-line group descriptions
While multi-line help texts for opts were already handled properly, the
same was not true of groups. Close this gap, allowing us to resolve the
FIXME in the process.

Change-Id: Ic564dbff983205134386ba5b7047d6d811c60ef1
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2018-08-21 14:50:51 +01:00
Stephen Finucane
a31cce7fde sphinxext: Print output to logs, not to file
The functionality previously added here was broken as it did not include
a required newline. Rather than fix it, modify the whole thing to print
to logs instead of a temporary file. This should achieve the same effect
with less side effects.

Change-Id: I74d11a54d57d30aab7f97f4fcce9fee0b7d87cbc
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Closes-Bug: #1782316
2018-08-21 14:50:49 +01:00
Doug Hellmann
6d0aa06218 add lib-forward-testing-python3 test job
This is a mechanically generated patch to add a functional test job
running under Python 3 as part of the python3-first goal.

See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html

Change-Id: Icfe03bc959cf8a0ae4911fb76c79fe74d5ba3e57
Story: #2002586
Task: #24322
2018-08-11 18:20:12 -04:00
Doug Hellmann
fbb9611454 add python 3.6 unit test job
This is a mechanically generated patch to add a unit test job running
under Python 3.6 as part of the python3-first goal.

See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html

Change-Id: I3a73ebc7682704eec5ee9819417d538fa78e13de
Story: #2002586
Task: #24322
2018-08-07 16:17:32 -04:00
Doug Hellmann
65a4d5fd99 import zuul job settings from project-config
This is a mechanically generated patch to complete step 1 of moving
the zuul job settings out of project-config and into each project
repository.

Because there will be a separate patch on each branch, the branch
specifiers for branch-specific jobs have been removed.

See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html

Change-Id: I3cf9088f49df81516e2fe1301cdd56d12daaaae9
Story: #2003250
2018-08-03 17:43:57 -04:00
Stephen Finucane
b596dc2d12 sphinxext: Stop emitting lines multiple times
A typo resulted in us dumping the help text for opts as many times as
there were lines in the group help text. Resolve this.

Change-Id: I7a6506235834022fa76588e55ec60921b1ae673f
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Closes-Bug: #1782253
2018-08-03 11:27:50 -04:00
Zuul
8c80590e8f Merge "Write sphinxext debug tempfile with linebreaks" 2018-08-01 03:48:36 +00:00
7da0446edb Update reno for stable/rocky
Change-Id: I528e65230c065a1d03f856e252106d8648af5397
2018-07-26 12:37:28 +00:00
Zane Bitter
9bbbcabf38 Write sphinxext debug tempfile with linebreaks
It's much easier to debug the output if it isn't all on one line.

Change-Id: I446036aa043b74bb0f39deedd3f2cb9d08fd4dce
2018-07-23 10:38:57 -04:00
Doug Hellmann
bd463ee3b6 move some documentation out of the source files
The cfg module is very large, and starts with a huge block of
documentation. This patch moves that information into separate files
in the reference section of the docs. A few formatting fixes need to
be made to have it build cleanly, but the content is not changed in a
substantive way.

Change-Id: I86aa90bbf180b5dc9acbcedb024e5361d49954c3
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-07-19 17:26:38 -04:00
Raildo Mascena
ce150b1037 New cache layer for external sources
Add a separate cache for values coming from external sources,
this will protect us from having options from external sources
mutated when we reload the configuration files

Change-Id: Icf72f4e1745a0ddf53e661cc08d0fe7428cd9a41
Blueprint: oslo-config-drivers
2018-07-17 23:54:56 +00:00
Eric Fried
6c30a1c8a3 generator: Pass conf to _OptFormatter
In preparation for adding another conf option that would have needed to
be kwarg'd all the way down the call stack to the .format method, this
patch removes the existing `minimal` and `summarize` kwargs from the
methods in that call stack and instead makes them instance attributes of
generator._OptFormatter, which is now initialized with the whole
ConfigOpts object rather than those individual options.

This is just a refactor. There is no functional change (assuming nobody
is being naughty and consuming the private _OptFormatter class).

Change-Id: Ide0f1c6d79d506b979b54e6446786ea9ff51c8d5
2018-07-17 11:43:29 -05:00
Doug Hellmann
b79f763b49 ensure we do not modify private data from drivers
When we ask for a list of options from a drivere for generating the
sample config or documentation, we want to insert the actual 'driver'
option at the front of the list. This keeps each driver from having to
do that, and allows us to generate the sample with good
defaults. However, if a driver returns us a static data structure, we
do not want to modify *that* set of data, because the driver tests
will need the original structure intact. So, have list_opts() deepcopy
the data it is given, again to avoid having to ensure each driver
author does that copy.

This change also requires updating the test for list_opts() for the
URI driver to no longer assume the 'driver' option will be in the
results. At the same time I am reordering the arguments to
assertEqual() so that the expected value is listed first and renaming
the variable holding the actual values from the discovery call.

Change-Id: Ie8c1bc606f5f69a72b2383200a40b26e252067a9
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-07-17 12:35:13 -04:00
Zuul
45425e938d Merge "User guide documentation for backend drivers for oslo.config" 2018-07-16 23:31:43 +00:00
Zuul
cd5b37be18 Merge "add detail to driver options in config generator" 2018-07-16 15:07:57 +00:00
Zuul
6ddee6d29a Merge "Add example group for the URI driver" 2018-07-13 16:08:33 +00:00
Zuul
084ac31f4c Merge "Add config_source option" 2018-07-13 16:08:32 +00:00
Zuul
e95a4ce8c3 Merge "Create INI file ConfigurationSourceDriver." 2018-07-13 16:08:31 +00:00
Zuul
adba3da492 Merge "ConfigurationSource base class" 2018-07-13 16:08:30 +00:00
Zuul
f51fb759b3 Merge "Base class for a configuration driver" 2018-07-13 16:08:29 +00:00
Zuul
f534a5e2ec Merge "move configuration option list to the configuration guide" 2018-07-10 06:29:12 +00:00
Raildo Mascena
e8c93ea5ba User guide documentation for backend drivers for oslo.config
New documentation sections explaning how to use the backend
drivers for oslo.config, also include explanation on how to
use the remote_file driver.

Change-Id: I45fc2155f6fe2d8bee058dac880afba8bb9bfd53
Blueprint: oslo-config-drivers
2018-07-05 14:44:15 +00:00
Doug Hellmann
5ad89d4021 add detail to driver options in config generator
Have the main _list_opts caller construct the driver option so
individual drivers do not need to repeat that.

Add choices with descriptions when emitting samples. We don't really
care about those for the runtime use, but they improve the output in
the config generator and documentation.

Use an OptGroup with the driver_option and dynamic_group_owner options
set instead of just a group name when describing the options.

Add sample_default values for some of the options in the URI driver.

Change-Id: I14c0a046e6c70a9108308db70a4efb70613d5bb3
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-07-05 11:42:34 -03:00
Zuul
df0a57438f Merge "Optionally use oslo.log for deprecated opt logging" 2018-06-29 14:42:11 +00:00
Doug Hellmann
6a94cbc6a2 move configuration option list to the configuration guide
We have a section in the reference guide showing the configuration
options built into oslo.config. Move that to the configuration guide
instead so it is linked from the Configuration Guide list at
https://docs.openstack.org/$series/configuration/ like the similar
documentation for other libraries.

Change-Id: I7da159cfaa643c2cc094f6fb1c0aeaa956f3eb77
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-06-28 10:55:07 -04:00