58 Commits

Author SHA1 Message Date
Jenkins
9abc7dc283 Merge "[autohelp] Move ignored modules list to a file" 2015-11-16 05:44:28 +00:00
Gauvain Pocentek
52537d2341 autohelp.py: switch to jinja templates
Change-Id: Ice55c665adf4c0ede819cfc7264ed13feeca14db
2015-11-15 19:40:17 +01:00
Gauvain Pocentek
1d309a0a47 [autohelp] Move ignored modules list to a file
Change-Id: I80a7ed680149e40cbbf84f22fb5735a00666c872
2015-11-15 17:50:55 +01:00
Gauvain Pocentek
a699f8d764 autohelp: properly sanitize paths
Change-Id: Id2449820c5c10eb40be1f688504ec6bf13cc42a2
2015-11-09 09:55:38 +01:00
Atsushi SAKAI
3dbacea2fb Update config reference package list for neutron
This is just for update package list changes.
 Not solved config reference autogeneration problem.

 Remove stackforge package handling in autohelp-wrapper
 Remove modules
   networking-ibm (SDNVE) (It seems no maintained)
   networking-bigswitch (It seems no maintained)
   networking-portforwarding (It seems no maintained)

 Changes from stackforge to openstack
   networking-bagpipe-l2, networking-brocade,
   networking-edge-vpn, networking-hyperv,
   networking-mlnx, networking-nec, networking-ovs-dpdk,
   networking-zvm, networking-6wind

 Add
   octavia

Change-Id: I5ba3b26e110f55b2cc76f37b903a1b07dd6a1202
Related-Bug: #1503967
2015-10-19 16:23:38 +09:00
Tom Fifield
cd74750d4b Increase visibility of the warning in autodoc files
Autogenerate-config-docs creates files that live in the
openstack-manuals repository, but are automatically generated
and should not be edited.

On occasion, contributors miss the warning not to edit the files.

This patch increases the previous warning so that it should take up
enough screen real-estate in a text editor that it would be less
easily skipped, and adds some additional information on what to do
instead.

Change-Id: I800e1ed19427e00f903046731fe8f22f22e4c6db
2015-10-08 16:43:16 +08:00
Atsushi SAKAI
b2f64d0012 Add supported modules for config ref generator
With this patch, it can run on
  designate
  manila
  zaqar

Related-Bug: #1407581

How to add
  see the Trace back and add last item. like follows
    For addin designate.api.v1.extensions.sync case,

====log===
Traceback (most recent call last):

snip

  File "/home/sakaia/work/openstack-doc-tools/autogenerate_config_docs/autohelp.py", line 138, in import_modules
    module = importlib.import_module(modname)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/sakaia/work/openstack-doc-tools/autogenerate_config_docs/sources/designate/designate/api/v1/extensions/sync.py", line 23, in <module>
    central_api = central_rpcapi.CentralAPI()
=====

Change-Id: I76101cba32c4dd09a303a878719db5f544eb3ebf
2015-09-16 16:16:30 +09:00
Gauvain Pocentek
04e4d4bd35 autohelp: add an exception for the cinder wsgi script
This script doesn't test the __main__ variable so the import tries to
run cinder and fails.

Change-Id: Ie61946235bb51e770941b433b56c537382f494d2
2015-09-07 08:54:18 +02:00
Gauvain Pocentek
11e46da154 autohelp: add oslo.service to the list of extensions
Change-Id: Id523dd72c5987f86d22bd5905ed1511b58ae25cf
Closes-Bug: #1487171
2015-08-22 15:12:45 +02:00
Jenkins
0b67e01e79 Merge "autohelp: support reading options from multiple repos" 2015-07-27 03:39:08 +00:00
Gauvain Pocentek
da4d785171 autohelp: add oslo.cache as extension
Change-Id: Ifac03228dfe43719fdc89cfffa2ac53865aa06ee
2015-07-24 09:58:39 +02:00
Gauvain Pocentek
2e63aba8ac autohelp: support reading options from multiple repos
This change modifies the -i option behavior of autohelp.py to add the
possibility to find configuration options in multiple repositories. The
change implies a few related modifications:

- do not check if the source repository is a git repository
- update autohelp-wrapper to clone and use the additional repositories
  (only for neutron at the moment)

Note: diff_branches.py will not work with this change, it'll be fixed in
another patch.

Closes-Bug: #1448107
Change-Id: I4999aca4f7926e19513acdf38f1e2828b52cee13
2015-07-23 16:49:43 +02:00
Gauvain Pocentek
a98a112832 autohelp.py: remove the oslo.i18n hack
All the projects use oslo.i18n in kilo and liberty, the workaround for
projects non using this module is not needed anymore.

Change-Id: I7a096c11eef2ae2f3e9006fa0379f49f18a1cc81
2015-07-23 11:02:08 +02:00
Gauvain Pocentek
b73375f1fe autohelp: fix a typo
This adds back the missing keystone database settings, and most likely
some other settings.

Change-Id: Idd0c327a6e8a95bc800a9abd695ef7a277d52f2f
2015-07-14 22:19:39 +02:00
Ruby Loo
6329eea32a autohelp: use correct option names from extensions
autohelp's OptionsCache.load_extension_options() loads options from
libraries. An oslo_config.cfg.Opt object has a 'name' attribute and
a 'dest' attribute. The option name that we're interested in (that
the user sets) is the 'dest' value, not the 'name' value. The
load_extension_options() method was incorrectly using the option's
'name' attribute.

This fixes it so that the option's 'dest' attribute is used instead.
Eg, from oslo_log library, it now correctly uses 'log_config_append'
instead of 'log-config-append'.

Change-Id: I01422bf33be286cc0176a3e1e4ac1c348c6821ef
Closes-Bug: #1464058
2015-06-11 02:00:51 +00:00
Gauvain Pocentek
b455d12bfb autohelp: generate rst tables
We are mmoving the guides to RST, so we need config-reference tables in
this format. This patch adds a `rst` subcommand to autohelp.py to handle
this new requirement.

RST has limited tables-related features, and openstackdocstheme will
need an update to improve the output.

Change-Id: I5e58d60e2308d208e850f903559da5f5a6a949e3
2015-05-22 09:44:29 -07:00
Gauvain Pocentek
9aa45dda9a autohelp: remove rootwrap discovery code
All the projects are using oslo_rootwrap in kilo, and don't provide
rootwrap options anymore.

Change-Id: I1a0bd98862b0504a723c7aef14f8a3f0c0945ed5
2015-05-21 07:16:19 -07:00
Gauvain Pocentek
e4c8992edb autohelp: load oslo.middleware options
Partial-Bug: #1452454
Change-Id: I38289912445117ed4d2bbee2d6a886fd88fac4c8
2015-05-07 06:00:40 +00:00
Gauvain Pocentek
feba2086be Autohelp: update the scripts for the juno/kilo diff
* include sahara
* remove the keystoneclient workaround
* ignore a problematic neutron import
* handle "None" values

Change-Id: I8f342dcb39f5ddd69b73baec2ecb1f3c6883c01a
2015-04-29 15:08:56 +00:00
Doug Hellmann
d797ef81cd Drop use of 'oslo' namespace package
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.

The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.

Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.

Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages

Change-Id: Ib17a74ada3122ce45c52fdd7dcf8d27fafbb1d70
2015-04-28 22:18:23 +00:00
Gauvain Pocentek
71b8d2175f autohelp: lookup options in more oslo libraries
Change-Id: I24942de43dbe04256f511ee372f203942f555cce
2015-04-24 12:11:58 +02:00
Gauvain Pocentek
044850f8cc autohelp: strip help strings on dump too
Change-Id: I639ffd4352b8e9b09debaca01fa2befd79ecf466
2015-04-02 15:13:42 +02:00
Gauvain Pocentek
05d96ac900 autohelp: update the ignored modules list
This includes 2 glance contrib modules, and the experimental pgsql trove
modules.

Change-Id: If18946b1cda5c5742f554b0635789415a50c02b8
2015-04-02 13:05:09 +02:00
Gauvain Pocentek
223286cadc Rework the autohelp serialization of options
Using oslo.config objects in pickle leads to multiple import problems on
deserialization. Since we don't actually need oslo objects, we convert
them into dicts before dumping them.

Change-Id: Icbd1a3957e4d4ae05fe6227fbdd1cb58e5a3a98d
2015-03-03 17:13:18 +01:00
Gauvain Pocentek
3d2f91cf8e autohelp: don't ignore cmd/ folders
These folders used to bring command line options that we don't want to
have in the generated tables, but the behavior has changed. This patch
restores the import of files in the cmd/ folders, and handles special
cases where the import is problematic.

Change-Id: I0779e0f689d2f6f64149c3d111a8dd1ecd6cfc20
Partial-Bug: #1363954
2015-02-19 21:06:24 +01:00
Gauvain Pocentek
7a1046484b autohelp: don't specify encoding in the base xml
Specifying the encoding and working with unicode strings in not
supported in lxml. The BASE_XML string is only used to init the XML tree
and does not need a specific encoding. The utf-8 encoding is explicitely
added when the final XML is written to file.

Change-Id: I74e0a316cf718ad9768a690f8ccc4a1cc9a9fc7e
2015-01-31 12:26:14 +01:00
Gauvain Pocentek
1cfa1fb2be Handle possible keystone duplicated opts
We use trickery to import the keystone middleware options, which might
result in duplicated opts errors. This patch ignore such errors on the
middleware import.

Also use keystonemiddleware instead of keystoneclient.middleware.

Change-Id: I4e003ec9f4a7a706a2bbdeea5f874624fc59ea40
2015-01-21 18:39:28 +01:00
Tom Fifield
3363d2e1cb Fix xml tagging for non-swift config tables
A previous patch attempted to introduce semantic tagging for
options and their default values in the automatically
generated configuration reference tables.

While this worked well for swift, incorrect code was used
for the XML generation of the main autohelp.py method.

This patch fixes the string based code to use XML subElements
as was already done for extract_swift_flags.

Change-Id: I93a89b726d66acd0778e8365f47d5ee2f2e0b3c5
2015-01-01 19:14:43 +08:00
Tom Fifield
a2970b1d39 Add <option> tags to options in autogenerateconfig
By marking options as <option>option_name</option>, we get
more flexibility in how to handle them.

In particular, currently these thousands of automatically generated
options are polluting the translation's common resource.
Marking these strings as <option> might allow us to
say "don't translate this".

Change-Id: I0282545d1a2c3ca3d14a5c411c2f5457fbd09780
2014-12-31 16:17:20 +08:00
Gauvain Pocentek
5442f593b5 Autohelp: sanitize the value during discovery
Most of the default value sanitization happened during the docbook
writting, and not during the 'dump' of the options. This resulted in
invalid values in the xml files produced by diff_branches.py.

This patch moves all the sanitization code to the same function,
ensuring that the sanitization happens during options discovery.

Change-Id: I4d7e5d601baac5294937525ede8655da6b25e6cf
2014-10-26 20:34:16 +01:00
Gauvain Pocentek
862b243b9f autohelp: Remove trailing spaces from help strings
Some help strings have a trailing space that make our tests fail. These
spaces should be removed from the help strings in the projects, but it
takes time and we often end up editing manually.

Help strings from the oslo incubator might already have been fixed but
not yet imported in the projects, so no proper solution exists in this
case.

Change-Id: I606f46b23a4c5a5114bb5577273657fdf4fa5295
2014-10-03 08:40:03 +02:00
Gauvain Pocentek
b4cb74cb07 autohelp: add a module blacklisting mechanism
Change-Id: I932bc5ad740c9eaee3d1db05462ccb2c2a2779a7
2014-09-06 08:44:39 +02:00
Andreas Jaeger
cbfe5d922b Allow comments in flag and headers
Allow comments and empty lines in flag and header files.

Change-Id: I127ec83ffc926d0b80dcc725663afcca3aa79c5d
2014-08-28 14:14:49 +02:00
Andreas Jaeger
807ac96bc9 Parse shared.headers for autohelp
To avoid duplication of entries, use a shared headers file as fall back.

Co-Authored-By: Gauvain Pocentek <gauvain.pocentek@objectif-libre.com>

Change-Id: I639a299d5b1d1e91b81ab15fc6ba2a93f35017a9
2014-08-27 11:50:18 +02:00
Andreas Jaeger
13019d8274 autohelp: Set table names
Use PROJECT.headers file to map categories to table names.

Change-Id: Iec437160c40f0c51f731e40e043bfbd163c0c83c
Closes-Bug: #1361026
2014-08-26 06:32:20 +02:00
Gauvain Pocentek
d0e1a97df9 Improve the autohelp scripts
wrapper:
- Don't modify the openstack-manuals repo if it exists, keep it as is
  git wise (no branch change, no pull)
- Install oslo.i18n and hplefthandclient
- First setup the environment, then do the update/docbook work

autohelp.py:
- Handle projects not using oslo.i18n (mostly for havana)

diff_branches.py:
- Update the output format
- Setup the venv for the requested projects only

Change-Id: I48c45a56f4b7b7298fef0b7595d62e445c92df39
2014-08-23 09:11:17 +02:00
Gauvain Pocentek
cccee65b46 Update sanitizer to handle '/sources' in pybasedir
Change-Id: Icafe29f1b64c2ce406f73fa7ebc93f97592c5313
2014-08-19 13:56:25 +02:00
Jenkins
52c86659af Merge "autohelp: make sure that oslo.i18n is setup" 2014-07-29 08:10:33 +00:00
Christian Berendt
760b421951 Replace usage of readlines()
Change-Id: Ic4c0af29f900b274b1a4b45d2e55f0012107d47c
2014-07-29 09:51:17 +02:00
Gauvain Pocentek
07fd34f29d autohelp: make sure that oslo.i18n is setup
Some module imports fail without a proper setup of oslo.i18n, so make
sure that it is the case for projects using this package.

Change-Id: I7082cb893ccb52fa7bf065a01655efb9a5bbd4e2
2014-07-29 09:33:19 +02:00
Gauvain Pocentek
88db667694 sanitizer: handle values ending with spaces
This patch also reworks the autohelp _sanitize_default to wotk on an Opt
object instead of its name and default value.

Change-Id: I2b1b0dd44f64a49b6e41c9922555cd968822d144
2014-07-23 07:38:56 +02:00
Jenkins
930256f72a Merge "Don't use oslo's _sanitize_default" 2014-07-16 18:58:23 +00:00
Gauvain Pocentek
3854794b3a Don't use oslo's _sanitize_default
This method disappeared from oslo, so provide our own sanitizer. This is
needed for now but can be removed as soon as projects provide
Opts.sample_default values.

Change-Id: I13e76c75a74667fa9bd3c779a6cf45d0281ed680
Closes-Bug: #1341250
2014-07-13 22:51:34 +02:00
Andreas Jaeger
ef07792647 Update XML root elements
Write our root elements in a consistent way.

Change-Id: I3f74cc7fd0346c44b2cf445bb4f8638af4c70e4b
2014-07-10 08:39:55 +02:00
Gauvain Pocentek
05fcf72428 Use lxml to handle the XML data
Use lxml to read/write the docbook data. This avoids dealing with
manually indented string.

Change-Id: Ia4e30105f5197214144a0391d0908c94a5397983
2014-06-29 19:41:56 +02:00
Gauvain Pocentek
24df23a5a3 autohelp: import oslo extensions only if needed
Oslo modules are not used by all the components, so configuration
options might be invalid if these modules are unconditionaly loaded.
This change uses the requirements.txt file of a project to detect if
oslo modules have to be imported.

Change-Id: I7cd81d0c9a5ae43f0a628e2d2a6f664507e54122
2014-06-28 20:59:41 +02:00
Jenkins
cd10913a77 Merge "Bump hacking to 0.9.x series" 2014-06-17 15:28:30 +00:00
Gauvain Pocentek
a2da2135f1 Fix the target argument
We always explicitely pass a target argument to the write_docbook*
functions, so we can't use a default value for it. This patch makes the
target argument mandatory.

Change-Id: I4a193abe2634f21b52ba61c5692fe03061ac7fd3
2014-06-16 21:21:11 +02:00
Christian Berendt
d449a6bcd2 Bump hacking to 0.9.x series
Require at least 0.9.1 because 0.9.0. had a minor bug.

This change also fixes all found issues.

Rewrote some docstrings to fit H405.

Change-Id: Ifeef11b783dbe70b2652d2b9ea29d5b20f69f2ce
2014-06-12 09:17:22 +02:00
Gauvain Pocentek
bdb2f2f003 Provide a script to gerenate options changes
diff_branches.py generates a listing of the configuration options
changes that occured between 2 openstack releases.

This involves a few changes in other tools:
- the 'dump' subcommand for autohelp.py generates the serialized dict of
  options
- add a special case for the 'bindir' option to avoid getting different
  default values in different virtual environments
- the autohelp-wrapper -e switch builds the needed venv without
  running autohelp.py commands

Change-Id: I80da172b91b8d2f0a15f89f4c812864da2fea471
2014-06-10 22:50:46 +02:00