Make oslo.tools more usable by newcomers

These changes make oslo.tools more user friendly by documenting tools
introducing commands shortcut etc...

Introduced changes:
- add doc and build it with tox
- add tox and shortcut commands
- centralize tools
- launch zuul jobs
- apply pep8
- apply bashate

Change-Id: I0f948c9f507b8c951c5748265699a2f515350ff9

.zuul.yaml

@@ -0,0 +1,12 @@
+- project:
+    templates:
+      - openstack-python3-ussuri-jobs
+      - build-openstack-docs-pti
+    check:
+      jobs:
+        - openstack-tox-pep8:
+            irrelevant-files:
+              - ^doc/.*$
+        - openstack-tox-bashate:
+            files:
+              - ^.*\.sh$

CONTRIBUTING.rst

@@ -0,0 +1,16 @@
+If you would like to contribute to the development of OpenStack,
+you must follow the steps in this page:
+
+   http://docs.openstack.org/infra/manual/developers.html
+
+Once those steps have been completed, changes to OpenStack
+should be submitted for review via the Gerrit tool, following
+the workflow documented at:
+
+   http://docs.openstack.org/infra/manual/developers.html#development-workflow
+
+Pull requests submitted through GitHub will be ignored.
+
+Bugs should be filed on Launchpad, not GitHub:
+
+   https://bugs.launchpad.net/oslo.utils

doc/requirements.txt

@@ -0,0 +1,9 @@
+# The order of packages is significant, because pip processes them in the order
+# of appearance. Changing the order has an impact on the overall integration
+# process, which may cause wedges in the gate later.
+# this is required for the docs build jobs
+sphinx>=1.8.0,<2.0.0;python_version=='2.7' # BSD
+sphinx>=1.8.0,!=2.1.0;python_version>='3.4' # BSD
+openstackdocstheme>=1.20.0 # Apache-2.0
+reno>=2.5.0 # Apache-2.0
+fixtures>=3.0.0 # Apache-2.0/BSD

doc/source/conf.py

@@ -0,0 +1,52 @@
+# -*- coding: utf-8 -*-
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath('../..'))
+# -- General configuration ----------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
+extensions = [
+    'sphinx.ext.autodoc',
+    'openstackdocstheme'
+]
+
+# openstackdocstheme options
+repository_name = 'openstack/oslo.tools'
+bug_project = 'oslo.tools'
+bug_tag = ''
+
+# The suffix of source filenames.
+source_suffix = '.rst'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+copyright = u'2020, OpenStack Foundation'
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# -- Options for HTML output --------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  Major themes that come with
+# Sphinx are currently 'default' and 'sphinxdoc'.
+# html_theme_path = ["."]
+# html_theme = '_theme'
+# html_static_path = ['static']
+html_theme = 'openstackdocs'

doc/source/configuration/index.rst

@@ -0,0 +1,35 @@
+=============
+Configuration
+=============
+
+oslo.tools uses oslo.config to define and manage configuration
+options to allow maintainers to define personal specific configuration
+to uses with the different tools available.
+
+Config
+======
+
+You can define your own local configuration by creating the ``~/.oslo.conf``
+file at the root of your home directory.
+
+For more further reading about how works oslo.config please take a look to the
+`documentation`_.
+
+The ``[oslo]`` section is expected in this config file.
+
+In this section you can declare the following options:
+
+- ``repo_root`` which is where you store locally your git repositories
+  and where you should clone the `governance`_ project directory
+
+Example of ``oslo.conf`` content::
+
+    [oslo]
+    repo_root=~/dev/governance/
+
+Also you can find a file named ``oslo.conf.sample`` at the root of this
+project directory, this is a config sample that you can reuse and adapt to
+your needs.
+
+.. _documentation: https://docs.openstack.org/oslo.config/latest/
+.. _governance: https://opendev.org/openstack/governance/

doc/source/contributor/index.rst

@@ -0,0 +1,5 @@
+============
+Contributing
+============
+
+.. include:: ../../../CONTRIBUTING.rst

doc/source/index.rst

@@ -0,0 +1,23 @@
+==========
+oslo.tools
+==========
+
+The *oslo.tools* project provides tools and utilities to help the `Oslo`_
+community and the oslo maintainers in daily tasks.
+
+.. toctree::
+   :maxdepth: 1
+
+   install/index
+   configuration/index
+   users/index
+   oslobot/index
+   contributor/index
+
+.. rubric:: Indices and tables
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+
+.. _Oslo: https://wiki.openstack.org/wiki/Oslo

doc/source/install/index.rst

@@ -0,0 +1,16 @@
+============
+Installation
+============
+
+From sources::
+
+    $ git clone https://opendev.org/openstack/oslo.tools
+    $ cd oslo.tools
+    $ mkvirtualenv oslo.tools
+    $ pip install -r requirements
+
+To use available tools you also need to clone the governance directory which
+will be used by scripts, for more informations please take a look to the
+`configuration`_ section.
+
+.. _configuration: configuration/index

doc/source/oslobot/index.rst

@@ -0,0 +1,26 @@
+==============
+Oslo admin bot
+==============
+
+A tiny bot that will help in doing some periodic checks and
+other oslo (the project) activities such as stating when weekly
+meetings are and (periodically) checking (and reporting on) the periodic
+oslo jobs and various other nifty functionality that we can think of
+in the future (if only we *all* had an administrative assistant).
+
+How to use it
+=============
+
+0. Read up on `errbot`_
+1. Setup a `virtualenv`_
+2. Enter that `virtualenv`_
+3. Install the ``requirements.txt`` into that virtualenv, typically
+   performed via ``pip install -r requirements.txt`` after doing
+   ``pip install pip --upgrade`` to get a newer pip package.
+4. Adjust ``config.py`` and provide it a valid (or unique IRC
+   nickname and admins).
+5. Run ``errbot -d -p $PWD/oslobot.pid``
+6. Profit.
+
+.. _virtualenv: https://virtualenv.pypa.io/en/stable/
+.. _errbot: http://errbot.io/

doc/source/users/index.rst

@@ -0,0 +1,31 @@
+=====
+Users
+=====
+
+This section describe how to use this project, available tools and
+how to use them.
+
+All tools are available under the ``tools`` directory.
+
+Please first refer on how to setup `configuration`_ for oslo.tools.
+
+List oslo projects
+==================
+
+Allow you to list oslo projects defined in governance.
+
+For the following example consider you store your git repositories
+under the ``~/dev`` directory, where you have already cloned the `governance`_
+project.
+
+Usage::
+
+    $ tools/list_oslo_projects.py
+    $ tools/list_oslo_projects.py --repo_root ~/dev/
+
+Or by using tox::
+
+    $ tox -e list-oslo-projects -- --repo_root ~/dev/
+
+.. _configuration: ../configuration/index.html
+.. _governance: https://opendev.org/openstack/governance/

oslo.conf.sample

@@ -0,0 +1,2 @@
+[oslo]
+repo_root=~/dev/governance/

oslobot/plugins/oslobot/oslobot.py

@@ -14,27 +14,25 @@ from __future__ import absolute_import
 from __future__ import unicode_literals
 
 import collections
+from concurrent import futures
 import copy
 from datetime import date
 import json
 import re
 
-from concurrent import futures
-import six
-from six.moves import http_client
-
 from dateutil import parser
 from dateutil.relativedelta import relativedelta
 
+from errbot import botcmd
+from errbot import BotPlugin
 import feedparser
 from oslo_utils import timeutils
 import requests
+import six
+from six.moves import http_client
 from six.moves.urllib.parse import urlencode as compat_urlencode
 from tabulate import tabulate
 
-from errbot import botcmd
-from errbot import BotPlugin
-
 BAD_VALUE = '??'
 NA_VALUE = "N/A"
 
@@ -311,8 +309,8 @@ class OsloBotPlugin(BotPlugin):
                     'last_fail_url': BAD_VALUE,
                 }
             else:
-                if (r.status_code == http_client.BAD_REQUEST
-                        and 'No Failed Runs' in r.text):
+                if (r.status_code == http_client.BAD_REQUEST and
+                        'No Failed Runs' in r.text):
                     return {
                         'status': 'All OK (no recent failures)',
                         'last_fail': NA_VALUE,
@@ -365,7 +363,7 @@ class OsloBotPlugin(BotPlugin):
         ]
         tbl_body = []
         if (self.config['periodic_shorten'] and
-                self.config['shortener_api_key']):
+           self.config['shortener_api_key']):
             s = GoogleShortener(
                 self.log, self.config['shortener_api_key'],
                 timeout=(self.config['shortener_connect_timeout'],

setup.cfg

@@ -0,0 +1,18 @@
+[metadata]
+name = oslo.tools
+summary = Openstack oslo tools and utilities for community
+description-file =
+    README.rst
+author = OpenStack
+author-email = openstack-discuss@lists.openstack.org
+home-page = https://www.openstack.org/
+classifier =
+    Environment :: OpenStack
+    Intended Audience :: Information Technology
+    Intended Audience :: System Administrators
+    License :: OSI Approved :: Apache Software License
+    Operating System :: POSIX :: Linux
+    Programming Language :: Python
+    Programming Language :: Python :: 3 :: Only
+    Programming Language :: Python :: 3.6
+    Programming Language :: Python :: 3.7

setup.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import setuptools
+
+setuptools.setup(
+    setup_requires=['pbr'],
+    pbr=True)

test-requirements.txt

@@ -0,0 +1,5 @@
+# The order of packages is significant, because pip processes them in the order
+# of appearance. Changing the order has an impact on the overall integration
+# process, which may cause wedges in the gate later.
+
+hacking>=1.0.0,<1.1.0

tools/filter_git_history.sh

@@ -6,8 +6,8 @@
 set -e
 
 if [ $# -lt 1 ]; then
-  echo "Usage $0 <files to keep>"
-  exit 1
+    echo "Usage $0 <files to keep>"
+    exit 1
 fi
 
 set -x

tools/list_latest_releases.sh

@@ -21,8 +21,7 @@ repodir=$(cd $bindir/../../.. && pwd)
 # Make sure no pager is configured so the output is not blocked
 export PAGER=
 
-if [ -z "$*" ]
-then
+if [ -z "$*" ]; then
     libs=$($bindir/list_oslo_projects.py | egrep -v -e '(oslo.version|cookiecutter|incubator)')
 else
     libs="$*"
@@ -36,13 +35,11 @@ function get_last_tag {
 
 function list_versions {
     # Show the tag for each library
-    for lib in $*
-    do
+    for lib in $*; do
         the_date=""
         cd $repodir/$lib
         highest_tag=$(get_last_tag)
-        if [ -z "$highest_tag" ]
-        then
+        if [ -z "$highest_tag" ]; then
             the_date="0000-00-00 00:00:00 +0000"
             highest_tag="UNRELEASED"
         else

tox.ini

@@ -0,0 +1,47 @@
+[tox]
+minversion=3.1.0
+envlist=py37,pep8,bashate,docs
+skipdist=True
+ignore_basepython_conflict=true
+
+[testenv]
+usedevelop=True
+deps = -r{toxinidir}/test-requirements.txt
+
+[testenv:list-oslo-projects]
+commands =
+    {toxinidir}/tools/list_oslo_projects.py {posargs}
+
+[testenv:pep8]
+commands = flake8
+
+[testenv:bashate]
+deps = bashate
+whitelist_externals = bash
+commands = bash -c "find {toxinidir}          \
+         -not \( -type d -name .?\* -prune \) \
+         -type f                              \
+         -not -name \*~                       \
+         -not -name \*.md                     \
+         -name \*.sh                          \
+         -print0 | xargs -0 bashate -i E006 -v"
+
+[testenv:venv]
+commands = {posargs}
+
+[testenv:docs]
+deps =
+  -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt}
+  -r{toxinidir}/requirements.txt
+  -r{toxinidir}/doc/requirements.txt
+commands =
+  sphinx-build -v -a -E -W -d doc/build/doctrees --keep-going -b html doc/source doc/build/html
+
+[flake8]
+# E123, E125 skipped as they are invalid PEP-8.
+# E501 skipped because some of the code files include templates
+#      that end up quite wide
+show-source = True
+ignore = E123,E125,E501,H405,W504,E731
+builtins = _
+exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,build,release-tag-*