Tools & docs for backlog & abandoned spec process

Adds tooling and enhances README documentation around the backlog specs
process.

- To move a spec from the backlog to the current release, we can now use
  the ``move-spec`` tox target, e.g.

 tox -e move-spec -- [-n] [-v] specs/backlog/approved/great-idea.rst specs/train/approved

- To abandon a backlog spec - i.e. move it from specs/backlog/approved
  to the (new) specs/abandoned directory, we can now use the
  ``abandon-spec`` tox target, e.g.

 tox -e abandon-spec -- [-n] [-v] specs/backlog/it-was-a-great-idea.rst

These utilities will move the specified spec into the target directory
and create an appropriate redirect for it.

To make it so, this commit factors out a helper method that a) moves a
spec from one subdirectory to another and b) adds a redirect for it.
This is used by the existing ``move-implemented-specs`` utility as well
as the new ``move-spec`` and ``abandon-spec``.

While I was in here, I spruced up the verbose output (including for
move-implemented-specs) to be a bit more readable.

Change-Id: I322eecbacd5dc52accf6ac69c9fe1116be8c216f

README.rst

@@ -54,6 +54,9 @@ is at ``specs/kilo/redirects``.
 
             tox -r -e move-implemented-specs -- --dry-run --verbose newton
 
+          Remove the ``--dry-run`` flag to perform the actual file
+          moves/writes.
+
 This directory structure allows you to see what we thought about doing,
 decided to do, and actually got done. Users interested in functionality in a
 given release should only refer to the ``implemented`` directory.
@@ -61,26 +64,64 @@ given release should only refer to the ``implemented`` directory.
 Example specifications
 ----------------------
 
-You can find an example spec in ``specs/template.rst``.
+You can find an example spec for a given release in
+``specs/<release>-template.rst``.
 
 Backlog specifications
 ----------------------
 
-Additionally, we allow the proposal of specifications that do not have a
-developer assigned to them. These are proposed for review in the same manner as
-above, but are added to::
+Additionally, we allow the proposal of specifications that either do not have a
+developer assigned to them or are not targeted for the current release. These
+are proposed for review in the same manner as above, but are added to::
 
   specs/backlog/approved
 
-Specifications in this directory indicate the original author has either
-become unavailable, or has indicated that they are not going to implement the
+Specifications in this directory indicate the original author has either become
+unavailable or has indicated that they are not going to implement the
 specification. The specifications found here are available as projects for
-people looking to get involved with Nova. If you are interested in
-claiming a spec, start by posting a review for the specification that moves it
-from this directory to the next active release. Please set yourself as the new
+people looking to get involved with Nova. Alternatively, they may be for ideas
+generated during a given release cycle to begin design discussions, but not
+intended to be implemented until a future cycle. If you are interested in
+claiming an unassigned backlog spec, or are the assignee and are ready to
+propose it for implementation in the current release, start by posting a review
+for the specification that moves it from this directory to the next active
+release. To ensure existing links are not broken, redirects must be created in
+a fashion similar to the process for ``implemented`` specs above. The
+``move-spec`` tox target is available to help with this. For example::
+
+  tox -e move-spec -- --dry-run --verbose specs/backlog/my-great-idea.rst specs/train/approved
+
+Remove the ``--dry-run`` option to perform the actual file moves/writes.
+
+.. note:: Please do not use ``move-spec`` to repropose an unimplemented spec
+          from one release to another. Instead follow the instructions at
+          `Previously approved specifications`_
+
+When claiming an unassigned backlog spec, please set yourself as the new
 `primary assignee` and maintain the original author in the `other contributors`
 list.
 
+Abandoning a specification
+--------------------------
+.. note:: For now, this process should only be used to abandon backlog specs.
+          Please do not use this process for specs in a real release's
+          ``approved`` directory. Currently the indication that such a spec is
+          abandoned is that it never appears in any release's ``implemented``
+          directory. We may change this process in the future.
+
+If it is decided that a ``backlog`` spec is "never" going to be implemented,
+post a review moving the specification from ``specs/<release>/approved`` to
+``specs/abandoned``. As with the above processes, redirects must be created to
+ensure existing links are not broken. The ``abandon-spec`` tox target is
+available to help with this. For example::
+
+  tox -e abandon-spec -- --dry-run --verbose specs/backlog/it-was-a-great-idea.rst
+
+Remove the ``--dry-run`` option to perform the actual file moves/writes.
+
+Please add an explanation to the spec indicating why it is being abandoned, and
+update the History section accordingly.
+
 Design documents for releases prior to Juno
 -------------------------------------------
 

doc/source/index.rst

@@ -54,6 +54,14 @@ There are also some approved backlog specifications that are looking for owners:
 
    specs/backlog/index
 
+Abandoned specs live here:
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   specs/abandoned/index
+
 Process
 =======
 

doc/source/specs/abandoned

@@ -0,0 +1 @@
+../../../specs/abandoned

doc/source/specs/backlog/redirects

@@ -0,0 +1 @@
+../../../../specs/backlog/redirects

specs/abandoned/index.rst

@@ -0,0 +1,22 @@
+========================
+Abandoned Specifications
+========================
+
+This folder is for specs that were at some point approved, but we have
+since determined, for whatever reason, shall not be implemented.
+
+For the time being, let's only use this for backlog specs, and leave
+unchanged the existing process for unimplemented specs in a real
+release (i.e. leave them there in the 'approved' directory).
+
+To move a spec to this folder, use the ``abandon-spec`` tox target. In
+addition to moving the file, this ensures the proper redirect is created
+so any existing links in the wild are not broken.
+
+For example, first do a dry run (``-n``) with verbose output (``-v``) to
+see what is going to happen::
+
+  tox -e abandon-spec -- -n -v specs/backlog/approved/it-was-a-great-idea.rst
+
+When you are satisfied, remove the ``-n`` option to make it happen for
+real.

tests/test_directories.py

@@ -21,6 +21,8 @@ class TestDirectories(testtools.TestCase):
     def test_directories(self):
         releases = [x.split('/')[1] for x in glob.glob('specs/*/')]
         for release in releases:
+            if release == 'abandoned':
+                continue
             files = os.listdir("specs/%s/" % release)
             valid = ['redirects', 'implemented', 'approved']
             for name in files:

tests/test_titles.py

@@ -119,6 +119,8 @@ class TestTitles(testtools.TestCase):
         releases = [x.split('/')[1] for x in glob.glob('specs/*/')]
         self.assertTrue(len(releases), "Not able to find spec directories")
         for release in releases:
+            if release == 'abandoned':
+                continue
             with open("specs/%s-template.rst" % release) as f:
                 template = f.read()
             spec = docutils.core.publish_doctree(template)

tools/abandon_spec.py

@@ -0,0 +1,56 @@
+#!/usr/bin/env python
+# Copyright 2019 OpenStack Foundation
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from __future__ import print_function
+
+import argparse
+import os
+
+import lib
+
+
+def get_options():
+    parser = argparse.ArgumentParser(
+        description='Move a spec to the `abandoned` folder and create a '
+                    'redirect for it.')
+    parser.add_argument('-v', '--verbose', help='Enable verbose output',
+                        action='store_true')
+    parser.add_argument('-n', '--dry-run',
+                        help='Do everything except move/write the files',
+                        action='store_true')
+    parser.add_argument('spec',
+                        help='Path to the spec to be abandoned. For example, '
+                             'specs/backlog/approved/it-was-a-great-idea.rst')
+    return parser.parse_args()
+
+
+def abandon_spec(spec, verbose, dry_run):
+    spec_abs = os.path.abspath(spec)
+    if not os.path.exists(spec_abs):
+        raise ValueError('Could not find spec %s at %s' % (spec, spec_abs))
+    if not os.path.isfile(spec_abs):
+        raise ValueError('%s is not a regular file' % spec)
+
+    lib.move_spec(
+        spec_abs, os.path.abspath('specs/abandoned'), verbose, dry_run)
+
+
+def main():
+    opts = get_options()
+    abandon_spec(opts.spec, opts.verbose, opts.dry_run)
+
+
+if __name__ == '__main__':
+    main()

tools/lib.py

@@ -13,7 +13,6 @@
 
 import os
 
-from launchpadlib import launchpad
 
 LPCACHEDIR = os.path.expanduser('~/.launchpadlib/cache')
 
@@ -29,6 +28,8 @@ def get_releases():
 
 
 def get_lp_nova(consumer_name):
+    # Local import so other tools' tox envs can be dep-free.
+    from launchpadlib import launchpad
     # NOTE(mriedem): We have to use the development API since getSpecification
     # is not in the v1.0 API.
     # NOTE(melwitt): We have to use the development API because the
@@ -36,3 +37,38 @@ def get_lp_nova(consumer_name):
     lp = launchpad.Launchpad.login_anonymously(
         consumer_name, 'production', LPCACHEDIR, version='devel')
     return lp.projects['nova']
+
+
+def move_spec(srcfile_abs, destpath_abs, verbose, dry_run):
+    srcfile_bname = os.path.basename(srcfile_abs)
+    destfile_abs = os.path.join(destpath_abs, srcfile_bname)
+
+    # Move the file
+    if verbose:
+        print("MOVING %s ==> %s" % (srcfile_abs, destfile_abs))
+    if not dry_run:
+        os.rename(srcfile_abs, destfile_abs)
+
+    srcdir_abs = os.path.dirname(srcfile_abs)
+    # The redirect file is one directory up from the source file
+    redir_file = os.path.join(
+        os.path.dirname(srcdir_abs), 'redirects')
+
+    # NOTE(efried): Don't use os.path.* for paths in the redirect file; it is
+    # interpreted by tooling that uses / as the path separator (HTTP redirects)
+    # The redirect source is the last directory and the file.
+    redir_src = '/'.join(
+        [os.path.basename(os.path.dirname(srcfile_abs)), srcfile_bname])
+
+    common_path = os.path.commonpath([srcfile_abs, destfile_abs])
+    srcdir_rel = os.path.relpath(srcdir_abs, start=common_path)
+    destfile_rel_split = os.path.relpath(
+        destfile_abs, start=common_path).split(os.path.sep)
+    backdirs = ['..'] * len(srcdir_rel.split(os.path.sep))
+    redir_dest = '/'.join(backdirs + destfile_rel_split)
+    redir_line = '%s %s\n' % (redir_src, redir_dest)
+    if verbose:
+        print("Adding redirect to %s:\n\t%s" % (redir_file, redir_line))
+    if not dry_run:
+        with open(redir_file, 'a') as redirects:
+            redirects.write(redir_line)

tools/move_implemented_specs.py

@@ -42,7 +42,6 @@ def move_implemented_specs(release, verbose=False, dry_run=False):
     cwd = os.getcwd()
     approved_dir = os.path.join(cwd, 'specs', release, 'approved')
     implemented_dir = os.path.join(cwd, 'specs', release, 'implemented')
-    redirects_file = os.path.join(cwd, 'specs', release, 'redirects')
     approved_specs = os.listdir(approved_dir)
     lp_nova = lib.get_lp_nova('move-specs')
     # yay for stats and summaries
@@ -61,7 +60,7 @@ def move_implemented_specs(release, verbose=False, dry_run=False):
 
         bp_name = spec_fname.split('.rst')[0]
         if verbose:
-            print('Processing approved blueprint: %s' % bp_name)
+            print('\n=== %s ===' % bp_name)
 
         # get the blueprint object from launchpad
         lp_spec = lp_nova.getSpecification(name=bp_name)
@@ -71,17 +70,9 @@ def move_implemented_specs(release, verbose=False, dry_run=False):
             # or obsolete.
             if (lp_spec.is_complete and
                     lp_spec.implementation_status == 'Implemented'):
-                if verbose:
-                    print('Moving blueprint to implemented: %s' % spec_fname)
-
-                if not dry_run:
-                    # move the file from approved to implemented
-                    os.rename(os.path.join(approved_dir, spec_fname),
-                              os.path.join(implemented_dir, spec_fname))
-                    # add an entry to the redirects file
-                    with open(redirects_file, 'a') as redirects:
-                        redirects.write('approved/%s ../implemented/%s\n' %
-                                        (spec_fname, spec_fname))
+                lib.move_spec(
+                    os.path.join(approved_dir, spec_fname), implemented_dir,
+                    verbose, dry_run)
                 move_count += 1
             else:
                 if verbose:

tools/move_spec.py

@@ -0,0 +1,67 @@
+#!/usr/bin/env python
+# Copyright 2019 OpenStack Foundation
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from __future__ import print_function
+
+import argparse
+import os
+
+import lib
+
+
+def get_options():
+    parser = argparse.ArgumentParser(
+        description='Move a spec from one folder to another and create a '
+                    'redirect for it.')
+    parser.add_argument('-v', '--verbose', help='Enable verbose output',
+                        action='store_true')
+    parser.add_argument('-n', '--dry-run',
+                        help='Do everything except move/write the files',
+                        action='store_true')
+    parser.add_argument('spec',
+                        help='Path to the spec to be moved. For example, '
+                             'specs/backlog/approved/my-great-idea.rst')
+    parser.add_argument('destdir',
+                        help='Directory to which the spec should be moved. '
+                             'For example, specs/train/approved')
+    return parser.parse_args()
+
+
+def move_spec(spec, destdir, verbose, dry_run):
+    spec_abs = os.path.abspath(spec)
+    if not os.path.exists(spec_abs):
+        raise ValueError('Could not find spec %s at %s' % (spec, spec_abs))
+    if not os.path.isfile(spec_abs):
+        raise ValueError('%s is not a regular file' % spec)
+
+    destdir_abs = os.path.abspath(destdir)
+    if not os.path.exists(destdir_abs):
+        raise ValueError('Could not find destination directory %s at %s (you '
+                         'may have to create it)' %
+                         (destdir, destdir_abs))
+    if not os.path.isdir(destdir_abs):
+        raise ValueError('%s is not a directory' % destdir)
+
+    lib.move_spec(
+        spec_abs, destdir_abs, verbose, dry_run)
+
+
+def main():
+    opts = get_options()
+    move_spec(opts.spec, opts.destdir, opts.verbose, opts.dry_run)
+
+
+if __name__ == '__main__':
+    main()

tox.ini

@@ -43,3 +43,19 @@ deps = {[testenv:move-implemented-specs]deps}
 envdir={toxworkdir}/launchpadlib
 commands =
   python {toxinidir}/tools/count_blueprints.py {posargs}
+
+[testenv:move-spec]
+# Usage:
+# tox -e move-spec -- [--dry-run] [--verbose] path/to/spec.rst path/to/destdir
+deps=
+envdir={toxworkdir}/nodeps
+commands =
+  python {toxinidir}/tools/move_spec.py {posargs}
+
+[testenv:abandon-spec]
+# Usage:
+# tox -e abandon-spec -- [--dry-run] [--verbose] path/to/obsolete-spec.rst
+deps=
+envdir={toxworkdir}/nodeps
+commands =
+  python {toxinidir}/tools/abandon_spec.py {posargs}