Add tutorial section on creating plugins

Expand and reorg the naming discussion.

Add a section on the mechanics of creating plugins,
with some tested example code.

Signed-off-by: Doug Hellmann <doug.hellmann@dreamhost.com>

docs/source/tutorial/creating_plugins.rst

@@ -0,0 +1,130 @@
+==================
+ Creating Plugins
+==================
+
+After a lot of trial and error, the easiest way I have found to define
+an API is to follow these steps:
+
+#. Use the `abc module`_ to create a base abstract class to define the
+   behaviors required of plugins of the API.  Developers don't have to
+   subclass from the base class, but it provides a convenient way to
+   document the API, and using an abstract base class keeps you
+   honest.
+#. Create plugins by subclassing the base class and implementing the
+   required methods.
+#. Define a unique namespace for each API by combining the name of the
+   application (or library) and a name of the API. Keep it
+   shallow. For example, "cliff.formatters" or
+   "ceilometer.pollsters.compute".
+
+Example Plugin Set
+==================
+
+The example program in this tutorial will create a plugin set with
+several data formatters, like what might be used by a command line
+program to prepare data to be printed to the console.  Each formatter
+will take as input a dictionary with string keys and built-in data
+types as values. It will return as output an iterator that produces
+the string with the data structure formatted based on the rules of the
+specific formatter being used. The formatter's constructor lets the
+caller specify the maximum width the output should have.
+
+A Plugin Base Class
+===================
+
+Step 1 above is to define an abstract base class for the API that
+needs to be implemented by each plugin. 
+
+.. literalinclude:: ../../../stevedore/example/base.py
+   :language: python
+   :linenos:
+   :prepend: # stevedore/example/base.py
+
+The constructor is a concrete method because subclasses do not need to
+override it, but the :func:`format` method does not do anything useful
+because there is no "default" implementation available.
+
+Concrete Plugins
+================
+
+The next step is to create a couple of plugin classes with concrete
+implementations of :func:`format`. A simple example formatter produces
+output with each variable name and value on a single line.
+
+.. literalinclude:: ../../../stevedore/example/simple.py
+   :language: python
+   :linenos:
+   :prepend: # stevedore/example/simple.py
+
+An alternate implementation produces a reStructuredText `field list`_.
+
+.. literalinclude:: ../../../stevedore/example/fields.py
+   :language: python
+   :linenos:
+   :prepend: # stevedore/example/fields.py
+
+There are plenty of other formatting options, but these two examples
+will give us enough to work with to demonstrate registering and using
+pluins.
+
+Registering the Plugins
+=======================
+
+To use setuptools entry points, you must package your application or
+library using setuptools. The build and packaging process generates
+metadata which is available after installation to find the plugins
+provided by each python distribution. 
+
+The entry points must be declared as belonging to a specific
+namespace, so we need to pick one before going any further. These
+plugins are formatters from the stevedore examples, so I will use the
+namespace "stevedore.example.formatter". Now it is possible to provide
+all of the necessary information in the packaging instructions:
+
+.. literalinclude:: ../../../stevedore/example/setup.py
+   :language: python
+   :linenos:
+   :emphasize-lines: 38-44
+   :prepend: # stevedore/example/setup.py
+
+The important lines are 38-44. The ``entry_points`` argument to
+:func:`setup` is a dictionary mapping the namespace for the plugins to
+a list of their definitions. Each item in the list should be a string
+with ``name = module:importable`` where *name* is the user-visible
+name for the plugin, *module* is the Python import reference for the
+module, and *importable* is the name of something that can be imported
+from inside the module.
+
+.. literalinclude:: ../../../stevedore/example/setup.py
+   :language: python
+   :lines: 37-43
+
+In this case, there are three plugins registered. The "simple" and
+"field" plugins defined above, and a "plain" plugin, which is just an
+alias for the simple plugin.
+
+setuptools Metadata
+===================
+
+During the build, setuptools copies entry point definitions to a file
+in the ".egg-info" directory for the package. For example, the file
+for stevedore is located in ``stevedore.egg-info/entry_points.txt``:
+
+::
+
+    [stevedore.example.formatter]
+    simple = stevedore.example.simple:Simple
+    field = stevedore.example.fields:FieldList
+    plain = stevedore.example.simple:Simple
+    
+    [stevedore.test.extension]
+    t2 = stevedore.tests.test_extension:FauxExtension
+    t1 = stevedore.tests.test_extension:FauxExtension
+
+:mod:`pkg_resources` uses the ``entry_points.txt`` file from all of
+the installed packages on the import path to find plugins. You should
+not modify these files, except by changing the list of entry points in
+``setup.py``.
+
+.. _abc module: http://docs.python.org/2/library/abc.html
+.. _field list: http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#field-lists

docs/source/tutorial/implementation.rst

@@ -1,68 +0,0 @@
-=====================================
- Guidelines for Implementing Plugins
-=====================================
-
-Stevedore uses setuptools entry points to define and load plugins. An
-entry point is standard way to refer to a named object defined inside
-a Python module or package. The name can be a reference to any class,
-function, or instance, as long as it is created when the containing
-module is imported (i.e., it needs to be a module-level global).
-
-Names and Namespaces
-====================
-
-Entry points are registered using a *name* in a *namespace*. 
-
-Entry point names are usually considered user-visible. For example,
-they frequently appear in configuration files where a driver is being
-enabled.  Because they are public, names are typically as short as
-possible while remaining descriptive. For example, database driver
-plugin names might be "mysql", "postgresql", "sqlite", etc.
-
-Namespaces, on the other hand, are an implementation detail, and while
-they are known to developers they are not usually exposed to users.
-The namespace naming syntax looks a lot like Python's package syntax
-(``a.b.c``) but *namespaces do not correspond to Python packages*. It
-can be convenient to use a package name as a namespace, but it's not
-required at all.  The main feature of entry points is that they can be
-discovered *across* packages. That means that a plugin can be
-developed and installed completely separately from the application
-that uses it, as long as they agree on the namespace and API.
-
-Each namespace is owned by the code that consumes the plugins and is
-used to search for entry points. The entry point names are typically
-owned by the plugin, but they can also be defined by the consuming
-code for named hooks (see :class:`~stevedore.hook.HookManager`).  The
-names of entry points must be unique within a given distribution, but
-are not necessarily unique in a namespace (again, for hook patterns).
-
-Keeping it Simple
-=================
-
-After a lot of trial and error, the easiest way I have found to define
-an API is to follow these steps:
-
-1. Define a unique namespace for each API by combining the name of the
-   application (or library) and a name of the API. Keep it
-   shallow. For example, "cliff.formatters" or
-   "ceilometer.pollsters.compute".
-2. Use the `abc module`_ to create a base abstract class to define the
-   behaviors required of plugins of the API. 
-3. Create plugins by subclassing the base class and implementing the
-   required methods.
-
-Developers don't have to subclass from the base class, but it provides
-a convenient way to document the API, and using an abstract base class
-keeps you honest.
-
-.. seealso::
-
-   * `abc module`_
-   * `Using setuptools entry points`_
-   * `Package Discovery and Resource Access using pkg_resources`_
-   * `Using Entry Points to Write Plugins | Pylons`_
-
-.. _Using setuptools entry points: http://reinout.vanrees.org/weblog/2010/01/06/zest-releaser-entry-points.html
-.. _Package Discovery and Resource Access using pkg_resources: http://pythonhosted.org/distribute/pkg_resources.html
-.. _Using Entry Points to Write Plugins | Pylons: http://docs.pylonsproject.org/projects/pylons-webframework/en/latest/advanced_pylons/entry_points_and_plugins.html
-.. _abc module: http://docs.python.org/2/library/abc.html

docs/source/tutorial/index.rst

@@ -9,12 +9,19 @@ application.
 .. toctree::
    :maxdepth: 2
 
-   implementation
+   naming
-   registration
+   creating_plugins
    loading
    calling
    testing
 
-.. seealso:: 
+.. seealso::
 
-   :doc:`/essays/pycon2013`
+   * :doc:`/essays/pycon2013`
+   * `Using setuptools entry points`_
+   * `Package Discovery and Resource Access using pkg_resources`_
+   * `Using Entry Points to Write Plugins | Pylons`_
+
+.. _Using setuptools entry points: http://reinout.vanrees.org/weblog/2010/01/06/zest-releaser-entry-points.html
+.. _Package Discovery and Resource Access using pkg_resources: http://pythonhosted.org/distribute/pkg_resources.html
+.. _Using Entry Points to Write Plugins | Pylons: http://docs.pylonsproject.org/projects/pylons-webframework/en/latest/advanced_pylons/entry_points_and_plugins.html

docs/source/tutorial/naming.rst

@@ -0,0 +1,38 @@
+===============================
+ Guidelines for Naming Plugins
+===============================
+
+Stevedore uses setuptools entry points to define and load plugins. An
+entry point is standard way to refer to a named object defined inside
+a Python module or package. The name can be a reference to any class,
+function, or instance, as long as it is created when the containing
+module is imported (i.e., it needs to be a module-level global).
+
+Names and Namespaces
+====================
+
+Entry points are registered using a *name* in a *namespace*. 
+
+Entry point names are usually considered user-visible. For example,
+they frequently appear in configuration files where a driver is being
+enabled.  Because they are public, names are typically as short as
+possible while remaining descriptive. For example, database driver
+plugin names might be "mysql", "postgresql", "sqlite", etc.
+
+Namespaces, on the other hand, are an implementation detail, and while
+they are known to developers they are not usually exposed to users.
+The namespace naming syntax looks a lot like Python's package syntax
+(``a.b.c``) but *namespaces do not correspond to Python
+packages*. Using a Python package name for an entry point namespace is
+an easy way to ensure a unique name, but it's not required at all.
+The main feature of entry points is that they can be discovered
+*across* packages. That means that a plugin can be developed and
+installed completely separately from the application that uses it, as
+long as they agree on the namespace and API.
+
+Each namespace is owned by the code that consumes the plugins and is
+used to search for entry points. The entry point names are typically
+owned by the plugin, but they can also be defined by the consuming
+code for named hooks (see :class:`~stevedore.hook.HookManager`).  The
+names of entry points must be unique within a given distribution, but
+are not necessarily unique in a namespace.

docs/source/tutorial/registration.rst

@@ -1,5 +0,0 @@
-======================
- Registering a Plugin
-======================
-
-.. describe basics of defining plugins with setuptools and entry points

setup.py

@@ -52,6 +52,11 @@ setup(
     include_package_data=True,
 
     entry_points={
+        'stevedore.example.formatter': [
+            'simple = stevedore.example.simple:Simple',
+            'field = stevedore.example.fields:FieldList',
+            'plain = stevedore.example.simple:Simple',
+        ],
         'stevedore.test.extension': [
             't1 = stevedore.tests.test_extension:FauxExtension',
             't2 = stevedore.tests.test_extension:FauxExtension',

stevedore/example/base.py

@@ -0,0 +1,21 @@
+import abc
+
+
+class FormatterBase(object):
+    """Base class for example plugin used in the tutoral.
+    """
+
+    __metaclass__ = abc.ABCMeta
+
+    def __init__(self, max_width=60):
+        self.max_width = max_width
+
+    @abc.abstractmethod
+    def format(self, data):
+        """Format the data and return unicode text.
+
+        :param data: A dictionary with string keys and simple types as
+                     values.
+        :type data: dict(str:?)
+        :returns: Iterable producing the formatted text.
+        """

stevedore/example/fields.py

@@ -0,0 +1,36 @@
+import textwrap
+
+from stevedore.example import base
+
+
+class FieldList(base.FormatterBase):
+    """Format values as a reStructuredText field list.
+
+    For example::
+
+      : name1 : value
+      : name2 : value
+      : name3 : a long value
+          will be wrapped with
+          a hanging indent
+    """
+
+    def format(self, data):
+        """Format the data and return unicode text.
+
+        :param data: A dictionary with string keys and simple types as
+                     values.
+        :type data: dict(str:?)
+        """
+        for name, value in sorted(data.items()):
+            full_text = ': {name} : {value}'.format(
+                name=name,
+                value=value,
+            )
+            wrapped_text = textwrap.fill(
+                full_text,
+                initial_indent='',
+                subsequent_indent='    ',
+                width=self.max_width,
+            )
+            yield wrapped_text + '\n'

stevedore/example/setup.py

@@ -0,0 +1,46 @@
+from setuptools import setup, find_packages
+
+setup(
+    name='stevedore-examples',
+    version='1.0',
+
+    description='Demonstration package for stevedore',
+
+    author='Doug Hellmann',
+    author_email='doug.hellmann@dreamhost.com',
+
+    url='https://github.com/dreamhost/stevedore',
+    download_url='https://github.com/dreamhost/stevedore/tarball/master',
+
+    classifiers=['Development Status :: 3 - Alpha',
+                 'License :: OSI Approved :: Apache Software License',
+                 'Programming Language :: Python',
+                 'Programming Language :: Python :: 2',
+                 'Programming Language :: Python :: 2.7',
+                 'Programming Language :: Python :: 3',
+                 'Programming Language :: Python :: 3.2',
+                 'Programming Language :: Python :: 3.3',
+                 'Intended Audience :: Developers',
+                 'Environment :: Console',
+                 ],
+
+    platforms=['Any'],
+
+    scripts=[],
+
+    provides=['stevedore.examples',
+              ],
+
+    packages=find_packages(),
+    include_package_data=True,
+
+    entry_points={
+        'stevedore.example.formatter': [
+            'simple = stevedore.example.simple:Simple',
+            'field = stevedore.example.fields:FieldList',
+            'plain = stevedore.example.simple:Simple',
+        ],
+    },
+
+    zip_safe=False,
+)

stevedore/example/simple.py

@@ -0,0 +1,20 @@
+from stevedore.example import base
+
+
+class Simple(base.FormatterBase):
+    """A very basic formatter.
+    """
+
+    def format(self, data):
+        """Format the data and return unicode text.
+
+        :param data: A dictionary with string keys and simple types as
+                     values.
+        :type data: dict(str:?)
+        """
+        for name, value in sorted(data.items()):
+            line = '{name} = {value}\n'.format(
+                name=name,
+                value=value,
+            )
+            yield line

stevedore/tests/test_example_fields.py

@@ -0,0 +1,27 @@
+"""Tests for stevedore.exmaple.fields
+"""
+
+from stevedore.example import fields
+
+
+def test_simple_items():
+    f = fields.FieldList(100)
+    text = ''.join(f.format({'a': 'A', 'b': 'B'}))
+    expected = '\n'.join([
+        ': a : A',
+        ': b : B',
+        '',
+    ])
+    assert text == expected
+
+
+def test_long_item():
+    f = fields.FieldList(25)
+    text = ''.join(f.format({'name': 'a value longer than the allowed width'}))
+    expected = '\n'.join([
+        ': name : a value longer',
+        '    than the allowed',
+        '    width',
+        '',
+    ])
+    assert text == expected

stevedore/tests/test_example_simple.py

@@ -0,0 +1,15 @@
+"""Tests for stevedore.exmaple.simple
+"""
+
+from stevedore.example import simple
+
+
+def test_simple_items():
+    f = simple.Simple(100)
+    text = ''.join(f.format({'a': 'A', 'b': 'B'}))
+    expected = '\n'.join([
+        'a = A',
+        'b = B',
+        '',
+    ])
+    assert text == expected