Port the Architecture, Dataflow, and Project Strucure docs

These documents used to live in the Wiki at
github.com/cloudkeep/barbican:

* https://github.com/cloudkeep/barbican/wiki/Architecture
* https://github.com/cloudkeep/barbican/wiki/Barbican-Contributions:-dataflow
* https://github.com/cloudkeep/barbican/wiki/Barbican-Contributions:-project-structure

Change-Id: I70568ce9b8deed5ac09ebe00541e892470615a75

doc/source/contribute/architecture.rst

@@ -0,0 +1,78 @@
+Architecture
+============
+
+This document describes the architecture and technology selections for
+Barbican. In general, a goal is to utilize the OpenStack architecture and
+technology selections as much as possible. An overall architecture is presented
+first, followed by technology selection details to implement the system.
+
+
+Overall Architecture
+--------------------
+
+The next figure presents an overall logical diagram for Barbican.
+
+.. image:: http://0a5aa9029f70acecc898-b578de5fa5a4bf8c8b1d7c616994834b.r11.cf1.rackcdn.com/barbican-overall-architecture.gif
+
+The API node(s) handle incoming REST requests to Barbican. These nodes can
+interact with the database directly if the request can be completed
+synchronously (such as for GET requests), otherwise the queue supports
+asynchronous processing by worker nodes. The latter could include interactions
+with third parties such as certificate authorities. As implied in the diagram,
+the architecture supports multiple API and worker nodes being added/removed
+to/from the network, to support advanced features such as auto scaling.
+Eventually, the database could be replicated across data centers supporting
+region-agnostic storage and retrieval of secured information, albeit with lags
+possible during data synchronization.
+
+
+Technology Selection
+--------------------
+
+In general, components from the `Oslo <https://wiki.openstack.org/wiki/Oslo>`_
+commons project are used within Barbican, such as config, messaging and
+logging.
+
+The next figure examines the components within Barbican.
+
+.. image:: http://0a5aa9029f70acecc898-b578de5fa5a4bf8c8b1d7c616994834b.r11.cf1.rackcdn.com/barbican-components.gif
+
+Several potential clients of the Barbican REST interface are noted, including
+`Castellan <https://github.com/openstack/castellan>`_ which presents a generic
+key management interface for other OpenStack projects with Barbican as an
+available plugin.
+
+The API node noted in the previous section is a WSGI server. Similar to
+OpenStack projects such as
+`Glance <http://docs.openstack.org/developer/glance>`_ it utilizes paste to
+support configurable middleware such as to interface with
+`Keystone <http://docs.openstack.org/developer/keystone>`_ for authentication
+and authorization services. `Pecan <http://pecan.readthedocs.org/en/latest>`_
+(a lean Python web framework inspired by CherryPy, TurboGears, and Pylons) is
+utilized to map resources to REST routes. These resources contain the controller
+business logic for Barbican and can interface with encryption/decryption
+processes (via crypto components), datastore (via repository components) and
+asynchronous tasks (via queue components).
+
+The crypto components provide a means to encrypt and decrypt information that
+accommodates a variety of encryption mechanisms and cryptographic backends (such
+as key management interoperability protocol (KMIP) or hardware security
+module (HSM)) via a plugin interface.
+
+The repository components provide an interface and database session context for
+the datastore, with model components representing entities such as Secrets
+(used to store encrypted information such as data encryption keys).
+`SQLAlchemy <http://www.sqlalchemy.org>`_ is used as the object relational
+model (ORM) layer to the database, including
+`MySQL <https://www.mysql.com/>`_ and
+`PostgreSQL <http://www.postgresql.org>`_.
+
+For asynchronous processing,
+`Oslo Messaging <https://wiki.openstack.org/wiki/Oslo/Messaging>`_ is
+used to interact with the queue, including
+`RabbitMQ <https://www.rabbitmq.com/>`_. The worker node processes tasks
+from the queue. Task components are similar to API resources in that they
+implement business logic and also interface with the datastore and follow on
+asynchronous tasks as needed. These asynchronous tasks can interface with
+external systems, such as certificate authorities for SSL/TLS certificate
+processing.

doc/source/contribute/dataflow.rst

@@ -0,0 +1,79 @@
+Dataflow
+========
+
+Bootup flow when the Barbican API service begins
+------------------------------------------------
+
+This is the sequence of calls for booting up the Barbican API server:
+
+#. ``bin/barbican.sh start``: Launches a WSGI service that performs a
+   PasteDeploy process, invoking the middleware components found in
+   ``barbican/api/middleware`` as configured in
+   ``etc/barbican/barbican-api-paste``. The middleware
+   components invoke and then execute the Pecan application created via
+   ``barbican/api/app.py:create_main_app()``, which also
+   defines the controllers (defined in ``barbican/api/controllers/``) used to
+   process requested URI routes.
+
+
+Typical flow when the Barbican API executes
+-------------------------------------------
+
+For **synchronous** calls, the following sequence is generally followed:
+
+#. A client sends an HTTP REST request to the Barbican API server.
+#. The WSGI server and routing invokes a method on one of the
+   ``XxxxController`` classes in ``barbican/api/controllers/xxxx.py``,
+   keyed to an HTTP verb (so one of POST, GET, DELETE, or PUT).
+
+   #. Example - GET /secrets:
+
+      #. In ``barbican/api/controllers/secrets.py``, the ``SecretController``'s
+         ``on_get()`` is invoked.
+      #. A ``SecretRepo`` repository class (found in
+         ``barbican/model/respositories.py``) is then used to retrieve the
+         entity of interest, in this case as a ``Secret`` entity  defined in
+         ``barbican/model/models.py``.
+      #. The payload is decrypted as needed, via
+         ``barbican/plugin/resources.py``'s ``get_secret()`` function.
+      #. A response JSON is formed and returned to the client.
+
+For **asynchronous** calls, the following sequence is generally followed:
+
+#. A client sends an HTTP REST request to the Barbican API server.
+#. The WSGI server and routing again invokes a method on one of the
+   ``XxxxcController`` classes in ``barbican/api/controllers/``.
+#. A remote procedure call (RPC) task is enqueue for later processing by a
+   worker node.
+
+   #. Example - POST /orders:
+
+      #. In ``barbican/api/controllers/orders.py``, the ``OrdersController``'s
+         ``on_post()`` is invoked.
+      #. The ``OrderRepo`` repository class (found in
+         ``barbican/model/respositories.py``) is then used to create the
+         ``barbican/model/models.py``'s ``Order``entity in a 'PENDING' state.
+      #. The Queue API's ``process_type_order()`` method on the ``TaskClient``
+         class (found in ``barbican/queue/client.py``) is invoked to send a
+         message to the queue for asynchronous processing.
+      #. A response JSON is formed and returned to the client.
+
+#. The Queue service receives the message sent above, invoking a corresponding
+   method on ``barbican/queue/server.py``'s ``Tasks`` class. This method then
+   invokes the ``process_and_suppress_exceptions()`` method on one of the
+   ``barbican/tasks/resources.py``'s ``BaseTask`` implementors. This method
+   can then utilize repository classes as needed to retrieve and update
+   entities. It may also interface with third party systems via plugins`. The
+   ``barbican/queue/client.py``'s ``TaskClient`` class above may also be
+   invoked from a worker node for follow on asynchronous processing steps.
+
+   #. Example - POST /orders (continued):
+
+      #. Continuing the example above, the queue would invoke the
+         ``process_type_order()`` method on ``barbican/queue/server.py``'s
+         ``Tasks`` class. Note the method is named the same as the
+         ``TaskClient`` method above by convention.
+
+      #. This method then invokes ``process_and_suppress_exceptions()`` on
+         the ``barbican/tasks/resources.py``'s ``BeginTypeOrder`` class. This
+         class is responsible for processing all newly-POST-ed orders.

doc/source/contribute/structure.rst

@@ -0,0 +1,36 @@
+Project Structure
+=================
+
+#. ``barbican/`` (Barbican-specific Python source files)
+
+   #. ``api/`` (REST API related source files)
+
+      #. ``controllers/`` (Pecan-based controllers handling REST-based requests)
+      #. ``middleware/`` (Middleware business logic to process REST requests)
+
+   #. ``common/`` (Modules shared across other Barbican folders)
+   #. ``model/`` (SQLAlchemy-based model classes)
+   #. ``openstack/`` (OpenStack utility Python source and folders - generated
+      from oslo-incubator)
+   #. ``plugin/`` (Plugin related logic, interfaces and look-up management)
+
+      #. ``resources.py`` (Supports interactions with plugins)
+      #. ``crypto/`` (Hardware security module (HSM) logic and plugins)
+      #. ``interface/`` (Certificate manager and secret store interface
+         classes)
+      #. (The remaining modules here are implementations of above interfaces)
+
+   #. ``queue/`` (Client and server interfaces to the queue)
+
+      #. ``client.py`` (Allows clients to publish tasks to queue)
+      #. ``server.py`` (Runs the worker service, responds to enqueued tasks)
+
+   #. ``tasks/`` (Worker-related controllers and implementations)
+   #. ``tests/`` (Unit tests)
+
+#. ``bin/`` (Start-up scripts for the Barbican nodes (API and worker))
+#. ``rpmbuild/`` (RPM package artifacts)
+#. ``etc/barbican/`` (Configuration files)
+#. ``functionaltests`` (Functional Barbican tests, DevStack gate configuration)
+#. ``doc/source`` (Sphinx documentation)
+#. ``docs/src`` (Docbook documentation - on hold pending project tagging)

doc/source/index.rst

@@ -19,6 +19,9 @@ Getting Started
    api/index
    contribute/getting_involved
    contribute/dependencies
+   contribute/architecture
+   contribute/dataflow
+   contribute/structure
    contribute/database_migrations
    setup/index
    testing