Documenting project direction and design

This commit adds documentation around project direction and conventions.
Additionally, I've added doc8 checks to make sure or .rst and .txt files
in this repository follow good conventions, and fixed a check related to
graphviz external .dot files.

Change-Id: I1b73b3839b86198f3d56587ca3fb2644dc231f00

CONSTITUTION.rst

@@ -0,0 +1,53 @@
+====================
+Octavia Constitution
+====================
+
+This document defines the guiding principles that project leadership will be
+following in creating, improving and maintaining the Octavia project.
+
+Octavia is an OpenStack project
+-------------------------------
+This means we try to run things the same way other "canonized" OpenStack
+projects operate from a procedural perspective. This is because we hope that
+Octavia will eventually become a standard part of any OpenStack deployment.
+
+Octavia is as open as OpenStack
+-------------------------------
+Octavia tries to follow the same standards for openness that the OpenStack
+project also strives to follow: https://wiki.openstack.org/wiki/Open
+We are committed to open design, development, and community.
+
+Octavia is "free"
+-----------------
+We mean that both in the "beer" and in the "speech" sense. That is to say, the
+reference implementation for Octavia should be made up only of open source
+components that share the same kind of unencumbered licensing that OpenStack
+uses.
+
+Note that this does not mean we are against having vendors develop products
+which can replace some of the components within Octavia. (For example, the
+Octavia VM images might be replaced by a vendor's proprietary VM image.)
+Rather, it means that:
+* The reference implementation should always be open source and unencumbered.
+* We are typically not interested in making design compromises in order to work
+  with a vendor's proprietary product. If a vendor wants to develop a component
+  for Octavia, then the vendor should bend to Octavia's needs, not the other
+  way around.
+
+Octavia is a load balancer for large operators
+----------------------------------------------
+That's not to say that small operators can't use it. (In fact, we expect it to
+work well for small deployements, too.) But what we mean here is that if in
+creating, improving or maintaining Octavia we somehow make it unable to meet
+the needs of a typical large operator (or that operator's users), then we have
+failed.
+
+Octavia follows the best coding and design conventions
+------------------------------------------------------
+For the most part, Octavia tries to follow the coding standards set forth for
+the OpenStack project in general: http://docs.openstack.org/developer/hacking/
+More specific additional standards can be found in the HACKING.rst file in the
+same directory as this constitution.
+
+Any exceptions should be well justified and documented. (Comments in or near
+the breach in coding standards are usually sufficient documentation.)

HACKING.rst

@@ -18,3 +18,104 @@ For every new feature, unit tests should be created that both test and
 bug that had no unit test, a new passing unit test should be added. If a
 submitted bug fix does have a unit test, be sure to add a new one that fails
 without the patch and passes with the patch.
+
+Everything is python
+--------------------
+Although OpenStack apparently allows either python or C++ code, at this time
+we don't envision needing anything other than python (and standard, supported
+open source modules) for anything we intend to do in Octavia.
+
+Idempotency
+-----------
+With as much as is going on inside Octavia, its likely that certain messages
+and commands will be repeatedly processed. It's important that this doesn't
+break the functionality of the load balancing service. Therefore, as much as
+possible, algorithms and interfaces should be made as idempotent as possible.
+
+Centralize intelligence, de-centralize workload
+-----------------------------------------------
+This means that tasks which need to be done relatively infrequently but require
+either additional knowledge about the state of other components in the Octavia
+system, advanced logic behind decisions, or otherwise a high degree of
+intelligence should be done by centralized components (ex. controllers) within
+the Octavia system. Examples of this might include:
+* Generating haproxy configuration files
+* Managing the lifecycle of Octavia VMs
+* Moving a loadbalancer instance from one Octavia VM to another.
+
+On the other hand, tasks done extremely often, or which entail a significant
+load on the system should be pushed as far out to the most horizontally
+scalable components as possible. Examples of this might include:
+* Serving actual client requests to end-users (ie. running haproxy)
+* Monitoring pool members for failure and sending notifications about this
+* Processing log files
+
+There will often be a balance that needs to be struck between these two design
+considerations for any given task for which an algorithm needs to be designed.
+In considering how to strike this balance, always consider the conditions
+that will be present in a large operator environment.
+
+Also, as a secondary benefit of centralizing intelligence, minor feature
+additions and bugfixes can often be accomplished in a large operator
+environment without having to touch every Octavia VM running in said
+environment.
+
+All APIs are versioned
+----------------------
+This includes "internal" APIs between Octavia components. Experience coding in
+the Neutron LBaaS project has taught us that in a large project with many
+heterogenous parts, throughout the lifecycle of this project, different parts
+will evolve at different rates. It is important that these components are
+allowed to do so without hindering or being hindered by parallel development
+in other components.
+
+It is also likely that in very large deployments, there might be tens- or
+hundreds-of-thousands of individual instances of a given component deployed
+(most likely, the Octavia VMs). It is unreasonable to expect a large operator
+to update all of these components at once. Therefore it is likely that for a
+significant amount of time during a roll-out of a new version, both the old
+and new versions of a given component must be able to be controlled or
+otherwise interfaced with by the new components.
+
+Both of the above considerations can be allowed for if we use versioning of
+APIs where components interact with each other.
+
+Scalability and resilience are as important as functionality
+------------------------------------------------------------
+Octavia is meant to be an *operator scale* load balancer. As such, it's usually
+not enough just to get something working: It also needs to be scalable. For
+most components, "scalable" implies horizontally scalable.
+
+In any large operational environment, resilience to failures is a necessity.
+Practically speaking, this means that all components of the system that make up
+Octavia should be monitored in one way or another, and that where possible
+automatic recovery from the most common kinds of failures should become a
+standard feature. Where automatic recovery is not an option, then some form
+of notification about the failure should be implemented.
+
+Avoid premature optimization
+----------------------------
+Understand that being "high performance" is often not the same thing as being
+"scalable." First get the thing to work in an intelligent way. Only worry about
+making it fast if speed becomes an issue.
+
+Don't repeat yourself
+---------------------
+Octavia strives to follow DRY principles. There should be one source of truth,
+and repetition of code should be avoided.
+
+Security is not an afterthought
+-------------------------------
+The load balancer is often both the most visible public interface to a given
+user application, but load balancers themselves often have direct access to
+sensitive components and data within the application environment. Security bugs
+will happen, but in general we should not approve designs which have known
+significant security problems, or which could be made more secure by better
+design.
+
+Octavia should follow industry standards
+----------------------------------------
+By "industry standards" we either mean RFCs or well-established best practices.
+We are generally not interested in defining new standards if a prior open
+standard already exists. We should also avoid doing things which directly
+or indirectly contradict established standards.

README.rst

@@ -1,6 +1,6 @@
-========
+=======
- Octavia
+Octavia
-========
+=======
 
 Operator-grade open source scalable load balancer.
 
@@ -15,3 +15,4 @@ contributors, please see the CONSTITUTION.rst file in this directory, or
 specifications in the specs/ subdirectory. Other documentation can be
 found in the docs/ directory.
 
+Please also see the ROADMAP.rst file for our project roadmap.

ROADMAP.rst

@@ -0,0 +1,134 @@
+===============
+Octavia Roadmap
+===============
+Given the complete lack of any real project management tools in the OpenStack
+environment, for the time being we'll be planning and tracking project
+progression using this ROADMAP.rst file. (It may be more appropriate to keep
+this document in the wiki-- we'll see what people think of this.)
+
+This file consists of three sections:
+
+1. A long-term timeline
+2. A short-term timeline (derived from the above, reads like a todo list)
+3. Outstanding design questions (things yet to be addressed of near-term to
+   medium-term importance)
+
+==================
+Long-term timeline
+==================
+
+
+Major milestone: Full-featured Neutron LBaaS in Juno
+----------------------------------------------------
+Description: Neutron LBaaS API and other interfaces are full-featured enough
+to allow for a user interface that delivers most of the features Octavia will
+be implementing. Work commenced on Octavia coding.
+
+OpenStack release target: Juno
+
+Neutron LBaaS progress:
+* New object model support
+* TLS support
+
+Octavia progress:
+* Consensus on:
+  * Constitution
+  * Road map
+  * Component design
+  * APIs
+* Initial code underway (perhaps alpha release?)
+
+
+Major milestone: Octavia Version 0.5
+------------------------------------
+Description: First usable release of Octavia. Delivers load balancing services
+on multiple Nova VMs. Single, centralized command and control (not scalable).
+
+OpenStack release target: Kilo
+
+Neutron LBaaS progress:
+* Flavor support
+* L7 switching support
+* Updated horizon UI
+* Hooks for Heat integration
+
+Octavia progress:
+* Octavia delivers all functionality of Neutron LBaaS user API
+* Octavia VMs image building scripts
+* Octavia operator API
+* Horizon UI for operators
+* Neutron LBaaS driver interface for Octavia
+* Non-voting Neutron third-party CI for Octavia to ensure Neutron code changes
+  don't break Octavia
+* Command-and-control layer handles:
+  * Octavia VM lifecycle maangement
+  * Octavia VM monitoring
+  * Octavia VM command and control
+  * Neutron LBaaS service deployment
+* Resilient topologies for Octavia VMs (ie. HA for the VMs)
+* "Experimental" project status
+
+
+Major milestone: Octavia Version 1.0
+------------------------------------
+Description: Operator-scale release of Octavia. Delivers load balancing
+services on multiple Nova VMs, and has scalable command and control layer.
+
+OpenStack release target: "L" release
+
+Octavia progress:
+* Possibly becomes reference implementation for Neutron LBaaS
+* Project becomes incubated
+* Fully scalable and HA command-and-control layer
+* Improvements to Horizon UI for operators
+
+
+Major milestone: Octavia Version 2.0
+------------------------------------
+Description: "Web scale" release of Octavia. Delivers all the features of
+1.0, plus allows for horizontal scaling of individual load-balanced services.
+(ie. n-node active-active topologies).
+
+OpenStack release target: ???
+
+Octavia progress:
+* "Two layer" load balancing topology implemented where layers 1-4 handled by
+  routing infrastructure, and 4-7 handled by Octavia VMs acting in parallel.
+* Improvements to Horizon UI for operators
+
+
+===================
+Short-term timeline
+===================
+
+Highest priority:
+* See Neutron LBaaS work scheduled for Juno through to completion.
+* Import google docs describing v0.5, v1.0 and v2.0 Octavia into specs folder
+  of this repository
+* Get reviews and consensus on the same
+
+Medium priority:
+* Define and document Octavia VM <=> Controller RESTful APIs
+* Define best practices for credential management between Octavia VM and
+  controllers (suggested: bi-direction server / client certificat verification)
+* Collect requirements for Operator API
+* Start work on Octavia VM image
+* Start work on Octavia VM agent
+* Start work on controllers
+* Create Neutron LBaaS driver for Octavia
+* Get Octavia to work in devstack
+* Flesh out the above items with more detailed checklists as work commences on
+  them
+
+Lower priority:
+* Create mock-ups of and start coding Horizon UI for Octavia operators
+* Create non-voting CI interface for testing changes relating to Octavia in
+  gerrit
+
+
+============================
+Outstanding design questions
+============================
+
+* We need to start putting together specifications for the Operator API for
+  Octavia.

specs-tests/tests/test_titles.py

@@ -95,10 +95,11 @@ class TestTitles(testtools.TestCase):
         self.assertEqual(0, len(titles[refs]))
 
     def test_template(self):
-        files = glob.glob('specs/*.rst') + glob.glob('specs/*/*')
+        files = set(glob.glob('specs/*.rst') + glob.glob('specs/*/*'))
+        files = files - set(glob.glob('specs/*/*.dot'))
         for filename in files:
             self.assertTrue(filename.endswith(".rst"),
-                            "spec's file must uses 'rst' extension.")
+                            "spec's file must use 'rst' extension.")
             with open(filename) as f:
                 data = f.read()
             spec = docutils.core.publish_doctree(data)

specs/example.dot

@@ -1,7 +1,8 @@
-// This work is licensed under a Creative Commons Attribution 3.0
+/* This work is licensed under a Creative Commons Attribution 3.0
-// Unported License.
+ * Unported License.
-//
+ *
-// http://creativecommons.org/licenses/by/3.0/legalcode
+ * http://creativecommons.org/licenses/by/3.0/legalcode
+ */
 digraph G {
   label="Sample Graph"
 

specs/template.rst

@@ -39,7 +39,8 @@ Some notes about using this template:
   For more complicated diagrams that need "real" graphics, yet still should
   be in the git revision control system, GraphViz .dot files are acceptable.
   If you require an image (screenshot) for your BP, attaching that to the BP
-  and checking it in is also accepted. However, text representations are prefered.
+  and checking it in is also accepted. However, text representations are
+  prefered.
 
 * Diagram examples
 
@@ -186,7 +187,8 @@ proposed changes to the data model.
 
 Questions which need to be addressed by this section include:
 
-* What new data objects and/or database schema changes is this going to require?
+* What new data objects and/or database schema changes is this going to
+  require?
 
 * What database migrations will accompany this change.
 
@@ -324,8 +326,9 @@ changes to an existing notification, or removing a notification.
 Other end user impact
 ---------------------
 
-Aside from the API, are there other ways a user will interact with this feature?
+Aside from the API, are there other ways a user will interact with this
-Keep in mind that 'user' in this context could mean either tenant or operator.
+feature? Keep in mind that 'user' in this context could mean either tenant or
+operator.
 
 * Does this change have an impact on python-neutronclient? What does the user
   interface there look like?
@@ -345,12 +348,12 @@ Examples of things to consider here include:
 * A small change in a utility function or a commonly used decorator can have a
   large impacts on performance.
 
-* Calls which result in a database queries (whether direct or via conductor) can
+* Calls which result in a database queries (whether direct or via conductor)
-  have a profound impact on performance when called in critical sections of the
+  can have a profound impact on performance when called in critical sections
-  code.
+  of the code.
 
-* Will the change include any locking, and if so what considerations are there on
+* Will the change include any locking, and if so what considerations are there
-  holding the lock?
+  on holding the lock?
 
 Other deployer impact
 ---------------------
@@ -415,8 +418,8 @@ but we're mostly trying to understand the timeline for implementation.
 Dependencies
 ============
 
-* Include specific references to specs and/or blueprints in octavia, or in other
+* Include specific references to specs and/or blueprints in octavia, or in
-  projects, that this one either depends on or is related to.
+  other projects, that this one either depends on or is related to.
 
 * If this requires functionality of another project that is not currently used
   by Octavia document that fact.

test-requirements.txt

@@ -10,3 +10,4 @@ ordereddict
 testrepository>=0.0.18
 testtools>=0.9.34
 WebTest>=2.0
+doc8

tox.ini

@@ -13,6 +13,8 @@ commands = python setup.py testr --slowest --testr-args='{posargs}'
 
 [testenv:pep8]
 commands = flake8
+           doc8 specs doc/source octavia \
+           CONSTITUTION.rst HACKING.rst README.rst ROADMAP.rst
 
 [testenv:docs]
 commands = python setup.py build_sphinx
@@ -21,3 +23,6 @@ commands = python setup.py build_sphinx
 ignore = None
 exclude = .tox,doc
 show-source = true
+
+[doc8]
+max-line-length = 79