docs: Add developer overview guide

This patch set introduces a developer overview documentation
page to Pegleg to help developers onboard.

Change-Id: Ia453d76f024db39c6bdd97a44bfe1db1c25193f9

doc/requirements.txt

@@ -1,3 +1,9 @@
+# The order of packages is significant, because pip processes them in the order
+# of appearance. Changing the order has an impact on the overall integration
+# process, which may cause wedges in the gate later.
+
 # Documentation
 sphinx>=1.6.2
 sphinx_rtd_theme==0.2.4
+# UML image generation
+plantuml

doc/source/cli.rst

@@ -14,6 +14,8 @@
       License for the specific language governing permissions and limitations
       under the License.
 
+.. _pegleg-cli:
+
 ==========
 Pegleg CLI
 ==========

doc/source/developer-overview.rst

@@ -0,0 +1,104 @@
+..
+      Copyright 2018 AT&T Intellectual Property.
+      All Rights Reserved.
+
+      Licensed under the Apache License, Version 2.0 (the "License"); you may
+      not use this file except in compliance with the License. You may obtain
+      a copy of the License at
+
+          http://www.apache.org/licenses/LICENSE-2.0
+
+      Unless required by applicable law or agreed to in writing, software
+      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+      License for the specific language governing permissions and limitations
+      under the License.
+
+============================
+Developer Overview of Pegleg
+============================
+
+Pegleg's core mission is to, alongside Deckhand, facilitate document authoring
+strategies within `Airship`_, by:
+
+* aggregating documents across multiple revisioned repositories, each of
+  which contains multiple documents defining sites' software and hardware
+  stacks
+* providing early linting of documents prior to their collection and
+  eventual deployment
+* including utility functions enabling operators and developers alike to list
+  available sites, render individual manifests via `Deckhand`_, bootstrap
+  repositories with Pegleg-compliant directory layouts, to name a few
+
+Architecture
+============
+
+Pegleg, as a CLI, has a rather simplistic architecture. It is meaningful to
+visualize Pegleg alongside Deckhand:
+
+.. image:: images/architecture-pegleg.png
+   :alt: High level architecture of Pegleg + Deckhand
+
+Components
+==========
+
+cli
+---
+
+The Pegleg ``cli`` module implements the user-facing CLI. For more information
+about this module, reference the :ref:`CLI documentation <pegleg-cli>`.
+
+engine
+------
+
+The ``engine`` module implements the following functionality:
+
+* document linting
+* document rendering via `Deckhand`_
+* document aggregation
+* additional document utility functions
+
+Developer Workflow
+==================
+
+Because Airship is a container-centric platform, the developer workflow heavily
+utilizes containers for testing and publishing. It also requires Pegleg to
+produce multiple artifacts that are related, but separate: the Python package,
+the Docker image and the Helm chart. The code is published via the
+Docker image artifact.
+
+Pegleg strives to conform to the `Airship coding conventions`_.
+
+Python
+------
+
+The Pegleg code base lives under ``/src/bin/pegleg``. Pegleg supports py35
+interpreters.
+
+Docker
+------
+
+The Pegleg Dockerfile is located in ``/images/pegleg`` along with any
+artifacts built specifically to enable the container image. Make targets are
+used for generating and testing the artifacts.
+
+* ``make images`` - Build the Pegleg Docker image.
+
+Pegleg, as a containerized CLI, uses Docker via ``tools/pegleg.sh`` to
+execute CLI commands. Commands can also be executed using the ``Makefile``
+target: ``run_pegleg``.
+
+Testing
+=======
+
+All Pegleg tests are nested under ``/src/bin/pegleg/tests``.
+
+Pegleg comes equipped with a number of `tox`_ targets for running unit tests,
+as well as `pep8`_ and `Bandit`_ checks.
+
+.. _Airship: https://airshipit.readthedocs.io
+.. _Deckhand: https://airship-deckhand.readthedocs.io/en/latest/
+.. _Airship coding conventions: http://airshipit.readthedocs.io/en/latest/conventions.html
+.. _tox: https://tox.readthedocs.io/en/latest/
+.. _pep8: https://www.python.org/dev/peps/pep-0008/
+.. _Bandit: https://github.com/PyCQA/bandit

doc/source/diagrams/architecture-pegleg.uml

@@ -0,0 +1,39 @@
+' PlantUML file to generate the architecture component diagram
+@startuml
+
+actor "DE"
+entity "HTTP"
+entity "SQL"
+entity "WSGI"
+
+frame "Deckhand" {
+  [Control] ..> [Engine]
+}
+
+frame "Pegleg" {
+  [Pegleg CLI] ..> [Pegleg Engine]
+}
+
+frame "OpenStack Components" {
+  [Barbican] ..> [Secret Store Back-end]
+  [Keystone]
+}
+
+database "PostgreSQL" {
+  SQL - [deckhand_db]
+}
+
+HTTP - [uWSGI]
+[uWSGI] --> [Keystone Middleware]
+[Keystone Middleware] --> WSGI
+WSGI - [Control]
+[Engine] --> [SQL]
+[Control] --> [Barbican]
+[Pegleg Engine] --> [Engine]
+DE -> [Pegleg CLI]
+
+legend right
+  DE: Deployment Engineer
+endlegend
+
+@enduml

doc/source/index.rst

@@ -35,6 +35,14 @@ Design
    artifacts
    authoring_strategy
 
+Developer's Guide
+-----------------
+
+.. toctree::
+   :maxdepth: 2
+
+   developer-overview
+
 Operator's Guide
 ----------------
 

tools/gate/build-docs.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+# Builds documentation and generates documentation diagrams from .uml
+# files. Must be run from root project directory.
+
+set -ex
+rm -rf doc/build
+sphinx-build -b html doc/source doc/build/html -W -n -v
+python -m plantuml doc/source/diagrams/*.uml
+mv doc/source/diagrams/*.png doc/source/images

tools/gate/playbooks/zuul-linter.yaml

@@ -15,6 +15,6 @@
 - hosts: primary
   tasks:
     - name: Execute a Whitespace Linter check
-      command: find . -not -path "*/\.*" -not -path "*/doc/build/*" -not -name "*.tgz"  -type f -exec egrep -l " +$" {} \;
+      command: find . -not -path "*/\.*" -not -path "*/doc/build/*" -not -path "*/doc/source/images/*" -not -name "*.tgz"  -type f -exec egrep -l " +$" {} \;
       register: result
       failed_when: result.stdout != ""

tox.ini

@@ -17,35 +17,33 @@ commands = {posargs}
 [testenv:py35]
 basepython = python3.5
 commands =
-    # Run all unit tests under src/bin/pegleg
+  # Run all unit tests under src/bin/pegleg
-    tox -c src/bin/pegleg/tox.ini -e py35
+  tox -c src/bin/pegleg/tox.ini -e py35
 whitelist_externals = tox
 
 [testenv:py36]
 basepython = python3.6
 commands =
-    # Run all unit tests under src/bin/pegleg
+  # Run all unit tests under src/bin/pegleg
-    tox -c src/bin/pegleg/tox.ini -e py36
+  tox -c src/bin/pegleg/tox.ini -e py36
 whitelist_externals = tox
 
 [testenv:fmt]
 basepython = python3
 commands =
-    tox -c src/bin/pegleg/tox.ini -e fmt
+  tox -c src/bin/pegleg/tox.ini -e fmt
 whitelist_externals = tox
 
 [testenv:pep8]
 basepython = python3
 commands =
-    tox -c src/bin/pegleg/tox.ini -e pep8
+  tox -c src/bin/pegleg/tox.ini -e pep8
 whitelist_externals = tox
 
 [testenv:docs]
 basepython = python3
 deps =
-    -r{toxinidir}/src/bin/pegleg/requirements.txt
+  -r{toxinidir}/src/bin/pegleg/requirements.txt
-    -r{toxinidir}/doc/requirements.txt
+  -r{toxinidir}/doc/requirements.txt
 commands =
-    rm -rf doc/build
+  {toxinidir}/tools/gate/build-docs.sh
-    sphinx-build -b html doc/source doc/build -n -W -v
-whitelist_externals = rm