Update README and remove contribution guide

This change refreshes the project README and removes the development
guide, which has moved to docs.airshipit.org/developers.html

Change-Id: Ie398f96fdef6236c7881ec858c4429f5f5cbb9e2
Signed-off-by: Drew Walters <andrew.walters@att.com>
This commit is contained in:
Drew Walters 2020-05-12 21:38:17 +00:00
parent 7fe5ca0645
commit d32541cef9
6 changed files with 251 additions and 460 deletions

View File

@ -1,9 +0,0 @@
# Treasuremap
This documentation project outlines a reference architecture for automated
cloud provisioning and management, leveraging a collection of interoperable
open-source tools.
## Architecture
To get started, see [architecture](https://airship-treasuremap.readthedocs.io/en/latest/index.html).

43
README.rst Normal file
View File

@ -0,0 +1,43 @@
Treasuremap
===========
Treasuremap is a collection of Airship documents, CICD tools, and reference
architecture diagrams that describe and deploy virtual and baremetal Airship
environments.
At the moment, this repository only contains Airship 1 artifacts. For Airship 2
documents, refer to the main `airshipctl repository`_.
To learn more about Airship, visit `Airship documentation`_.
.. _Airship documentation: https://docs.airshipit.org
.. _airshipctl repository: https://opendev.org/airship/airshipctl
Deploy Treasuremap
------------------
Learn how to use Treasuremap, use Airship-in-a-Bottle, and author Airship
documents in the `Treasuremap documentation`_.
.. _Treasuremap documentation: https://docs.airshipit.org/treasuremap
Contributing
------------
Airship is under active development and welcomes new contributors! Please
read our `developer guidelines`_ to begin contributing.
.. _developer guidelines: https://docs.airshipit.org/developers.html
Get in Touch
------------
We primarily use the `Airship Slack workspace`_ and the `Airship mailing
lists`_ for communication.
For a list of meetings and additional communication channels, visit the
`Airship wiki`_.
.. _Airship mailing lists: http://lists.airshipit.org
.. _Airship Slack workspace: https://airshipit.org/slack
.. _Airship wiki: https://wiki.openstack.org/wiki/Airship#Get_in_Touch

196
doc/source/architecture.rst Normal file
View File

@ -0,0 +1,196 @@
Architecture
============
Airship is a collection of components that coordinate to form means of
configuring and deploying and maintaining
a `Kubernetes <https://kubernetes.io/>`__ environment using a
declarative set of `yaml <http://yaml.org/>`__ documents.
More specifically, the current focus of this project is the implementation of
OpenStack on Kubernetes (OOK).
.. image:: diagrams/architecture.png
Component Projects
==================
Pegleg
------
`Pegleg <https://opendev.org/airship/pegleg>`_ is a document aggregator
that provides early linting and validations via Deckhand, a document management
micro-service within Airship.
Shipyard
--------
`Shipyard <https://opendev.org/airship/shipyard>`_ is the directed acyclic
graph controller for Kubernetes and OpenStack control plane life cycle
management.
Shipyard provides the entrypoint for the following aspects of the control plane:
Designs and Secrets
^^^^^^^^^^^^^^^^^^^
Site designs, including the configuration of bare metal host nodes, network
design, operating systems, Kubernetes nodes, Armada manifests, Helm charts,
and any other descriptors that define the build out of a group of servers enter
the Airship via Shipyard. Secrets, such as passwords and certificates, use the same
mechanism.
The designs and secrets are stored in Airship's Deckhand, providing for version
history and secure storage among other document-based conveniences.
Actions
^^^^^^^
Interaction with the site's control plane is done via invocation of actions in
Shipyard. Each action is backed by a workflow implemented as a directed acyclic
graph (DAG) that runs using Apache Airflow. Shipyard provides a mechanism to
monitor and control the execution of the workflow.
Drydock
-------
`Drydock <https://opendev.org/airship/drydock>`_ is a provisioning orchestrator
for baremetal servers that translates a YAML-based declaritive site topology into a
physical undercloud that can be used for building out a enterprise Kubernetes cluster.
It uses plugins to leverage existing provisioning systems to build the servers allowing
integration with the provisioning system that best fits the goals and environment of a site.
Capabilities
^^^^^^^^^^^^
* Initial IPMI configuration for PXE booting new servers.
* Support for Canonical MAAS provisioning.
* Configuration of complex network topologies including bonding,
tagged VLANs and static routes
* Support for running behind a corporate proxy
* Extensible boot action system for placing files and SystemD
units on nodes for post-deployment execution
* Supports Keystone-based authentication and authorization
Deckhand
--------
`Deckhand <https://opendev.org/airship/deckhand>`_ is a document-based
configuration storage service built with auditability and validation in mind.
Core Responsibilities
^^^^^^^^^^^^^^^^^^^^^
* layering - helps reduce duplication in configuration by applying the notion
of inheritance to documents
* substitution - provides separation between secret data and other
configuration data for security purposes and reduces data duplication by
allowing common data to be defined once and substituted elsewhere dynamically
* revision history - maintains well-defined collections of documents within
immutable revisions that are meant to operate together, while providing the
ability to rollback to previous revisions
* validation - allows services to implement and register different kinds of
validations and report errors
* secret management - leverages existing OpenStack APIs -- namely
`Barbican`_ -- to reliably and securely store sensitive data
Armada
------
`Armada <https://opendev.org/airship/armada>`_ is a tool for managing multiple
Helm charts with dependencies by centralizing all configurations in a single
Armada YAML and providing life-cycle hooks for all Helm releases.
Core Responsibilities
^^^^^^^^^^^^^^^^^^^^^
* Multiple Chart Deployments and Upgrades driven by Armada Manifests
* Manage multiple chart dependencies using Chart Groups
* Enhancing base Helm functionality
* Supports Keystone-based authentication and authorization
Kubernetes
----------
`Kubernetes <https://github.com/kubernetes/kubernetes>`_ is an open source
system for managing containerized applications across multiple hosts, providing
basic mechanisms for deployment, maintenance, and scaling of applications.
Promenade
---------
`Promenade <https://opendev.org/airship/promenade>`_ is a tool for
bootstrapping a resilient, self-hosted Kubernetes cluster and managing its
life-cycle.
Bootstrapping begins by provisioning a single-node cluster with a complete,
configurable Airship infrastructure. After hosts are added to the cluster, the
original bootstrapping node can be re-provisioned to avoid subtle differences
that could result in future issues.
Promenade provides cluster resiliency against both node failures and full
cluster restarts. It does so by leveraging
`Helm <https://github.com/kubernetes/helm>`_ charts to manage core Kubernetes
assets directly on each host, to ensure their availability.
Helm
----
`Helm <https://github.com/kubernetes/helm>`_ is a package manager for Kubernetes.
It helps you define, install, and upgrade even the most complex Kubernetes
applications using Helm charts.
A chart is a collection of files that describe a related set of Kubernetes
resources. Helm wraps up each chart's deployment into a concrete release,
a tidy little box that is a collection of all the Kubernetes resources that
compose that service, and so you can interact with a collection of Kubernetes
resources that compose a release as a single unit, either to install, upgrade,
or remove.
At its core, the value that Helm brings to the table -- at least for us -- is
allowing us to templatize our experience with Kubernetes resources, providing
a standard interface for operators or high-level software orchestrators to
control the installation and life cycle of Kubernetes applications.
OpenStack-Helm
--------------
The `OpenStack-Helm <https://github.com/openstack/openstack-helm>`_ project
provides a framework to enable the deployment, maintenance, and upgrading of
loosely coupled OpenStack services and their dependencies individually or as
part of complex environments.
OpenStack-Helm is essentially a marriage of Kubernetes, Helm, and OpenStack,
and seeks to create Helm charts for each OpenStack service. These Helm charts
provide complete life cycle management for these OpenStack services.
Users of OpenStack-Helm either deploy all or individual OpenStack components
along with their required dependencies. It heavily borrows concepts from
Stackanetes and complex Helm application deployments. Ideally, at the end of
the day, this project is meant to be a collaborative project that brings
OpenStack applications into a cloud-native model.
Divingbell
----------
`Divingbell <https://opendev.org/airship/divingbell>`_ is a lightweight
solution for:
1. Bare metal configuration management for a few very targeted use cases
2. Bare metal package manager orchestration
What problems does it solve?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The needs identified for Divingbell were:
1. To plug gaps in day 1 tools (e.g., Drydock) for node configuration
2. To provide a day 2 solution for managing these configurations going forward
3. [Future] To provide a day 2 solution for system level host patching
Berth
-----
`Berth <https://opendev.org/airship/berth>`_ is a deliberately minimalist VM
runner for Kubernetes.
Process Flows
=============
.. image:: diagrams/genesis.png
.. image:: diagrams/deploy_site.png
.. _Barbican: https://docs.openstack.org/barbican/latest/api/
.. _Helm Homepage: https://helm.sh/
.. _Kubernetes Homepage: https://kubernetes.io/
.. _Openstack: https://www.openstack.org/
.. _Openstack Helm: https://github.com/openstack/openstack-helm
.. _Treasuremap: https://opendev.org/airship/treasuremap
.. _yaml: http://yaml.org/

View File

@ -50,8 +50,8 @@ source_suffix = '.rst'
master_doc = 'index'
# General information about the project.
project = u'Airship Integration'
copyright = u'2019 The Airship Authors.'
project = u'Treasuremap'
copyright = u'2019-2020 The Airship Authors.'
author = u'Airship Authors'
# The version info for the project you're documenting, acts as replacement for

View File

@ -1,242 +0,0 @@
=================
Development Guide
=================
Welcome
-------
Thank you for your interest in Airship. Our community is eager to help you
contribute to the success of our project and welcome you as a member of our
community!
We invite you to reach out to us at any time via the `Airship mailing list`_ or
`#airshipit IRC channel`_ on freenode.
Welcome aboard!
.. _Airship mailing list: http://lists.airshipit.org
.. _#airshipit IRC channel: irc://chat.freenode.net:6667
Getting Started
---------------
Airship is a collection of open source tools for automating cloud provisioning
and management. Airship provides a declarative framework for defining and
managing the life cycle of open infrastructure tools and the underlying
hardware. These tools include OpenStack for virtual machines, Kubernetes for
container orchestration, and MaaS for bare metal, with planned support for
OpenStack Ironic.
We recommend that new contributors begin by reading the high-level architecture
overview included in our `treasuremap`_ documentation. The architectural
overview introduces each Airship component, their core responsibilities, and
their integration points.
.. _treasuremap: https://airship-treasuremap.readthedocs.io/en/latest
Deep Dive
---------
Each Airship component is accompanied by its own documentation that provides an
extensive overview of the component. With so many components, it can be
challenging to find a starting point.
We recommend the following:
Try an Airship environment
~~~~~~~~~~~~~~~~~~~~~~~~~~
Airship provides two single-node environments for demo and development purpose.
`Airship-in-a-Bottle`_ is a set of reference documents and shell scripts that
stand up a full Airship environment with the execution of a script.
`Airskiff`_ is a light-weight development environment bundled with a set of
deployment scripts that provides a single-node Airship environment. Airskiff
uses minikube to bootstrap Kubernetes, so it does not include Drydock, MaaS, or
Promenade.
Additionally, we provide a reference architecture for easily deploying a
smaller, demo site.
`Airsloop`_ is a fully-authored Airship site that can be quickly deployed as a
baremetal, demo lab.
.. _Airship-in-a-Bottle: https://opendev.org/airship/in-a-bottle
.. _Airskiff: https://airship-treasuremap.readthedocs.io/en/latest/airskiff.html
.. _Airsloop: https://airship-treasuremap.readthedocs.io/en/latest/airsloop.html
Focus on a component
~~~~~~~~~~~~~~~~~~~~
When starting out, focusing on one Airship component allows you to become
intricately familiar with the responsibilities of that component and understand
its function in the Airship integration. Because the components are modeled
after each other, you will also become familiar with the same patterns and
conventions that all Airship components use.
Airship source code lives in the `OpenDev Airship namespace`_. To clone an
Airship project, execute the following, replacing `<component>` with the name
of the Airship component you want to clone.
.. code-block bash::
git clone https://opendev.org/airship/<component>.git
Refer to the component's documentation to get started. A list of each
component's documentation is listed below for reference:
* `Armada`_
* `Deckhand`_
* `Divingbell`_
* `Drydock`_
* `Pegleg`_
* `Promenade`_
* `Shipyard`_
.. _OpenDev Airship namespace: https://opendev.org/airship
.. _Armada: https://airship-armada.readthedocs.io
.. _Deckhand: https://airship-deckhand.readthedocs.io
.. _Divingbell: https://airship-divingbell.readthedocs.io
.. _Drydock: https://airship-drydock.readthedocs.io
.. _Pegleg: https://airship-pegleg.readthedocs.io
.. _Promenade: https://airship-promenade.readthedocs.io
.. _Shipyard: https://airship-shipyard.readthedocs.io
Find a Storyboard task or story
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Airship work items are tracked using Storyboard. A board of items can be found
`here`_.
Once you find an item to work on, simply assign the item to yourself or leave a
comment that you plan to provide implementation for the item.
.. _here: https://storyboard.openstack.org/#!/project_group/85
Testing Changes
---------------
Testing of Airship changes can be accomplished several ways:
#. Standalone, single component testing
#. Integration testing
#. Linting, unit, and functional tests/linting
.. note:: Testing changes to charts in Airship repositories is best
accomplished using the integration method describe below.
Standalone Testing
~~~~~~~~~~~~~~~~~~
Standalone testing of Airship components, i.e. using an Airship component as a
Python project, provides the quickest feedback loop of the three methods and
allows developers to make changes on the fly. We recommend testing initial code
changes using this method to see results in real-time.
Each Airship component written in Python has pre-requisites and guides for
running the project in a standalone capacity. Refer to the documentation listed
below.
* `Armada`_
* `Deckhand`_
* `Drydock`_
* `Pegleg`_
* `Promenade`_
* `Shipyard`_
Integration Testing
~~~~~~~~~~~~~~~~~~~
While each Airship component supports individual usage, Airship components
have several integration points that should be exercised after modifying
functionality.
We maintain several environments that encompass these integration points:
#. `Airskiff`_: Integration of Armada, Deckhand, Shipyard, and Pegleg
#. `Airship-in-a-Bottle Multinode`: Full Airship integration
For changes that merely impact software delivery components, exercising a full
Airskiff deployment is often sufficient. Otherwise, we recommend using the
Airship-in-a-Bottle Multinode environment.
Each environment's documentation covers the process required to build and test
component images.
.. _Airskiff: https://airship-treasuremap.readthedocs.io/en/latest/
airskiff.html
.. _Airship-in-a-Bottle Multinode: http://git.openstack.org/cgit/openstack/
airship-in-a-bottle/tree/tools/multi_nodes_gate/README.rst
Final Checks
~~~~~~~~~~~~
Airship projects provide Makefiles to run unit, integration, and functional
tests as well as lint Python code for PEP8 compliance and Helm charts for
successful template rendering. All checks are gated by Zuul before a change can
be merged. For more information on executing these checks, refer to
project-specific documentation.
Third party CI tools, such as Jenkins, report results on Airship-in-a-Bottle
patches. These can be exposed using the "Toggle CI" button in the bottom
left-hand page of any gerrit change.
Pushing code
------------
Airship uses the `OpenDev gerrit`_ for code review. Refer to the `OpenStack
Contributing Guide`_ for a tutorial on submitting changes to Gerrit code
review.
.. _OpenDev gerrit: https://review.opendev.org
.. _OpenStack Contributing Guide: https://docs.openstack.org/horizon/latest/contributor/contributing.html
Next steps
----------
Upon pushing a change to gerrit, Zuul continuous integration will post job
results on your patch. Refer to the job output by clicking on the job itself to
determine if further action is required. If it's not clear why a job failed,
please reach out to a team member in IRC. We are happy to assist!
Assuming all continuous integration jobs succeed, Airship community members and
core developers will review your patch and provide feedback. Many patches are
submitted to Airship projects each day. If your patch does not receive feedback
for several days, please reach out using IRC or the Airship mailing list.
Merging code
------------
Like most OpenDev projects, Airship patches require two +2 code review votes
from core members to merge. Once you have addressed all outstanding feedback,
your change will be merged.
Beyond
------
Congratulations! After your first change merges, please keep up-to-date with
the team. We hold two weekly meetings for project and design discussion:
Our weekly #airshipit IRC meeting provides an opportunity to discuss project
operations.
Our weekly design call provides an opportunity for in-depth discussion of new
and existing Airship features.
For more information on the times of each meeting, refer to the `Airship
wiki`_.
.. _Airship wiki: https://wiki.openstack.org/wiki/Airship

View File

@ -1,210 +1,13 @@
Airship Treasuremap
===================
Airship is a collection of components that coordinate to form means of
configuring and deploying and maintaining
a `Kubernetes <https://kubernetes.io/>`__ environment using a
declarative set of `yaml <http://yaml.org/>`__ documents.
More specifically, the current focus of this project is the implementation of
OpenStack on Kubernetes (OOK).
.. image:: diagrams/architecture.png
Component Projects
==================
Pegleg
------
`Pegleg <https://opendev.org/airship/pegleg>`_ is a document aggregator
that provides early linting and validations via Deckhand, a document management
micro-service within Airship.
Shipyard
--------
`Shipyard <https://opendev.org/airship/shipyard>`_ is the directed acyclic
graph controller for Kubernetes and OpenStack control plane life cycle
management.
Shipyard provides the entrypoint for the following aspects of the control plane:
Designs and Secrets
^^^^^^^^^^^^^^^^^^^
Site designs, including the configuration of bare metal host nodes, network
design, operating systems, Kubernetes nodes, Armada manifests, Helm charts,
and any other descriptors that define the build out of a group of servers enter
the Airship via Shipyard. Secrets, such as passwords and certificates, use the same
mechanism.
The designs and secrets are stored in Airship's Deckhand, providing for version
history and secure storage among other document-based conveniences.
Actions
^^^^^^^
Interaction with the site's control plane is done via invocation of actions in
Shipyard. Each action is backed by a workflow implemented as a directed acyclic
graph (DAG) that runs using Apache Airflow. Shipyard provides a mechanism to
monitor and control the execution of the workflow.
Drydock
-------
`Drydock <https://opendev.org/airship/drydock>`_ is a provisioning orchestrator
for baremetal servers that translates a YAML-based declaritive site topology into a
physical undercloud that can be used for building out a enterprise Kubernetes cluster.
It uses plugins to leverage existing provisioning systems to build the servers allowing
integration with the provisioning system that best fits the goals and environment of a site.
Capabilities
^^^^^^^^^^^^
* Initial IPMI configuration for PXE booting new servers.
* Support for Canonical MAAS provisioning.
* Configuration of complex network topologies including bonding,
tagged VLANs and static routes
* Support for running behind a corporate proxy
* Extensible boot action system for placing files and SystemD
units on nodes for post-deployment execution
* Supports Keystone-based authentication and authorization
Deckhand
--------
`Deckhand <https://opendev.org/airship/deckhand>`_ is a document-based
configuration storage service built with auditability and validation in mind.
Core Responsibilities
^^^^^^^^^^^^^^^^^^^^^
* layering - helps reduce duplication in configuration by applying the notion
of inheritance to documents
* substitution - provides separation between secret data and other
configuration data for security purposes and reduces data duplication by
allowing common data to be defined once and substituted elsewhere dynamically
* revision history - maintains well-defined collections of documents within
immutable revisions that are meant to operate together, while providing the
ability to rollback to previous revisions
* validation - allows services to implement and register different kinds of
validations and report errors
* secret management - leverages existing OpenStack APIs -- namely
`Barbican`_ -- to reliably and securely store sensitive data
Armada
------
`Armada <https://opendev.org/airship/armada>`_ is a tool for managing multiple
Helm charts with dependencies by centralizing all configurations in a single
Armada YAML and providing life-cycle hooks for all Helm releases.
Core Responsibilities
^^^^^^^^^^^^^^^^^^^^^
* Multiple Chart Deployments and Upgrades driven by Armada Manifests
* Manage multiple chart dependencies using Chart Groups
* Enhancing base Helm functionality
* Supports Keystone-based authentication and authorization
Kubernetes
----------
`Kubernetes <https://github.com/kubernetes/kubernetes>`_ is an open source
system for managing containerized applications across multiple hosts, providing
basic mechanisms for deployment, maintenance, and scaling of applications.
Promenade
---------
`Promenade <https://opendev.org/airship/promenade>`_ is a tool for
bootstrapping a resilient, self-hosted Kubernetes cluster and managing its
life-cycle.
Bootstrapping begins by provisioning a single-node cluster with a complete,
configurable Airship infrastructure. After hosts are added to the cluster, the
original bootstrapping node can be re-provisioned to avoid subtle differences
that could result in future issues.
Promenade provides cluster resiliency against both node failures and full
cluster restarts. It does so by leveraging
`Helm <https://github.com/kubernetes/helm>`_ charts to manage core Kubernetes
assets directly on each host, to ensure their availability.
Helm
----
`Helm <https://github.com/kubernetes/helm>`_ is a package manager for Kubernetes.
It helps you define, install, and upgrade even the most complex Kubernetes
applications using Helm charts.
A chart is a collection of files that describe a related set of Kubernetes
resources. Helm wraps up each chart's deployment into a concrete release,
a tidy little box that is a collection of all the Kubernetes resources that
compose that service, and so you can interact with a collection of Kubernetes
resources that compose a release as a single unit, either to install, upgrade,
or remove.
At its core, the value that Helm brings to the table -- at least for us -- is
allowing us to templatize our experience with Kubernetes resources, providing
a standard interface for operators or high-level software orchestrators to
control the installation and life cycle of Kubernetes applications.
OpenStack-Helm
--------------
The `OpenStack-Helm <https://github.com/openstack/openstack-helm>`_ project
provides a framework to enable the deployment, maintenance, and upgrading of
loosely coupled OpenStack services and their dependencies individually or as
part of complex environments.
OpenStack-Helm is essentially a marriage of Kubernetes, Helm, and OpenStack,
and seeks to create Helm charts for each OpenStack service. These Helm charts
provide complete life cycle management for these OpenStack services.
Users of OpenStack-Helm either deploy all or individual OpenStack components
along with their required dependencies. It heavily borrows concepts from
Stackanetes and complex Helm application deployments. Ideally, at the end of
the day, this project is meant to be a collaborative project that brings
OpenStack applications into a cloud-native model.
Divingbell
----------
`Divingbell <https://opendev.org/airship/divingbell>`_ is a lightweight
solution for:
1. Bare metal configuration management for a few very targeted use cases
2. Bare metal package manager orchestration
What problems does it solve?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The needs identified for Divingbell were:
1. To plug gaps in day 1 tools (e.g., Drydock) for node configuration
2. To provide a day 2 solution for managing these configurations going forward
3. [Future] To provide a day 2 solution for system level host patching
Berth
-----
`Berth <https://opendev.org/airship/berth>`_ is a deliberately minimalist VM
runner for Kubernetes.
Process Flows
=============
.. image:: diagrams/genesis.png
.. image:: diagrams/deploy_site.png
.. include:: ../../README.rst
.. toctree::
:maxdepth: 2
authoring_and_deployment
config_update_guide
troubleshooting_guide
seaworthy
airsloop
airskiff
development_guide
.. _Barbican: https://docs.openstack.org/barbican/latest/api/
.. _Helm Homepage: https://helm.sh/
.. _Kubernetes Homepage: https://kubernetes.io/
.. _Openstack: https://www.openstack.org/
.. _Openstack Helm: https://github.com/openstack/openstack-helm
.. _Treasuremap: https://opendev.org/airship/treasuremap
.. _yaml: http://yaml.org/
:hidden:
:maxdepth: 2
architecture
authoring_and_deployment
config_update_guide
troubleshooting_guide
seaworthy
airsloop
airskiff