x/anvil
Code Issues Proposed changes

Folsom continued.

Browse Source 
1. Update how status is returned/structured/shown
2. Cleanup documentation.
This commit is contained in:
Joshua Harlow
2012-08-17 14:44:07 -07:00
parent 6c019b0f0b
commit 5fbbd572af
29 changed files with 251 additions and 619 deletions
Download Patch File Download Diff File Expand all files Collapse all files

55
docs/source/topics/advanced/addingyourowndistro.rst
View File

@@ -1,55 +0,0 @@
==========
Adding your own distribution
==========
Your mission...
=============
So you have decided you want to venture into the bowels of ANVIL
and want to get support for your latest and greatest distribution. This
wiki will hopefully make that adventure simpler by listing out the key
places, files and configs that may have to be adjusted to get that to
work. This tape will self-destruct in 5 seconds. 1..2..3..4..5, boom!
Steps
=====
Snapshot
--------
One of the most useful things to do will be to get a virtual machine
with your distribution and setup a stable state. Create a snapshot of
that stable state just in-case you *bork* your machine later on.
Logging
-------
Now turn ensure you run ``DEBUG/VERBOSE`` logging using ``-vv``. This
will be useful to see exactly what actions and commands are being ran
instead of the default ``INFO`` level logging which is just meant for
simple informational messages about the underlying actions which are
occurring.
Configs
-------
By looking at the config folder ``distros`` you should exactly which
packages and pips and commands are needed for each component by looking
at a similar distribution. So you first task is to determine exactly
what versions are available for your distribution. If a version doesn’t
exist the you may need to resort to either using the `pypi`_ index or
having to package it yourself. If a version is to new, this is usually
ok (your mileage may vary) and if its to old then that might also not be
ok (your mileage may vary).
Try it
------
Now that you have provided a new `YAML`_ distro file you should be able
to run through the simple setup wiki and see if the install will pass.
If that does try starting and then seeing if everything has started up
correctly.
.. _pypi: http://pypi.python.org
.. _YAML: http://yaml.org/

60
docs/source/topics/advanced/addingyourownpersona.rst
View File

@@ -1,60 +0,0 @@
==========
Adding your own persona
==========
Your mission...
=============
So you have decided you want to venture into the bowels of ANVIL
and want to alter what is installed/started/stopped, the order of what
is installed/started/stopped, what subsystems are activated (or the
component options). This wiki will hopefully make that adventure simpler
by listing out the key places, files and configs that may have to be
adjusted to get that to work. This tape will self-destruct in 5 seconds.
1..2..3..4..5, boom!
Steps
=====
Snapshot
--------
One of the most useful things to do will be to get a virtual machine
with your distribution and setup a stable state. Create a snapshot of
that stable state just in-case you *bork* your machine later on.
Logging
-------
Now turn ensure you run ``DEBUG/VERBOSE`` logging using ``-vv``. This
will be useful to see exactly what actions and commands are being ran
instead of the default ``INFO`` level logging which is just meant for
simple informational messages about the underlying actions which are
occurring.
Configs
-------
By looking at the config folder ``personas`` you should a file called
``devstack.sh.yaml``. This file contains the component order of
installation (ie the ``db`` before ``keystone``), a nice useful
description of the persona and subsystems for the previously specified
components and any options these components may have. So you first task
is to determine exactly what of these you wish to change (if any). Note
that changing the component order may not always work (ie typically
starting components are dependent, ie the message queue needs to be
started before nova). To add in new components check the ``distros``
folder to determine exactly what that component is named (typically this
is common) and alter the persona file as desired. To alter the
``subsystems`` or ``options`` section you will have to jump in the code
and check for what these values could be (TODO make that better).
Try it
------
Now that you have provided a new `YAML`_ persona file you should be able
to run the ``stack`` program with that persona through the ``-p``
option.
.. _YAML: http://yaml.org/

163
docs/source/topics/advanced/design.rst
View File

@@ -1,163 +0,0 @@
========
Design
========
How it works
------------
ANVIL is based along the following system design
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Having shared components/actions be shared (using object oriented
practices)
- Having specific actions be isolated to its component (and easily
readable)
- Being simple enough to read yet following standard python software
development practices and patterns
Directory structure is the following
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Parent classes located in *anvil/component.py*, it contains the
root install/uninstall/start/stop classes
- Subclasses in *anvil/components/*, it contains the individual
install classes for each openstack component
- Running modes implementations in *anvil/runners/* (ie fork,
screen, upstart)
- Packaging implementations in *anvil/packaging/*
- Image uploading/registry/management (for glance) in *anvil/image/*
- Shared classes and utils in *anvil/*
- Main entry point/s in *anvil/progs/*
- Other various tools in *tools/*
- Configuration in *conf/* (see below)
Example
~~~~~~~
Install object model
^^^^^^^^^^^^^^^^^^^^
Here is the **install** components (default set only) class hierarchy:
.. figure:: http://farm8.staticflickr.com/7043/6894250521_d882b770d1_o.png
:align: center
Classes
'''''''
From this example the root classes job are:
- Python install component
- Ensures *pips* are installed (child classes specify which pips)
- Ensures python directories (ie *setup.py*) are setup (child classes
specify which directories)
- Package install component
- Installs packages that are required for install (child classes
specify which packages)
- Sets up and performs parameter replacement on config files (child
classes specify which config files)
- Sets up symlinks to config or other files (child classes specify
these symlinks)
- Tracks what was setup so that it can be removed/uninstalled
- Component base (used by **all** install/uninstall/start/stop
component classes)
- Holds configuration object, component name, packaging and other
shared members…
- Allows for overriding of dependency function and pre-run verification
Functions
'''''''''
For a install class the following functions are activated (in the
following order by *anvil/actions.py*):
::
download()
Performs the main git download (or other download type) to the
application target directory.
::
configure()
Configures the components files (symlinks, configuration and logging
files…)
::
pre_install()
Child class specific function that can be used to do anything before
an install (ie set a ubuntu mysql pre-install root password)
::
install()
Installs distribution packages, python packages (*pip*), sets up
python directories (ie *python setup.py develop*) and any other
child class specific actions.
::
post_install()
Child class specific function that can be used to do anything after
an install (ie run *nova-manage db sync*)
Other object models
^^^^^^^^^^^^^^^^^^^
- `Start object model (for default set only)`_
- `Stop object model (for default set only)`_
- `Uninstall object model (for default set only)`_
Configuring
-----------
For those of you that are brave enough to change *stack* here are some
starting points.
conf/stack.ini
~~~~~~~~~~~~~~
Check out *conf/stack.ini* for various configuration settings applied
(branches, git repositories…). Check out the header of that file for how
the customized configuration values are parsed and what they may result
in.
conf/distros
~~~~~~~~~~~~
Check out *conf/distros* for the `YAML`_ files that describe
pkgs/cmds/pips for various distributions which are required by the
different OpenStack components to run correctly. The versions and pip
names listed for each distribution should be the correct version that is
known to work with a given OpenStack release.
conf/templates/
~~~~~~~~~~~~~~~
Check out *conf/templates/* for various component specific settings and
files. All of these files are *templates* with sections or text that
needs to be filled in by the ``stack`` script to become a *complete*
file.
These files may have strings of the format ``%NAME%`` where ``NAME``
will most often be adjusted to a real value by the *stack* script.
An example where this is useful is say for the following line:
::
admin_token = %SERVICE_TOKEN%
Since the script will either prompt for this value (or generate it for
you) we can not have this statically set in a configuration file.
.. _Start object model (for default set only): http://farm8.staticflickr.com/7046/6894981327_a583bcb4fc_o.png
.. _Stop object model (for default set only): http://farm8.staticflickr.com/7059/6894981341_e6d4901b20_o.png
.. _Uninstall object model (for default set only): http://farm8.staticflickr.com/7177/6894981357_fef65b28d3_o.png
.. _YAML: http://yaml.org/

11
docs/source/topics/advanced/index.rst
View File

@@ -1,11 +0,0 @@
===============
Advanced
===============
.. toctree::
:maxdepth: 2
addingyourowndistro
addingyourownpersona
design

8
docs/source/topics/bugshugscode.rst
View File

@@ -25,13 +25,10 @@ Stable *tags* can also be downloaded:
https://github.com/yahoo/Openstack-Anvil/tags
**Note:** that for these tags you may have to edit ``conf/anvil.ini``
to point to tags other than ``master``
Bugs/Features
=============
Please use `github’s issue tracking system`_ or `launchpad’s issue tracking system`_ to report or follow bugs or to discuss features and get support.
Please use `launchpad’s issue tracking system`_ to report or follow bugs or to discuss features and get support.
Hacking
=============
@@ -42,9 +39,8 @@ Feel free to hack but please try to follow the `hacking guidelines`_
Discussions
===========
Please either use launchpad’s email system or find us on ``irc.freenode.net`` in channel ``#openstack-anvil`` or in the main openstack dev channel ``#openstack-dev``. Feel free to bug us!
Please use launchpad’s blueprint/bug system for contacting us about bugs or new features (or submit them!). Much appreciated!
.. _apache version 2.0 license: https://github.com/yahoo/Openstack-Anvil/blob/master/LICENSE
.. _github’s issue tracking system: https://github.com/yahoo/Openstack-Anvil/issues
.. _launchpad’s issue tracking system: http://launchpad.net/anvil
.. _hacking guidelines: https://github.com/yahoo/Openstack-Anvil/blob/master/HACKING.md

47
docs/source/topics/features.rst
View File

@@ -2,33 +2,38 @@
Features
========
- Supports more than one distribution.
- Currently RHEL 6.2 (with `epel`_), Ubuntu 11.10, Fedora 16, Ubuntu 12.10 (seems to work)
- Supports dry-run mode (to see what *would* happen)
- Supports varying installation *personas*
- A single ``anvil.ini`` file that shows common configuration
- Supports install/uninstall/starting/stopping of OpenStack components.
- In various styles (daemonizing via `forking`_, `screen`_, `upstart`_)
- Multi distribution installs via a single tool (TODO: fix for folsom)
- A single ``anvil.ini`` file that shows common/component configuration
- Supports the following *actions* on the various of OpenStack components.
#. **Installing**: downloading, installing dependencies (`pypi`_ and distribution packaging specifics)
and configuring component files and symlinks
#. **Starting**: starting of the components sub-programs with
the needed configuration via the common `daemon`_ model (with a ``pid``, ``stderr`` and ``stdout`` file set)
#. **Stopping**: stopping of the previously started components
#. **Uninstalling**: removing installed configuration, undoing of installed files/directories,
and removing of packaging to get back to an initial 'clean' state
#. **Testing**: running each components unit tests (and in the future performing a simple set of integration tests)
#. **Packaging**: creating a basic set of packages for the desired distributions
#. **Status**: checking the status of the running components sub-programs
- Supports dry-run mode (to see what *would* happen for each action)
- Tracking of all actions taken by a component via tracking like files.
- Written in python so it matches the style of other `OpenStack`_ components.
- Code decoupling (thus encouraging re-use by others)
#. Components/actions are isolated as individual classes (and so on). This
decouples component installation from the action and decoupling of the
commands/packages/pips the component will use to install itself from the
component...
#. Supports installation *personas* that define what is to be installed, thus
decoupling the 'what' from the 'how'
- Extensively documented distribution specifics
- Packages and pip (with versions known to work!) dependencies
- Any needed distribution specific actions (ie service names…)
- Follows standard software development practices (for everyones sanity).
- Functions, classes, objects and more (oh my!)
- Still *readable* by someone with limited python knowledge.
- The ability to be unit-tested!
- Extensive logging
- Progress resuming so that when you install you can ``ctrl+c`` ``./smithy`` and resume later (where applicable).
- Extensive logging (and debug mode)
.. _epel: http://fedoraproject.org/wiki/EPEL
.. _forking: http://users.telenet.be/bartl/classicperl/fork/all.html
.. _screen: http://www.manpagez.com/man/1/screen/
.. _upstart: http://upstart.ubuntu.com/
.. _OpenStack: http://openstack.org/
.. _pypi: http://pypi.python.org/pypi
.. _daemon: http://en.wikipedia.org/wiki/Daemon_(computing)

131
docs/source/topics/gettingstarted.rst
View File

@@ -16,18 +16,9 @@ Prerequisites
Linux
-----
One of the tested Linux distributions (RHEL 6.2, Ubuntu 11.10, Fedora
16)
One of the tested Linux distributions (RHEL 6.2+ until further updated)
You can get Ubuntu 11.10 (**64-bit** is preferred) from
http://releases.ubuntu.com/11.10/
You can get RHEL 6.2 (**64-bit** is preferred) from
http://rhn.redhat.com/.
You can get Fedora 16 (**64-bit** is preferred) from
https://fedoraproject.org/get-fedora, so don’t worry if you do not have
a RHN subscription.
You can get RHEL 6.2+ (**64-bit** is preferred) from http://rhn.redhat.com/.
Networking
----------
@@ -84,14 +75,12 @@ Installation
Pre-setup
---------
Since RHEL/Fedora requires a `tty`_ to perform ``sudo`` commands we need
Since RHEL requires a `tty`_ to perform ``sudo`` commands we need
to disable this so ``sudo`` can run without a `tty`_. This seems needed
since nova and other components attempt to do ``sudo`` commands. This
isn’t possible in RHEL/Fedora unless you disable this (since those
isn’t possible in RHEL unless you disable this (since those
instances won’t have a `tty`_ ).
**For RHEL and Fedora 16:**
::
$ sudo visudo
@@ -127,38 +116,10 @@ This can be typically solved by running the following (and then updating ``anvil
$ sudo chmod -R a+rwx /home/openstack
**For Ubuntu:**
You are off the hook.
Users
-----
We need to add a admin user so that horizon can run under `apache`_.
**For Ubuntu:**
::
$ apt-get install sudo -y
$ sudo adduser horizon
$ sudo adduser horizon admin
**For RHEL/Fedora 16:**
You are off the hook as long as your user has ``sudo`` access.
Get git!
--------
**For Ubuntu:**
::
$ sudo apt-get install git -y
**For RHEL/Fedora 16:**
::
$ sudo yum install git -y
@@ -173,52 +134,9 @@ We’ll grab the latest version of ANVIL via git:
$ git clone git://github.com/yahoo/Openstack-Anvil.git anvil
Now setup the prerequisites needed to run (select the appropriate shell script for your distro):
::
$ cd anvil/warmups && sudo ./$DISTRO.sh
Configuration
-------------
Apache configuration
~~~~~~~~~~~~~~~~~~~~
We need to adjust the configuration of ANVIL to reflect the above
user (``iff you created a user``).
Open ``conf/anvil.ini``
**Change section:**
::
[horizon]
# What user will apache be serving from.
#
# Root will typically not work (for apache on most distros)
# sudo adduser <username> then sudo adduser <username> admin will be what you want to set this up (in ubuntu)
# I typically use user "horizon" for ubuntu and the runtime user (who will have sudo access) for RHEL.
#
# NOTE: If blank the currently executing user will be used.
apache_user = ${APACHE_USER:-}
**To:**
::
[horizon]
# What user will apache be serving from.
#
# Root will typically not work (for apache on most distros)
# sudo adduser <username> then sudo adduser <username> admin will be what you want to set this up (in ubuntu)
# I typically use user "horizon" for ubuntu and the runtime user (who will have sudo access) for RHEL.
#
# NOTE: If blank the currently executing user will be used.
apache_user = ${APACHE_USER:-horizon}
Network configuration
~~~~~~~~~~~~~~~~~~~~~
@@ -269,7 +187,7 @@ Now install *OpenStacks* components by running the following:
::
sudo ./smithy -a install -d ~/openstack
sudo ./smithy -a install
You should see a set of distribution packages and/or pips being
installed, python setups occurring and configuration files being written
@@ -285,7 +203,7 @@ Now that you have installed *OpenStack* you can now start your
::
sudo ./smithy -a start -d ~/openstack
sudo ./smithy -a start
If you desire more informational output add a ``-v`` or a ``-vv`` to
that command.
@@ -332,29 +250,28 @@ EC2 apis run the following to get your EC2 certs:
::
euca.sh $OS_USERNAME $OS_TENANT_NAME
./euca.sh $OS_USERNAME $OS_TENANT_NAME
It broke?
~~~~~~~~~
*Otherwise* you may have to look at the output of what was started. To
accomplish this you may have to log at the ``stderr`` and ``stdout``
that is being generated from the running *OpenStack* process (by default
they are forked as daemons). For this information check the output of
the start command for a line like
``Check * for traces of what happened``. This is usually a good starting
point, to check out those files contents and then look up the files that
contain the applications `PID`_ and ``stderr`` and ``stdout``.
First run the following to check the status of each component.
If the install section had warning messages or exceptions were thrown
there, that may also be the problem. Sometimes running the uninstall
section below will clean this up, your mileage may vary though.
::
Another tip is to edit run with more verbose logging by running with the
following ``-v`` option or the ``-vv`` option. This may give you more
insights by showing you what was executed/installed/configured
(uninstall & start by installing again to get the additional logging
output).
sudo ./smithy -a status
If you do not see all green status then you should run the following and see
if any of the ``stderr`` and ``stdout`` files will give you more information
about what is occuring
::
sudo ./smithy -a status --show
This will dump out those files (truncated to not be to verbose) so that anything
peculaliar can be seen. If nothing can be then go to the installation directory (typically ``~/openstack``)
and check the ``traces`` directory of each component and check if anything looks fishy.
Stopping
--------
@@ -364,7 +281,7 @@ the following:
::
sudo ./smithy -a stop -d ~/openstack
sudo ./smithy -a stop
You should see a set of stop actions happening and ``stderr`` and
``stdout`` and ``pid`` files being removed (if you desire more
@@ -391,7 +308,7 @@ can uninstall them by running the following:
::
sudo ./smithy -a uninstall -d ~/openstack
sudo ./smithy -a uninstall
You should see a set of packages, configuration and directories, being
removed (if you desire more informational output add a ``-v`` or a

9
docs/source/topics/goals.rst
View File

@@ -1,24 +1,19 @@
===============
Goals
Goals
===============
- To aid developers getting involved with `OpenStack`_!
- To quickly build developer `OpenStack`_ environments in a clean
environment (as well as start, stop, and uninstall those
environments) with as little baggage as possible.
- To describe working configurations of `OpenStack`_.
- Which code branches work together?
- What do config files look like for those branches?
- What packages are needed for installation for a given
distribution?
- How is a component installed/configured/ran?
- To make it easier for developers to dive into `OpenStack`_ so that
they can productively contribute without having to understand every
part of the system at once.
- To make it easy to prototype cross-project features.
- To have an installer that works!
.. _OpenStack: http://openstack.org/

4
docs/source/topics/knownissues.rst
View File

@@ -14,10 +14,6 @@ There is a script in ``tools/clear-net-ubuntu.sh`` that might help with this.
RHEL 6.2
--------
- *numpy* (for novnc) pulls in *python-nose* which we are installing
from *EPEL* and *symlinking* so that we don’t have to modify github
code directly but the *numpy* dependency can’t be installed with the
previous symlink. We are currently just using *pip* to fix this.
- Fixing up the network on uninstall doesn’t seem to be 100% correct
We might need a script like the ``tools/clear-net-ubuntu.sh`` to help in this situation.

31
docs/source/topics/qanda.rst
View File

@@ -4,11 +4,6 @@
Questions and Answers
===============
Can I use ANVIL for production?
------------------------------------
Up to u! Beware of the sea and the black waters!
How do I get program usage?
---------------------------
@@ -19,29 +14,11 @@ How do I get program usage?
How do I run a specific OpenStack milestone?
--------------------------------------------
OpenStack milestones have tags set in the git repo. Set the appropriate
setting in the **branch** variables in *conf/anvil.ini*.
OpenStack milestones have tags set in the git repo. Anvil also has the same
tags so please checkout the corresponding tag for anvil to match the OpenStack
milestone you wish to use.
**Note:** Swift is on its own release schedule so pick a tag in the
Swift repo that is just before the milestone release.
For example:
::
# Quantum client git repo
quantum_client_repo = git://github.com/openstack/python-quantumclient.git
quantum_client_branch = essex-3
# Melange service
melange_repo = git://github.com/openstack/melange.git
melange_branch = essex-3
# Python melange client library
melangeclient_repo = git://github.com/openstack/python-melangeclient.git
melangeclient_branch = essex-3
OMG the images take forever to download!
`OMG` the images take forever to download!
----------------------------------------
Sometimes the images that will be uploaded to glance take a long time to

16
docs/source/topics/solvedproblems.rst
View File

@@ -37,22 +37,6 @@ To resolve this the following seems to work:
sudo apt-get remove --purge $MYSQL_PKGS
Libguestfs not syncing
---------------------
On RHEL 6.2 people have seen files not being synced when the temporary file location is unmounted.
Getting a newer version of libguestfs seems to help. To do this follow the ``README`` at
http://people.redhat.com/~rjones/libguestfs-RHEL-6.3-preview/. Hopefully this will not be a problem
on RHEL 6.3.
Then run:
::
sudo yum -y install libguestfs* guestfs*
Horizon dead on start
---------------------

4
docs/source/topics/usageexamples.rst
View File

@@ -21,8 +21,8 @@ will do (with installation in ``~/openstack``) try:
$ sudo ./smithy -d ~/openstack -a install --dryrun
With more information/debugging/auditing output try:
With more information via debug statements output try:
::
$ sudo ./smithy -d ~/openstack -a install -vv
$ sudo ./smithy -d ~/openstack -a install -v

10
docs/source/topics/warning.rst
View File

@@ -2,11 +2,13 @@
Important!
==========
**Warning:** Be sure to carefully read ``smithy`` and any other scripts
Warning!
==========
Be sure to carefully read ``smithy`` and any other scripts
you execute before you run them, as they install software and may alter
your networking configuration. We strongly recommend that you run
``smithy`` in a clean and disposable virtual machine when you are first
getting started.
your networking configuration.
.. _epel: http://fedoraproject.org/wiki/EPEL
.. _forking: http://users.telenet.be/bartl/classicperl/fork/all.html