fuel-web/docs/devops.rst

Devops Guide

Introduction

Fuel-Devops is a sublayer between application and target environment (currently only supported under libvirt).

This application is used for testing purposes like grouping virtual machines to environments, booting KVM VMs locally from the ISO image and over the network via PXE, creating, snapshotting and resuming back the whole environment in single action, create virtual machines with multiple NICs, multiple hard drives and many other customizations with a few lines of code in system tests.

For sources please refer to fuel-devops repository on github.

Installation

The installation procedure can be implemented in two different ways (suppose you are using Ubuntu 12.04 or Ubuntu 14.04):

• from deb packages <DevOpsApt> (using apt)
• from python packages <DevOpsPyPI> (using PyPI) (also in virtualenv <DevOpsPyPIvenv>)

Each of the above approaches is described in detail below.

Devops installation from packages

1. Checking and applying latest updates

sudo apt-get update && sudo apt-get upgrade -y

2. Installing dependencies

sudo apt-get install postgresql \
git \
python-pip \
python-psycopg2 \
python-ipaddr \
python-libvirt \
python-paramiko \
python-django \
python-django-south \
python-ipaddr \
python-yaml \
python-mock

Note

Depending on your Linux distribution some of the above packages may not exists in your upstream repositories. In this case, please exclude them from the installation list and repeat step 2. Missing packages will be installed by python (from PyPI) during the next step

3. Clone fuel-devops repo and run setup.py

git clone git://github.com/stackforge/fuel-devops.git
cd fuel-devops
sudo python ./setup.py install

4. Now we can check that devops has been installed successfully

user@host:~$ pip list

apt-xapian-index (0.45)
argparse (1.2.1)
chardet (2.0.1)
colorama (0.2.5)
devops (2.5.2)
Django (1.6.1)
html5lib (0.999)
ipaddr (2.1.10)
libvirt-python (1.2.2)
mock (1.0.1)
paramiko (1.10.1)
pip (1.5.4)
psycopg2 (2.4.5)
pycrypto (2.6.1)
python-apt (0.9.3.5)
python-debian (0.1.21-nmu2ubuntu2)
PyYAML (3.10)
requests (2.2.1)
setuptools (3.3)
six (1.5.2)
South (0.7.5)
ssh-import-id (3.21)
urllib3 (1.7.1)
wsgiref (0.1.2)
xmlbuilder (1.0)

Note

In case of Ubuntu 12.04 LTS we need to update pip and Django<1.7:

sudo pip install pip --upgrade
hash -r
sudo pip install Django\<1.7 --upgrade

5. Next, follow DevOpsConf section

Devops installation using PyPI

The installation procedure should be implemented by following the next steps:

1. Checking and applying latest updates

sudo apt-get update && sudo apt-get upgrade -y

2. Install packages needed for building python eggs and working with devops (postgresql, git)

sudo apt-get install git \
postgresql \
python-dev \
python-pip \
python-libvirt \
libyaml-dev \
libpq-dev \
libgmp-dev

3. In case you are using Ubuntu 12.04 let's update pip, otherwise you can skip this step

sudo pip install pip --upgrade
hash -r

4. Install devops package using python setup tools. Clone fuel-devops and run setup.py

git clone git://github.com/stackforge/fuel-devops.git
cd fuel-devops
sudo python ./setup.py install

5. Next, follow DevOpsConf section

Devops installation in virtualenv

Installation procedure is the same as in the case of DevOpsPyPI, but we should also configure virtualenv

1. Checking and applying latest updates

sudo apt-get update && sudo apt-get upgrade -y

2. Install packages needed for building python eggs and working with devops (postgresql, git, virtualenv)

sudo apt-get install git \
postgresql \
python-dev \
python-pip \
python-libvirt \
python-virtualenv \
libyaml-dev \
libpq-dev \
libgmp-dev

3. In case you are using Ubuntu 12.04 let's update pip and virtualenv, otherwise you can skip this step

sudo pip install pip virtualenv --upgrade
hash -r

4. Create virtualenv for the devops project

virtualenv --system-site-packages devops-venv

5. Activate virtualenv and install devops package using python setup tools

. devops-venv/bin/activate
pip install git+https://github.com/stackforge/fuel-devops.git --upgrade

setup.py in fuel-devops repository does everything required.

Hint

You can also use virtualenvwrapper which can help you manage virtual environments

6. Next, follow DevOpsConf section

Configuration

Basically devops requires that the following system-wide settings are configured:

• Default libvirt storage pool is active (called 'default')
• Current user must have permission to run KVM VMs with libvirt
• PostgreSQL server running with appropriate grants and schema for devops
• [Optional] Nested Paging is enabled

Configuring libvirt pool

Create libvirt's pool

sudo virsh pool-define-as --type=dir --name=default --target=/var/lib/libvirt/images
sudo virsh pool-autostart default
sudo virsh pool-start default

Permissions to run KVM VMs with libvirt with current user

Give current user permissions to use libvirt

sudo usermod $(whoami) -a -G libvirtd,sudo

Configuring Postgresql database

Set local peers to be trusted by default and load fixtures

sudo sed -ir 's/peer/trust/' /etc/postgresql/9.*/main/pg_hba.conf
sudo service postgresql restart
django-admin syncdb --settings=devops.settings
django-admin migrate devops --settings=devops.settings

Note

Depending on your Linux distribution, django-admin may refer to system-wide django installed from package. If this happens you could get an exception that says that devops.settings module is not resolvable. To fix this, run django-admin.py (or django-admin) with a relative path :

./bin/django-admin syncdb --settings=devops.settings
./bin/django-admin migrate devops --settings=devops.settings

[Optional] Enabling Nested Paging

This option is enabled by default in the KVM kernel module

$ cat /etc/modprobe.d/qemu-system-x86.conf
options kvm_intel nested=1

In order to be sure that this feature is enabled on your system, please run:

sudo kvm-ok && cat /sys/module/kvm_intel/parameters/nested

The result should be:

INFO: /dev/kvm exists
KVM acceleration can be used
Y

Environment creation via Devops + Fuel_main

1. Install basic packages

sudo apt-get install -y \
git \
libxslt1-dev \
libffi-dev \
libyaml-dev \
python-virtualenv \
python-dev \
postgresql \
postgresql-server-dev-all

2. Create virtualenv for the devops project

virtualenv --system-site-packages devops-venv
. devops-venv/bin/activate

3. Clone fuel-main

git clone https://github.com/stackforge/fuel-main
cd fuel-main/

4. Install requirements

pip install -r ./fuelweb_test/requirements.txt --upgrade

5. Check DevOpsConf section
6. Prepare environment

Download Fuel ISO from Nightly builds or build it yourself (please, refer to building-fuel-iso)

Next, you need to define several variables for the future environment

export ISO_PATH=<path_to_iso>
export NODES_COUNT=<number_nodes>
export ENV_NAME=<name_of_env>
export VENV_PATH=<path_to_virtualenv>

Alternatively, you can edit this file to set them as a default values

fuelweb_test/settings.py

Start tests by running this command

export PYTHONPATH=$(pwd)
./utils/jenkins/system_tests.sh -t test -w $(pwd) -j fuelweb_test -i $ISO_PATH -o --group=setup

For more information about how tests work, read the usage information

./utils/jenkins/system_tests.sh -h

Important notes for Sahara and Murano tests

• It is not recommended to start tests without KVM.
• For the best performance Put Sahara image savanna-0.3-vanilla-1.2.1-ubuntu-13.04.qcow2 (md5: 9ab37ec9a13bb005639331c4275a308d) in /tmp/ before start, otherwise (If Internet access is available) the image will download automatically.
• Put Murano image ubuntu-murano-agent.qcow2 (md5: b0a0fdc0b4a8833f79701eb25e6807a3) in /tmp before start.
• Running Murano tests on instances without an Internet connection will fail.
• For Murano tests execute 'export SLAVE_NODE_MEMORY=5120' before starting.
• Heat autoscale tests require the image F17-x86_64-cfntools.qcow2 (md5: afab0f79bac770d61d24b4d0560b5f70) be placed in /tmp before starting.

Run single OSTF tests several times

• Export environment variable OSTF_TEST_NAME. Example: export OSTF_TEST_NAME='Request list of networks'
• Export environment variable OSTF_TEST_RETRIES_COUNT. Example: export OSTF_TEST_RETRIES_COUNT=120
• Execute test_ostf_repetable_tests from tests_strength package

Run tests :

sh "utils/jenkins/system_tests.sh" -t test \
     -w $(pwd) \
     -j "fuelweb_test" \
     -i "$ISO_PATH" \
     -V $(pwd)/venv/fuelweb_test \
     -o \
     --group=create_delete_ip_n_times_nova_flat