* Fuel-Devops is now separated to Version 2.5.x and 2.9.x. * Installation can be implemented via PyPI in Python virtual env. Change-Id: I6083964a51b219c2dcffd517d4d80b7f3288f22f Closes-bug: #1430314
12 KiB
Devops Guide
Introduction
Fuel-Devops is a sublayer between application and target environment (currently only supported under libvirt).
This application is used for testing purposes like grouping virtual machines to environments, booting KVM VMs locally from the ISO image and over the network via PXE, creating, snapshotting and resuming back the whole environment in single action, create virtual machines with multiple NICs, multiple hard drives and many other customizations with a few lines of code in system tests.
After 6.0 release, fuel-devops was divided into 2.5.x and 2.9.x versions. Two separate versions of fuel-devops provide backward compatibility for system tests which have been refactored since the last major release. How to migrate
For sources please refer to fuel-devops repository on github.
Installation
The installation procedure can be implemented via PyPI in Python virtual environment (suppose you are using Ubuntu 12.04 or Ubuntu 14.04):
Before using it, please install the following required dependencies:
sudo apt-get install git \
\
postgresql \
postgresql-server-dev-all \
libyaml-dev \
libffi-dev \
python-dev \
python-libvirt \
python-pip \
qemu-kvm \
qemu-utils \
libvirt-bin \
libvirt-dev \
ubuntu-vm-builder
bridge-utils
sudo apt-get update && sudo apt-get upgrade -y
Devops installation in virtualenv
- Install packages needed for building python eggs
sudo apt-get install python-virtualenv libpq-dev libgmp-dev
- In case you are using Ubuntu 12.04 let's update pip and virtualenv, otherwise you can skip this step
sudo pip install pip virtualenv --upgrade
hash -r
- Create virtualenv for the devops project
virtualenv --system-site-packages <path>/fuel-devops-venv
Note
If you want to use different devops versions in the same time, you can create several different folders for each version, and then activate required virtual environment for each case. For example: :: virtualenv --system-site-packages <path>/fuel-devops-venv # For fuel-devops 2.5.x virtualenv --system-site-packages <path>/fuel-devops-venv-2.9 # For fuel-devops 2.9.x
<path> represents the path where your Python virtualenv will be located. (e.g. ~/venv). If it is not specified, it will use the current working directory.
- Activate virtualenv and install devops package using PyPI.
source <path>/fuel-devops-venv/bin/activate
pip install git+https://github.com/stackforge/fuel-devops.git@<version> --upgrade
where <version> is the specific version of fuel-devops you would like to install. For Fuel 6.0 and earlier, take the latest fuel-devops 2.5.x. For Fuel 6.1 and later, use 2.9.x or newer. See more information on the latest available versions in fuel-devops repo.
setup.py in fuel-devops repository does everything required.
Hint
You can also use virtualenvwrapper which can help you manage virtual environments
- Next, follow
DevOpsConf
section
Configuration
Basically devops requires that the following system-wide settings are configured:
- Default libvirt storage pool is active (called 'default')
- Current user must have permission to run KVM VMs with libvirt
- PostgreSQL server running with appropriate grants and schema for devops
- [Optional] Nested Paging is enabled
Configuring libvirt pool
Create libvirt's pool
sudo virsh pool-define-as --type=dir --name=default --target=/var/lib/libvirt/images
sudo virsh pool-autostart default
sudo virsh pool-start default
Permissions to run KVM VMs with libvirt with current user
Give current user permissions to use libvirt (Do not forget to log out and log back in!)
sudo usermod $(whoami) -a -G libvirtd,sudo
Configuring Postgresql database
Set local peers to be trusted by default, create user and db and load fixtures.
sudo sed -ir 's/peer/trust/' /etc/postgresql/9.*/main/pg_hba.conf
sudo service postgresql restart
sudo -u postgres createuser -P <user> # see default <user> and <db> below
sudo -u postgres createdb <db> -O <user>
django-admin.py syncdb --settings=devops.settings
django-admin.py migrate devops --settings=devops.settings
- in 2.5.x version, default <user> and <db> are devops
- in 2.9.x version, default <user> and <db> are fuel_devops
Note
Depending on your Linux distribution, django-admin may refer to system-wide django installed from package. If this happens you could get an exception that says that devops.settings module is not resolvable. To fix this, run django-admin.py (or django-admin) with a relative path :
./bin/django-admin syncdb --settings=devops.settings
./bin/django-admin migrate devops --settings=devops.settings
[Optional] Enabling Nested Paging
This option is enabled by default in the KVM kernel module
$ cat /etc/modprobe.d/qemu-system-x86.conf
options kvm_intel nested=1
In order to be sure that this feature is enabled on your system, please run:
sudo kvm-ok && cat /sys/module/kvm_intel/parameters/nested
The result should be:
INFO: /dev/kvm exists
KVM acceleration can be used
Y
Environment creation via Devops + Fuel_QA or Fuel_main
Depending on the Fuel release, you may need a different repository. In case of 6.0 or earlier, please use fuel-main repository. For 6.1 and later, the fuel-qa is required.
- Clone GIT repository
git clone https://github.com/stackforge/fuel-qa # fuel-main for 6.0 and earlier
cd fuel-qa/
- Install requirements
source <path>/fuel-devops-venv/bin/activate
pip install -r ./fuelweb_test/requirements.txt --upgrade
- Check
DevOpsConf
section - Prepare environment
Download Fuel ISO from Nightly builds or
build it yourself (please, refer to building-fuel-iso
)
Next, you need to define several variables for the future environment
export ISO_PATH=<path_to_iso>
export NODES_COUNT=<number_nodes>
export ENV_NAME=<name_of_env>
export VENV_PATH=<path>/fuel-devops-venv
Alternatively, you can edit this file to set them as a default values
fuelweb_test/settings.py
Start tests by running this command
./utils/jenkins/system_tests.sh -t test -w $(pwd) -j fuelweb_test -i $ISO_PATH -o --group=setup
For more information about how tests work, read the usage information
./utils/jenkins/system_tests.sh -h
Important notes for Sahara and Murano tests
- It is not recommended to start tests without KVM.
- For the best performance Put Sahara image savanna-0.3-vanilla-1.2.1-ubuntu-13.04.qcow2 (md5: 9ab37ec9a13bb005639331c4275a308d) in /tmp/ before start, otherwise (If Internet access is available) the image will download automatically.
- Put Murano image ubuntu-murano-agent.qcow2 (md5: b0a0fdc0b4a8833f79701eb25e6807a3) in /tmp before start.
- Running Murano tests on instances without an Internet connection will fail.
- For Murano tests execute 'export SLAVE_NODE_MEMORY=5120' before starting.
- Heat autoscale tests require the image F17-x86_64-cfntools.qcow2 (md5: afab0f79bac770d61d24b4d0560b5f70) be placed in /tmp before starting.
Run single OSTF tests several times
- Export environment variable OSTF_TEST_NAME. Example: export OSTF_TEST_NAME='Request list of networks'
- Export environment variable OSTF_TEST_RETRIES_COUNT. Example: export OSTF_TEST_RETRIES_COUNT=120
- Execute test_ostf_repetable_tests from tests_strength package
Run tests :
sh "utils/jenkins/system_tests.sh" -t test \
-w $(pwd) \
-j "fuelweb_test" \
-i "$ISO_PATH" \
-V $(pwd)/venv/fuelweb_test \
-o \
--group=create_delete_ip_n_times_nova_flat
Upgrade from system-wide devops to devops in Python virtual environment
To migrate from older devops, follow these steps:
- Remove system-wide fuel-devops (e.g. python-devops)
You must remove system-wide fuel-devops and switch to separate venvs with different versions of fuel-devops, for Fuel 6.0.x (and older) and 6.1 release.
Repositories 'fuel-main' and 'fuel-qa', that contain system tests, must use different Python virtual environments, for example:
- ~/venv-nailgun-tests - used for 6.0.x and older releases. Contains version 2.5.x of fuel-devops
- ~/venv-nailgun-tests-2.9 - used for 6.1 and above. Contains version 2.9.x of fuel-devops
If you have scripts which use system fuel-devops, fix them, and activate Python venv before you start working in your devops environment.
By default, the network pool is configured as follows:
- 10.108.0.0/16 for devops 2.5.x
- 10.109.0.0/16 for 2.9.x
Please check other settings in devops.settings, especially the connection settings to the database.
Before using devops in Python venv, you need to install system dependencies
- Update fuel-devops and Python venv on CI servers
To update fuel-devops, you can use the following examples:
# DevOps 2.5.x for system tests from 'fuel-main' repository
if [ -f ~/venv-nailgun-tests/bin/activate ]; then
echo "Python virtual env exist"
else
rm -rf ~/venv-nailgun-tests
virtualenv --system-site-packages ~/venv-nailgun-tests
fi
source ~/venv-nailgun-tests/bin/activate
pip install -r https://raw.githubusercontent.com/stackforge/fuel-main/master/fuelweb_test/requirements.txt --upgrade
django-admin.py syncdb --settings=devops.settings --noinput
django-admin.py migrate devops --settings=devops.settings --noinput
deactivate
# DevOps 2.9.x for system tests from 'fuel-qa' repository
if [ -f ~/venv-nailgun-tests-2.9/bin/activate ]; then
echo "Python virtual env exist"
else
rm -rf ~/venv-nailgun-tests-2.9
virtualenv --system-site-packages ~/venv-nailgun-tests-2.9
fi
source ~/venv-nailgun-tests-2.9/bin/activate
pip install -r https://raw.githubusercontent.com/stackforge/fuel-qa/master/fuelweb_test/requirements.txt --upgrade
django-admin.py syncdb --settings=devops.settings --noinput
django-admin.py migrate devops --settings=devops.settings --noinput
deactivate
- Setup new repository of system tests for 6.1 release
All system tests for 6.1 and higher were moved to fuel-qa repo.
To upgrade 6.1 jobs, follow these steps:
- make a separate Python venv, for example in ~/venv-nailgun-tests-2.9
- install requirements of system tests
- if you are using system tests on CI, please configure your CI to use new Python venv, or export path to the new Python venv in the variable VENV_PATH: export VENV_PATH=<path>/fuel-devops-venv-2.9