Fuel Development Environment ============================ .. warning:: Fuel ISO build works only on 64-bit operating system. If you are modifying or augmenting the Fuel source code or if you need to build a Fuel ISO from the latest branch, you will need an environment with the necessary packages installed. This page lays out the steps you will need to follow in order to prepare the development environment, test the individual components of Fuel, and build the ISO which will be used to deploy your Fuel master node. The basic operating system for Fuel development is Ubuntu Linux. The setup instructions below assume Ubuntu 14.04 (64 bit) though most of them should be applicable to other Ubuntu and Debian versions, too. Each subsequent section below assumes that you have followed the steps described in all preceding sections. By the end of this document, you should be able to run and test all key components of Fuel, build the Fuel master node installation ISO, and generate documentation. .. _getting-source: Getting the Source Code ----------------------- Source code of OpenStack Fuel can be found at git.openstack.org or GitHub. Follow these steps to clone the repositories for each of the Fuel components: :: apt-get install git git clone https://github.com/openstack/fuel-main git clone https://github.com/openstack/fuel-web git clone https://github.com/openstack/fuel-ui git clone https://github.com/openstack/fuel-agent git clone https://github.com/openstack/fuel-astute git clone https://github.com/openstack/fuel-ostf git clone https://github.com/openstack/fuel-library git clone https://github.com/openstack/fuel-docs .. _building-fuel-iso: Building the Fuel ISO --------------------- The "fuel-main" repository is the only one required in order to build the Fuel ISO. The make script then downloads the additional components (Fuel Library, Nailgun, Astute and OSTF). Unless otherwise specified in the makefile, the master branch of each respective repo is used to build the ISO. The basic steps to build the Fuel ISO from trunk in an Ubuntu 14.04 environment are: :: apt-get install git git clone https://github.com/openstack/fuel-main cd fuel-main ./prepare-build-env.sh make iso If you want to build an ISO using a specific commit or repository, you will need to modify the "Repos and versions" section in the config.mk file found in the fuel-main repo before executing "make iso". For example, this would build a Fuel ISO against the v5.0 tag of Fuel: :: # Repos and versions FUELLIB_COMMIT?=tags/5.0 NAILGUN_COMMIT?=tags/5.0 FUEL_UI_COMMIT?=tags/5.0 ASTUTE_COMMIT?=tags/5.0 OSTF_COMMIT?=tags/5.0 FUELLIB_REPO?=https://github.com/openstack/fuel-library.git NAILGUN_REPO?=https://github.com/openstack/fuel-web.git FUEL_UI_REPO?=https://github.com/openstack/fuel-ui.git ASTUTE_REPO?=https://github.com/openstack/fuel-astute.git OSTF_REPO?=https://github.com/openstack/fuel-ostf.git To build an ISO image from custom gerrit patches on review, edit the "Gerrit URLs and commits" section of config.mk, e.g. for https://review.openstack.org/#/c/63732/8 (id:63732, patch:8) set: :: FUELLIB_GERRIT_COMMIT?=refs/changes/32/63732/8 If you are building Fuel from an older branch that does not contain the "prepare-build-env.sh" script, you can follow these steps to prepare your Fuel ISO build environment on Ubuntu 14.04: #. ISO build process requires sudo permissions, allow yourself to run commands as root user without request for a password:: echo "`whoami` ALL=(ALL) NOPASSWD: ALL" | sudo tee -a /etc/sudoers #. Install software:: sudo apt-get update sudo apt-get install apt-transport-https echo deb http://mirror.yandex.ru/mirrors/docker/ docker main | sudo tee /etc/apt/sources.list.d/docker.list sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 sudo apt-get update sudo apt-get install lxc-docker sudo apt-get update sudo apt-get remove nodejs nodejs-legacy npm sudo apt-get install software-properties-common python-software-properties sudo add-apt-repository -y ppa:chris-lea/node.js sudo apt-get update sudo apt-get install build-essential make git ruby ruby-dev rubygems debootstrap createrepo \ python-setuptools yum yum-utils libmysqlclient-dev isomd5sum \ python-nose libvirt-bin python-ipaddr python-paramiko python-yaml \ python-pip kpartx extlinux unzip genisoimage nodejs multistrap \ lrzip python-daemon sudo gem install bundler -v 1.2.1 sudo gem install builder sudo pip install xmlbuilder jinja2 #. If you haven't already done so, get the source code:: git clone https://github.com/openstack/fuel-main #. Now you can build the Fuel ISO image:: cd fuel-main make iso #. If you encounter issues and need to rebase or start over:: make clean #remove build/ directory make deep_clean #remove build/ and local_mirror/ .. note:: In case you use a virtual machine for building the image, verify the following: - Both ``BUILD_DIR`` and ``LOCAL_MIRROR`` build directories are out of the shared folder path in the `config.mk `_ file. For more information, see: - `Shared folders of VitrualBox `_ documentation - `Synced folders of Vargant `_ documentation - To prevent a random docker unexpected termination, a virtual machine has a kernel that supports the ``aufs`` file system. To install the kernel, run: .. code-block:: console sudo apt-get install --yes linux-image-extra-virtual Reboot the kernel when the installation is complete. Check that docker is using ``aufs`` by running: .. code-block:: console sudo docker info 2>&1 | grep -q 'Storage Driver: aufs' \ && echo OK || echo KO For more information, see `Select a storage driver for docker `_. You can also use the following tools to make your work and development process with Fuel easier: * CGenie fuel-utils - a set of tools to interact with code on a Fuel Master node created from the ISO. It provides the *fuel* command that gives a simple way to upload Python or UI code (with staticfiles compression) to Docker containers, SSH into machine and into the container, display the logs etc. * Vagrant SaltStack-based -Vagrant box definition with quick and basic Fuel environment with fake tasks. This is useful for UI or Nailgun development. You can download both tools from the `fuel-dev-tools `_. Nailgun (Fuel-Web) ------------------ Nailgun is the heart of Fuel project. It implements a REST API as well as deployment data management. It manages disk volume configuration data, network configuration data and any other environment specific data necessary for a successful deployment of OpenStack. It provides the required orchestration logic for provisioning and deployment of the OpenStack components and nodes in the right order. Nailgun uses a SQL database to store its data and an AMQP service to interact with workers. Requirements for preparing the nailgun development environment, along with information on how to modify and test nailgun can be found in the Nailgun Development Instructions document: :ref:`nailgun-development` Astute ------ Astute is the Fuel component that represents Nailgun's workers, and its function is to run actions according to the instructions provided from Nailgun. Astute provides a layer which encapsulates all the details about interaction with a variety of services such as Cobbler, Puppet, shell scripts, etc. and provides a universal asynchronous interface to those services. #. Astute can be found in fuel-astute repository #. Install Ruby dependencies:: sudo apt-get install git curl curl -sSL https://get.rvm.io | bash -s stable source ~/.rvm/scripts/rvm rvm install 2.1 rvm use 2.1 git clone https://github.com/nulayer/raemon.git cd raemon git checkout b78eaae57c8e836b8018386dd96527b8d9971acc gem build raemon.gemspec gem install raemon-0.3.0.gem cd .. rm -Rf raemon #. Install or update dependencies and run unit tests:: cd fuel-astute ./run_tests.sh #. (optional) Run Astute MCollective integration test (you'll need to have MCollective server running for this to work):: cd fuel-astute bundle exec rspec spec/integration/mcollective_spec.rb Running Fuel Puppet Modules Unit Tests -------------------------------------- If you are modifying any puppet modules used by Fuel, or including additional modules, you can use the PuppetLabs RSpec Helper to run the unit tests for any individual puppet module. Follow these steps to install the RSpec Helper: #. Install PuppetLabs RSpec Helper:: cd ~ gem2deb puppetlabs_spec_helper sudo dpkg -i ruby-puppetlabs-spec-helper_0.4.1-1_all.deb gem2deb rspec-puppet sudo dpkg -i ruby-rspec-puppet_0.1.6-1_all.deb #. Run unit tests for a Puppet module:: cd fuel/deployment/puppet/module rake spec Installing Cobbler ------------------ Install Cobbler from GitHub (it can't be installed from PyPi, and deb package in Ubuntu is outdated):: cd ~ git clone git://github.com/cobbler/cobbler.git cd cobbler git checkout release24 sudo make install Building Documentation ---------------------- You should prepare your build environment before you can build this documentation. First you must install Java, using the appropriate procedure for your operating system. Java is needed to use PlantUML to automatically generate UML diagrams from the source. You can also use `PlantUML Server `_ for a quick preview of your diagrams and language documentation. Then you need to install all the packages required for creating of the Python virtual environment and dependencies installation. :: sudo apt-get install make postgresql postgresql-server-dev-9.1 sudo apt-get install python-dev python-pip python-virtualenv Now you can create the virtual environment and activate it. :: virtualenv fuel-web-venv . fuel-web-venv/bin/activate And then install the dependencies. :: pip install -r nailgun/test-requirements.txt Now you can look at the list of available formats and generate the one you need: :: cd docs make help make html There is a helper script **build-docs.sh**. It can perform all the required steps automatically. The script can build documentation in required format. :: Documentation build helper -o - Open generated documentation after build -c - Clear the build directory -n - Don't install any packages -f - Documentation format [html,signlehtml,latexpdf,pdf,epub] For example, if you want to build HTML documentation you can just use the following script, like this: :: ./build-docs.sh -f html -o It will create virtualenv, install the required dependencies and build the documentation in HTML format. It will also open the documentation with your default browser. If you don't want to install all the dependencies and you are not interested in building automatic API documentation there is an easy way to do it. First remove autodoc modules from extensions section of **conf.py** file in the **docs** directory. This section should be like this: :: extensions = [ 'rst2pdf.pdfbuilder', 'sphinxcontrib.plantuml', ] Then remove **develop/api_doc.rst** file and reference to it from **develop.rst** index. Now you can build documentation as usual using make command. This method can be useful if you want to make some corrections to text and see the results without building the entire environment. The only Python packages you need are Sphinx packages: :: Sphinx sphinxcontrib-actdiag sphinxcontrib-blockdiag sphinxcontrib-nwdiag sphinxcontrib-plantuml sphinxcontrib-seqdiag Just don't forget to rollback all these changes before you commit your corrections.