openstack/fuel-web
Code Issues Proposed changes
11c76d3010
fuel-web/docs/buildsystem.rst
irina povolotskaya 9d94d8a610 Describes Fuel build system 
Co-Authored-By: Vladimir Kozhukalov <vkozhukalov@mirantis.com>

Change-Id: I434960e8ae7707608e4c1b1235300fcb2db26596
Closes-Bug: 1365441
2014-12-18 13:52:14 +03:00

11 KiB
Raw Blame History

Fuel ISO build system

Use the fuel-main repository to build Fuel components such as an ISO or an upgrade tarball. This repository contains a set of GNU Make build scripts.

Quick start

  1. You must use one of the following distributions to build Fuel components or the build process may fail. Note that build only works for x64 platforms.
  • Ubuntu 12.04
  • Ubuntu 14.04
  1. Check whether you have git installed in your system. To do that, use the following command:
which git

If git is not found, install it with the following command:

:

apt-get install git
  1. Clone the fuel-main git repository to the location where you will work. The root of your repo will be named fuel-main. In this example, it will be located under the ~/fuel directory:
mkdir ~/fuel
cd ~/fuel
git clone https://github.com/stackforge/fuel-main.git
cd fuel-main

Note

Fuel build system consists of the following components:

  • a shell script (./prepare-build-env.sh) - prepares the build environment by checking

    that all necessary packages are installed and installing any that are not.

  • fuel-main directory - the only one required repository for building the Fuel ISO.

The make script then downloads the additional components (Fuel Library, Nailgun, Astute and OSTF). Unless otherwise specified in the makefile, the master branch of each respective repo is used to build the ISO.

  1. Run the shell script:
./prepare-build-env.sh

and wait until **prepare-build-env.sh**
installs the Fuel build evironment on your computer.
  1. After the script runs successfully, issue the following command to build a Fuel ISO:
make iso
  1. Use the following command to list the available make targets:
make help

For the full list of available targets with description, see Build targets <build-targets> section below.

Build system structure

Fuel consists of several components such as web interface, puppet modules, orchestration components, testing components. Source code of all those components is split into multiple git repositories like:

The main component of the Fuel build system is fuel-main directory.

Fuel build processes are quite complicated, so to make the fuel-main code easily maintainable, it is split into a bunch of files and directories.

Those files and directories contain independent (or at least almost independent) pieces of Fuel build system:

  • Makefile - the main Makefile which includes all other make modules.
  • config.mk - contains parameters used to customize the build process, specifying items such as build paths, upstream mirrors, source code repositories and branches, built-in default Fuel settings and ISO name.
  • rules.mk - defines frequently used macros.
  • repos.mk - contains make scripts to download the other repositories to develop Fuel components put into separate repos.
  • sandbox.mk - shell script definitions that create and destroy the special chroot environment required to build some components. For example, for building RPM packages, CentOS images we use CentOS chroot environment.
  • mirror - contains the code which is used to download all necessary packages from upstream mirrors and build new ones which are to be copied on Fuel ISO even if Internet connection is down.
  • puppet - contains the code used to pack Fuel puppet modules into a tarball that is afterwards put on Fuel ISO.
  • packages - contains DEB and RPM specs as well as make code for building those packages, included in Fuel DEB and RPM mirrors.
  • bootstrap - contains a make script intended to build CentOS-based miniroot image (a.k.a initrd or initramfs).
  • image - contains make scripts for building CentOS and Ubuntu images using the Fuel mirrors, built from the scripts in the mirror directory. The images are alternative to using the standard anaconda and debian installers.
  • docker - contains the make scripts to build docker containers, deployed on the Fuel Master node.
  • upgrade - contains make scripts for building Fuel upgrade tarball.
  • iso - contains make scripts for building Fuel ISO file.

Fuel-main also contains a set of directories which are not directly related to Fuel build processes:

  • virtualbox - contains a set of shell scripts which allow one to deploy Fuel demo lab easily using VirtualBox.
  • utils - contains a set of utilities used for maintaining Fuel components.
  • fuelweb_test and fuelweb_ui_test - contain the code of Fuel system tests.

Build targets

  • all - used for building all Fuel artifacts. Currently, it is an alias for iso target.
  • bootstrap - used for building in-memory bootstrap image which is used for auto-discovering.
  • mirror - used for building local mirrors (the copies of CentOS and Ubuntu mirrors which are then placed into Fuel ISO). They contain all necessary packages including those listed in requirements-.txt* files with their dependencies as well as those which are Fuel packages. Packages listed in requirements-.txt* files are downloaded from upstream mirrors while Fuel packages are built from source code.
  • iso - used for building Fuel ISO. If build succeeds, ISO is put into build/artifacts folder.
  • img - used for building Fuel flash stick image, binary copied to a flash stick. That stick is then used as a bootable device and contains Fuel ISO as well as some auxiliary boot files.
  • clean - removes build directory.
  • deep_clean - removes build directory and local mirror. Note that if you remove a local mirror, then next time the ISO build job will download all necessary packages again. So, the process goes faster if you keep local mirrors. You should also mind the following: it is better to run make deep_clean every time when building an ISO to make sure the local mirror is consistent.

Customizing build process

There are plenty of variables in make files. Some of them represent a kind of build parameters. They are defined in config.mk file:

  • TOP_DIR - a default current directory. All other build directories are relative to this path.
  • BUILD_DIR - contains all files, used during build process. By default, it is $(TOP_DIR)/build.
  • ARTS_DIR - contains build artifacts such as ISO and IMG files By default, it is $(BUILD_DIR)/artifacts.
  • LOCAL_MIRROR - contains local CentOS and Ubuntu mirrors By default, it is $(TOP_DIR)/local_mirror.
  • DEPS_DIR - contains build targets that depend on artifacts of the previous build jobs, placed there before build starts. By default, it is $(TOP_DIR)/deps.
  • ISO_NAME - a name of Fuel ISO without file extension: if ISO_NAME = MY_CUSTOM_NAME, then Fuel ISO file will be placed into $(MY_CUSTOM_NAME).iso.
  • ISO_PATH - used to specify Fuel ISO full path instead of defining just ISO name. By default, it is $(ARTS_DIR)/$(ISO_NAME).iso.
  • UPGRADE_TARBALL_NAME - defines the name of upgrade tarball. By default, it is $(UPGRADE_TARBALL_NAME).tar.
  • UPGRADE_TARBALL_PATH - used to define full upgrade tarball path. By default, it is $(ARTS_DIR)/$(UPGRADE_TARBALL_NAME).tar.
  • VBOX_SCRIPTS_NAME - defines the name of the archive with VirtualBox scripts. By default, it is placed into $(VBOX_SCRIPTS_NAME).zip.
  • VBOX_SCRIPTS_PATH - defines full path for VirtualBox scripts archive. By default, it is $(ARTS_DIR)/$(VBOX_SCRIPTS_NAME).zip
  • Fuel ISO contains some default settings for the Fuel Master node. These settings can be customized during Fuel Master node installation. One can customize those settings using the following variables:
  • MASTER_IP - the Fuel Master node IP address. By default, it is 10.20.0.2.
  • MASTER_NETMASK - Fuel Master node IP netmask. By default, it is 255.255.255.0.
  • MASTER_GW - Fuel Master node default gateway. By default, it is is 10.20.0.1.
  • MASTER_DNS - the upstream DNS location for the Fuel master node. FUel Master node DNS will redirect there all DNS requests that it is not able to resolve itself. By default, it is 10.20.0.1.

Other options

  • BUILD_OPENSTACK_PACKAGES - list of Openstack packages to be rebuilt from source.

  • [repo]_REPO - remote source code repo. URL or git repository can be specified for each of the Fuel components. (FUELLIB, NAILGUN, ASTUTE, OSTF).

  • [repo]_COMMIT - source branch for each of the Fuel components to build.

  • [repo]_GERRIT_URL - gerrit repo.

  • [repo]_GERRIT_COMMIT - list of extra commits from gerrit.

  • [repo]_SPEC_REPO - repo for RPM/DEB specs of OpenStack packages.

  • [repo]_SPEC_COMMIT - branch for checkout.

  • [repo]_SPEC_GERRIT_URL - gerrit repo for OpenStack specs.

  • [repo]_SPEC_GERRIT_COMMIT - list of extra commits from gerrit for specs.

  • USE_MIRROR - pre-built mirrors from Fuel infrastructure. The following mirrors can be used:

    • ext (external mirror, available from outside of Mirantis network)

    * none (reserved for building local mirrors: in this case CentOS and Ubuntu packages will be fetched from upstream mirrors, so that it will make the build process much slower).

  • MIRROR_CENTOS - download CentOS packages from a specific remote repo.

  • MIRROR_UBUNTU - download Ubuntu packages from a specific remote repo.

  • MIRROR_DOCKER - download docker images from a specific remote url.

  • EXTRA_RPM_REPOS - extra repos with RPM packages. Each repo must be comma separated tuple with repo-name and repo-path: <first_repo_name>,<repo_path> <second_repo_name>,<second_repo_path> For example, qemu2,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-15943/centos/ libvirt,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17019/centos/.

  • EXTRA_DEP_REPOS - extra repos with DEB packages. Each repo must consist of an url, distro and section parts. Repos must be separated by bar: <first_repo_path>|<second_repo_path> For example, http://fuel-repository.mirantis.com/repos/ubuntu-fuel-5.1-stable-15955/ubuntu/|http://fuel-repository.mirantis.com/repos/ubuntu-fuel-5.1-stable-15953/ubuntu/.

** FEATURE_GROUPS - options for the ISO.

Combination of the following:

  • mirantis (use mirantis logos and logic)
  • experimental (allow experimental features on Fuel web UI)

Note that if you want to add more packages to the Fuel Master node, you should update the requirements-rpm.txt and the requirements-deb.txt files.