[fuel-web] Perestroika build system description
This patch adds a new sub-section to the Packaging section at the https://docs.fuel-infra.org/fuel-dev/packaging.html website. Co-Authored-By: Dmitry Burmistrov <dburmistrov@mirantis.com> Change-Id: Id564e4dcc150e48ece6d2710bc726d2dfe4ee842 Implements: blueprint replace-obs
This commit is contained in:
Olena Logvinova
Dmitry Burmistrov
parent
b09d472543
commit
e38a045569
|
|
@ -8,3 +8,4 @@ Packaging
|
|||
:numbered:
|
||||
|
||||
packaging/package_versions
|
||||
packaging/perestroika
|
||||
|
|
@ -0,0 +1,165 @@
|
|||
Perestroika build system
|
||||
========================
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
Fuel 7.0 introduces a new build system called Perestroika. It uses
|
||||
standard upstream Linux distribution tools in order to:
|
||||
|
||||
* build packages (SBuild/Mock)
|
||||
* publish packages to repositories
|
||||
* manage package repositories (using reprepro/createrepo tools).
|
||||
|
||||
Every package is built in a clean and up-to-date buildroot. Packages,
|
||||
their dependencies, and build dependencies are fully self-contained
|
||||
for each Mirantis OpenStack release. Any package included in any
|
||||
release can be rebuilt at any point in time using only the packages
|
||||
from that release.
|
||||
|
||||
The package build CI is reproducible and can be recreated from scratch
|
||||
in a repeatable way.
|
||||
|
||||
Perestroika is based on Docker which provides an easy distribution.
|
||||
Each supported Linux distribution contains proper Docker images with
|
||||
necessary tools and scripts.
|
||||
|
||||
For the advantages of Perestroika over OBS build system, see
|
||||
`Replace OBS with another build system <http://specs.fuel-infra.org/fuel-specs-master/specs/7.0/replace-obs.html>`_.
|
||||
|
||||
Perestroika structure
|
||||
---------------------
|
||||
|
||||
Code storage
|
||||
~~~~~~~~~~~~
|
||||
|
||||
Gerrit code review system is used as a code storage.
|
||||
|
||||
Gerrit projects structure:
|
||||
|
||||
* Mirantis OpenStack/Fuel Master node packages:
|
||||
|
||||
- code projects: *[customer-name]/openstack/{package name}*
|
||||
- spec projects: *[customer-name]/openstack-build/{package name}-build*
|
||||
|
||||
* Mirantis OpenStack Linux packages:
|
||||
|
||||
- code/spec projects: *[customer-name]/packages/{distribution}/{packagename}*
|
||||
|
||||
* Fuel Master node Linux packages (separated from Mirantis OpenStack
|
||||
Linux in 7.0):
|
||||
|
||||
- code/spec projects: *[customer-name]/packages/fuel/{distribution}/{package name}*
|
||||
|
||||
* Versioning scheme supported by project branches:
|
||||
|
||||
- OpenStack: *openstack-ci/fuel-{fuel version}/{openstack version}*
|
||||
- Mirantis OpenStack Linux/Fuel Master node: *{fuel version}*
|
||||
|
||||
**where**
|
||||
* *customer-name* should be empty for Mirantis OpenStack projects due
|
||||
to a backward compatibility with releases older than 7.0.
|
||||
* supported values of the ``{distribution}`` parameter are {centos6},
|
||||
{centos7} and {trusty}.
|
||||
|
||||
Scheduler
|
||||
~~~~~~~~~
|
||||
|
||||
This part is based on Jenkins CI tool. All jobs are configured using
|
||||
``jenkins-job-builder``. Jenkins has a separate set of jobs for each
|
||||
*[customer name]+[fuel version]* case. Gerrit-trigger is configured
|
||||
to track events from the *{version}* branch of all the *[customer-name]*
|
||||
Gerrit projects.
|
||||
|
||||
Each set of jobs contains:
|
||||
|
||||
* jobs for OpenStack packages for a cluster (``.rpm`` and ``.deb``)
|
||||
* jobs for Mirantis OpenStack Linux packages for a cluster (``.rpm``
|
||||
and ``.deb``)
|
||||
* jobs for OpenStack packages for Fuel Master node (``.rpm``). In case
|
||||
of using cluster packages, they are optional.
|
||||
* jobs for non-OpenStack Fuel Master node packages (``.rpm``)
|
||||
* jobs for Fuel packages (``.rpm`` and ``.deb``)
|
||||
* a job for package publishing
|
||||
|
||||
Build workers
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
These are hardware nodes with preconfigured build tools for all the
|
||||
supported distributions. They are configured as Jenkins slaves.
|
||||
|
||||
Each worker contains:
|
||||
|
||||
* preconfigured Docker images with native build tools for each
|
||||
distribution type:
|
||||
|
||||
- ``mockbuild`` builds packages using Mock (CentOS 6 and 7 target
|
||||
distributions are supported).
|
||||
- ``sbuild`` builds packages using SBuild tool (Ubuntu Trusty
|
||||
Tahr target distribution is supported only).
|
||||
|
||||
* prepared minimal build chroots for all the supported distributions:
|
||||
|
||||
- These chroots are updated on a daily basis in order to be up-to-date
|
||||
against the upstream state.
|
||||
- Chroot updating is performed by a separate Jenkins job.
|
||||
- No build jobs can run on a build host while the updating Jenkins job
|
||||
is running on it.
|
||||
|
||||
**Building stage flow**:
|
||||
|
||||
#. Checking out sources from Gerrit.
|
||||
#. Preparing sources to build (creating tarball files, updating
|
||||
changelogs).
|
||||
#. Building sources (performed in an isolated environment inside a
|
||||
Docker container).
|
||||
#. Getting the build stage exit status, build logs, and built
|
||||
packages.
|
||||
#. Parsing and archiving build logs for Jenkins artifacts.
|
||||
|
||||
Packaging CI uses short-lived Docker containers to perform package
|
||||
building. Docker images contain preconfigured build tools only. There
|
||||
are no chroots inside images. Build chroots are mounted to a Docker
|
||||
container at start in a read-only mode. Additionally *tmpfs* partition
|
||||
is mounted over a read-only chroot folder with AUFS overlays inside
|
||||
a Docker container. The container is destroyed once the build stage is
|
||||
completed.
|
||||
|
||||
**Goals of this scheme**:
|
||||
|
||||
* It can run a number of containers with the only chroot simultaneously
|
||||
on the same build host.
|
||||
* There is no need to perform clean-up operations after the build (all
|
||||
changes matter inside the container only and will be purged after the
|
||||
container is destroyed).
|
||||
* *tmpfs* works much faster than disk FS/LVM snapshots.
|
||||
|
||||
Publisher node
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
If the build stage finishes successfully, Jenkins runs a publishing
|
||||
job. The Publisher node contains all repositories for all customer
|
||||
projects. It is configured as a Jenkins slave. The repositories are
|
||||
maintained by native tools of their respective distribution
|
||||
(reprepro or createrepo).
|
||||
|
||||
The Publisher slave is fully private and available from Jenkins Master
|
||||
node only because of containing a GPG key. All the packages and
|
||||
repositories are signed in terms of their respective distribution by
|
||||
GPG keys that are stored on the Publisher node.
|
||||
|
||||
**Publishing stage**:
|
||||
|
||||
#. Getting built packages from the build host (over ``scp``).
|
||||
#. Checking if packages can be added to a repository (version checking
|
||||
against existing packages in order to prevent downgrading).
|
||||
#. Signing new packages (all ``.rpm`` and source ``.deb``) with GPG keys.
|
||||
#. Removing existing and adding new packages to a repository.
|
||||
#. Resigning the repository metadata.
|
||||
#. Syncing new repository state to a Mirror host (over ``rsync``).
|
||||
|
||||
Mirror node
|
||||
~~~~~~~~~~~
|
||||
|
||||
All repositories are available through http or rsync protocols and are
|
||||
synced by a Publisher to a Mirror host.
|
||||
Loading…
Reference in New Issue