Improve readability of Horizon Packaging doc
Include various changes for readability. Make bulleted list parallel. Shorten long sentences. In some cases include articles and clarify subject in sentences. Spell out "for example." Eliminate redundant words / phrases. Per Matthias's recommendation, shuffle order of topics and create two sections to focus first on general packaging principles, and then Horizon specifics. Change-Id: Ic864dd926d723c6dc9749249655ae2e7d08ed225 Co-Authored-By: Matthias Runge <mrunge@redhat.com> Closes-Bug: #1516986
This commit is contained in:
Gudrun Wolfgram
@@ -1,209 +1,215 @@
|
|||||||
Packaging generalities
|
Packaging Software
|
||||||
======================
|
==================
|
||||||
|
|
||||||
Packaging can easily be understood as: you are an engineer building a car.
|
|
||||||
Unfortunately, you don't have anything else, other than a manual and very
|
|
||||||
few tools. Any more specific tool you will require to actually build your car
|
|
||||||
has to be created, too.
|
|
||||||
|
|
||||||
As example, if you are going to add a library named "foo", it has to be
|
|
||||||
|
|
||||||
- Free Software
|
|
||||||
- All tools required to build it, have to be provided (as package) as well.
|
|
||||||
- actively maintained, i.e. we need an active and responsive upstream.
|
|
||||||
- It should NOT require a specific file system layout, FHS applies.
|
|
||||||
|
|
||||||
Embedded copies
|
|
||||||
---------------
|
|
||||||
|
|
||||||
Imagine if all packages had a local copy of jQuery. If a security hole was
|
|
||||||
discovered in jQuery, then we would have to write more than 90 patches in
|
|
||||||
Debian (as more than 90 packages are depending on jQuery). This is simply
|
|
||||||
not practical. Therefore, it is not acceptable to purely copy code from
|
|
||||||
other repositories into any software code (and no, Horizon cannot be a
|
|
||||||
special case). This tends to become a fork, diverging from its real
|
|
||||||
upstream. The main reason for this is that it's not being maintained, and if a
|
|
||||||
bug is discovered in original upstream, it can't easily be fixed by updating
|
|
||||||
just a single package.
|
|
||||||
|
|
||||||
Another reason not to simply copy a library into Horizon source code is, it
|
|
||||||
might have conflicting licenses. Distributing sources with conflicting
|
|
||||||
licenses in one tarball will revoke rights in best case. In worst case,
|
|
||||||
one might be liable.
|
|
||||||
|
|
||||||
|
|
||||||
Why did we decide to use XStatic?
|
Software packages
|
||||||
---------------------------------
|
-----------------
|
||||||
|
|
||||||
There's a bunch of problems that are addressed by XStatic, which would
|
This section describes some general things that a developer should know about
|
||||||
have to be addressed if we are to find a new solution. Non-exhaustively,
|
packaging software. This content is mostly derived from best practices.
|
||||||
XStatic provides these things which are inherently not available by default
|
|
||||||
with systems like NPM, Grunt and such:
|
|
||||||
|
|
||||||
- Dependency checks: The general rule makes it possible to check that
|
A developer building a package is comparable to an engineer building a car
|
||||||
dependencies (including fonts, JavaScript libs, etc.) are available in
|
with only a manual and very few tools. If the engineer needs a specific tool
|
||||||
downstream distributions. The current XStatic system enables these checks.
|
to build the car, he must create the tool, too.
|
||||||
- Reusable components across projects: The XStatic system ensures
|
|
||||||
components are reusable by other packages (ie: non-Horizon things,
|
|
||||||
like Fuel for example).
|
|
||||||
- System-wide registry of static content: XStatic brings a system-wide
|
|
||||||
registry of components, so it is easy to check if one is missing (ie:
|
|
||||||
no egg-info, broken package dependency, etc.). Currently, NPM doesn't
|
|
||||||
offer this.
|
|
||||||
- No embedded contents: The XStatic system makes sure we aren't embedding
|
|
||||||
files which are already available in the distribution (ie: libjs-* or
|
|
||||||
fonts-* packages). It even provides a compatibility layer for
|
|
||||||
distributions. Not every distribution places static files in the same
|
|
||||||
position of the file system.
|
|
||||||
On the other side, if you are packaging an XStatic package for your
|
|
||||||
distribution, make sure, you are using distribution provided static files.
|
|
||||||
Having put together something as XStatic package is *no* guarantee to
|
|
||||||
get it into a distribution. Preferably, XStatic provides only the
|
|
||||||
abstraction layer to use distribution provided static files.
|
|
||||||
- Package build systems are disconnected from the outside network (for several
|
|
||||||
reasons).
|
|
||||||
|
|
||||||
If you want to redesign a new system to replace XStatic, please keep the
|
As a developer, if you are going to add a library named “foo”, the package
|
||||||
above 4 points in mind. The new system *must* keep these features: please
|
must adhere to the following standards:
|
||||||
don't sacrifice them just because it's more convenient for Horizon
|
|
||||||
developers. Yes, XStatic are making your life harder, we understand that.
|
- Be a free package created with free software.
|
||||||
But we really need the above.
|
- Include all tools that are required to build the package.
|
||||||
|
- Have an active and responsive upstream to maintain the package.
|
||||||
|
- Adhere to Filesystem Hierarchy Standards (FHS). A specific file system
|
||||||
|
layout is not required.
|
||||||
|
|
||||||
|
|
||||||
|
Embedded copies not allowed
|
||||||
|
---------------------------
|
||||||
|
|
||||||
|
Imagine if all packages had a local copy of jQuery. If a security hole is
|
||||||
|
discovered in jQuery, we must write more than 90 patches in Debian, one for
|
||||||
|
each package that includes a copy. This is simply not practical. Therefore,
|
||||||
|
it is unacceptable for Horizon to copy code from other repositories when
|
||||||
|
creating a package. Copying code from another repository tends to create a
|
||||||
|
fork, diverging from the upstream code. The fork includes code that is not
|
||||||
|
being maintained, so if a bug is discovered in the original upstream, it
|
||||||
|
cannot easily be fixed by updating a single package.
|
||||||
|
|
||||||
|
Another reason to avoid copying a library into Horizon source code is that
|
||||||
|
it might create conflicting licenses. Distributing sources with conflicting
|
||||||
|
licenses in one tarball revokes rights in best case. In the worst case, you
|
||||||
|
could be held legally responsible.
|
||||||
|
|
||||||
|
|
||||||
Free software
|
Free software
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
Red Hat, Debian, and SUSE distributions are made only of free software (free
|
Red Hat, Debian, and SUSE distributions are made only of free software (free
|
||||||
as in Libre, or free speech). This means that not only the software we
|
as in Libre, or free speech). The software that we include in our repository
|
||||||
include in our repository is free, but also all the tools that are used are
|
is free. The tools are also free, and available in the distribution.
|
||||||
free as well, and also available in the distribution. As package maintainers
|
|
||||||
care about the quality of the packages they upload, they do run the unit
|
|
||||||
tests which are available from upstream repositories. This also qualifies test
|
|
||||||
requirements as build requirements, i.e. the same rules apply for building
|
|
||||||
the software as for the software itself. Build requirements not included in
|
|
||||||
the distribution are not allowed.
|
|
||||||
|
|
||||||
One famous example is Selenium. For a long time, it was only available from
|
Because package maintainers care about the quality of the packages we upload,
|
||||||
the non-free repositories of Debian. The reason was that upstream included
|
we run tests that are available from upstream repositories. This also
|
||||||
some .xpi binaries. These .xpi included some Windows .dll and Linux .so
|
qualifies test requirements as build requirements. The same rules apply for
|
||||||
files. As they couldn't be rebuilt from source, the whole of python-selenium
|
building the software as for the software itself. Special build requirements
|
||||||
was declared non-free. If we made Horizon build-depends on python-selenium,
|
that are not included in the overall distribution are not allowed.
|
||||||
this would mean Horizon wouldn't be in Debian main anymore (remember:
|
|
||||||
contrib and non-free are *not* considered part of Debian). Recently, the
|
An example of historically limiting, non-free software is Selenium. For a
|
||||||
package maintainer of python-selenium decided to remove the .xpi files from
|
long time, Selenium was only available from the non-free repositories of
|
||||||
python-selenium, and upload it to Debian Experimental (this time, in main,
|
Debian. The reason was that upstream included some .xpi binaries. These .xpi
|
||||||
not in non-free). So if it was possible for Horizon to use python-selenium
|
included some Windows .dll and Linux .so files. Because they could not be
|
||||||
(without the non-free .xpi files), then we could run Selenium tests at
|
rebuilt from the source, all of python-selenium was declared non-free. If we
|
||||||
package build time.
|
made Horizon build-depends on python-selenium, this would mean Horizon
|
||||||
|
wouldn't be in Debian main anymore (contrib and non-free are *not* considered
|
||||||
|
part of Debian). Recently, the package maintainer of python-selenium decided
|
||||||
|
to remove the .xpi files from python-selenium, and upload it to Debian
|
||||||
|
Experimental (this time, in main, not in non-free). If at some point it is
|
||||||
|
possible for Horizon to use python-selenium (without the non-free .xpi files),
|
||||||
|
then we could run Selenium tests at package build time.
|
||||||
|
|
||||||
|
|
||||||
Running unit tests at build time
|
Running unit tests at build time
|
||||||
--------------------------------
|
--------------------------------
|
||||||
|
|
||||||
The build environment inside a distribution isn't exactly the same as the
|
The build environment inside a distribution is not exactly the same as the
|
||||||
one in the OpenStack gate. For example, versions of a given library can be
|
one in the OpenStack gate. For example, versions of a given library can be
|
||||||
slightly different from the one in the gate. And we do want to detect when
|
slightly different from the one in the gate. We want to detect when
|
||||||
this is a problem, so it can be fixed. So whenever possible, try to make the
|
problematic differences exist so that we can fix them. Whenever possible, try
|
||||||
lives of package maintainer easier, and allow them (or help them) to run
|
to make the lives of the package maintainer easier, and allow them (or help
|
||||||
unit tests when it is possible.
|
them) to run unit tests.
|
||||||
|
|
||||||
|
|
||||||
Minified JavaScript policy
|
Minified JavaScript policy
|
||||||
--------------------------
|
--------------------------
|
||||||
|
|
||||||
In all free software distribution which actively maintain OpenStack
|
In free software distributions that actively maintain OpenStack packages (such
|
||||||
packages (ie: at least RDO, Debian, and Ubuntu), minified JavaScript are
|
as RDO, Debian, and Ubuntu), minified JavaScript is considered non-free. This
|
||||||
considered non-free. This means that they should *not* be present in
|
means that minified JavaScript should *not* be present in upstream source
|
||||||
upstream source code, or at least, a non-minified version should be present
|
code. At the very least, a non-minified version should be present next to the
|
||||||
next to the minified version. Also, you should be aware of potential
|
minified version. Also, be aware of potential security issues with minifiers.
|
||||||
security issues with minifiers. This `blog post`_ explains it very well.
|
This `blog post`_ explains it very well.
|
||||||
|
|
||||||
.. _`blog post`: https://zyan.scripts.mit.edu/blog/backdooring-js/
|
.. _`blog post`: https://zyan.scripts.mit.edu/blog/backdooring-js/
|
||||||
|
|
||||||
|
|
||||||
Component version
|
Component version
|
||||||
-----------------
|
-----------------
|
||||||
|
|
||||||
One very important thing to take care about, is the version of all the
|
Be careful about the version of all the components you use in your
|
||||||
components you will use in your app. Since it is not acceptable to embed a
|
application. Since it is not acceptable to embed a given component within
|
||||||
given component within Horizon, then we must use what's in the distribution
|
Horizon, we must use what is in the distribution, including all fonts,
|
||||||
(including all fonts, JavaScript, etc.). This is where it becomes a bit
|
JavaScript, etc. This is where it becomes a bit tricky.
|
||||||
tricky.
|
|
||||||
|
|
||||||
In most distribution, it is not acceptable to have multiple version of the
|
In most distributions, it is not acceptable to have multiple versions of the
|
||||||
same piece of software.
|
same piece of software. In Red Hat systems, it is technically possible to
|
||||||
|
install 2 versions of one library at the same time, but a few restrictions
|
||||||
In Red Hat systems, it is technically possible to install 2 versions of
|
apply, especially for usage. However, package maintainers try to avoid
|
||||||
one library at the same time, but a few restrictions apply, especially for
|
multiple versions as much as possible. For package dependency resolution, it
|
||||||
usage. However, package maintainers try to avoid this as much as possible.
|
might be necessary to provide packages for depending packages as well. For
|
||||||
For package dependency resolution, it might be necessary to provide packages
|
example, if you had Django-1.4 and Django-1.8 in the same release, you must
|
||||||
for depending packages as well. For example, if you had Django-1.4 and
|
provide Horizon built for Django-1.4 and another package providing Horizon
|
||||||
Django-1.8 in the same release, you would have to provide Horizon built for
|
built for Django-1.8. This is a large effort and needs to be evaluated
|
||||||
Django-1.4 and another package providing Horizon built for Django-1.8. This
|
carefully.
|
||||||
is a high effort and needs to be evaluated carefully.
|
|
||||||
|
|
||||||
In Debian, it is generally forbidden to have multiple versions of the same
|
In Debian, it is generally forbidden to have multiple versions of the same
|
||||||
library in the same Debian release; there are very few specific exceptions
|
library in the same Debian release. Very few exceptions exist.
|
||||||
to that rule.
|
|
||||||
|
|
||||||
This has consequences for an upstream author willing to integrate their
|
Component versioning has consequences for an upstream author willing to
|
||||||
software in a downstream distribution. The best situation is when it is
|
integrate their software in a downstream distribution. The best situation
|
||||||
possible to support whatever version is currently available in the target
|
is when it is possible to support whatever version is currently available
|
||||||
distributions, up to the latest version upstream. Declaring lower
|
in the target distributions, up to the latest version upstream. Declaring
|
||||||
and upper bounds within your requirements.txt doesn't solve the issue. It
|
lower and upper bounds within your requirements.txt does not solve the issue.
|
||||||
allows all the tests to pass on gate because they are run against a narrow set
|
It allows all the tests to pass on gate because they are run against a narrow
|
||||||
of versions in requirements.txt, while the downstream distribution may still
|
set of versions in requirements.txt. The downstream distribution might still
|
||||||
have some dependencies with versions outside of the range specified in
|
have some dependencies with versions outside of the range that is specified
|
||||||
requirements.txt - which may lead to failures not caught in the OpenStack gate.
|
in requirements.txt. These dependencies may lead to failures that are not
|
||||||
|
caught in the OpenStack gate.
|
||||||
|
|
||||||
When it's not possible to support all versions of a library (because it would
|
At times it might not be possible to support all versions of a library. It
|
||||||
be too much work, or when it would then be very hard to test in the gate),
|
might be too much work, or it might be very hard to test in the gate. In this
|
||||||
then the best recommendation is to use whatever is available inside the
|
case, it is best to use whatever is available inside the target distributions.
|
||||||
target distributions. For example, Horizon currently supports
|
For example, Horizon currently supports jQuery >= 1.7.2, as this is what is
|
||||||
jQuery >= 1.7.2, as this is what is currently available in Debian Jessie
|
currently available in Debian Jessie and Ubuntu Trusty (the last LTS).
|
||||||
and Ubuntu Trusty (the last LTS).
|
|
||||||
|
|
||||||
One would search in a distribution for a piece of software foo using a command
|
You can search in a distribution for a piece of software foo using a command
|
||||||
like ``dnf search foo``, or ``zypper se -s foo``. ``dnf info foo`` returns
|
like ``dnf search foo``, or ``zypper se -s foo``. ``dnf info foo`` returns
|
||||||
more detailed information about the package.
|
more detailed information about the package.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Filesystem Hierarchy Standards
|
Filesystem Hierarchy Standards
|
||||||
------------------------------
|
------------------------------
|
||||||
|
|
||||||
Every distribution has to comply with the FHS (Filesystem Hierarchy
|
Every distribution must comply with the Filesystem Hierarchy Standards (FHS).
|
||||||
Standards). This defines a set of rules which we *must* follow as package
|
The FHS defines a set of rules that we *must* follow as package
|
||||||
maintainers. Some of the most important ones are:
|
maintainers. Some of the most important ones are:
|
||||||
|
|
||||||
- /usr should be considered as read only, and no software should write in it
|
- /usr is considered read only. Software must not write in /usr at
|
||||||
at runtime (however, it is fine for a package post installation script
|
runtime. However, it is fine for a package post-installation script to write
|
||||||
to write there). As a consequence, distributions had to write many
|
in /usr. When this rule was not followed, distributions had to write many
|
||||||
tricks to convince horizon to write in /var/lib only (for example:
|
tricks to convince Horizon to write in /var/lib only. For example,
|
||||||
writing symlinks to /var/lib/openstack-dashboard, or patch the default
|
distributions wrote symlinks to /var/lib/openstack-dashboard, or patched
|
||||||
local_settings.py to write the SECRET_KEY in /var).
|
the default local_settings.py to write the SECRET_KEY in /var.
|
||||||
- Configuration should always be in /etc, no matter what. As a consequence,
|
- Configuration must always be in /etc, no matter what. When this rule
|
||||||
package maintainers had to place symlinks to
|
was not followed, package maintainers had to place symlinks to
|
||||||
/etc/openstack-dashboard/local_settings in Red Hat based distributions
|
/etc/openstack-dashboard/local_settings in Red Hat based distributions
|
||||||
instead of using directly
|
instead of using directly
|
||||||
/usr/share/openstack-dashboard/openstack_dashboard/local/local_settings.py
|
/usr/share/openstack-dashboard/openstack_dashboard/local/local_settings.py
|
||||||
which Horizon expects. In Debian the configuration file is named
|
which Horizon expects. In Debian,the configuration file is named
|
||||||
/etc/openstack-dashboard/local_settings.py
|
/etc/openstack-dashboard/local_settings.py.
|
||||||
|
|
||||||
|
|
||||||
|
Packaging Horizon
|
||||||
|
=================
|
||||||
|
|
||||||
|
|
||||||
|
Why we use XStatic
|
||||||
|
------------------
|
||||||
|
|
||||||
|
XStatic provides the following features that are are not currently available
|
||||||
|
by default with systems like NPM and Grunt:
|
||||||
|
|
||||||
|
- Dependency checks: XStatic checks that dependencies, such as fonts
|
||||||
|
and JavaScript libs, are available in downstream distributions.
|
||||||
|
- Reusable components across projects: The XStatic system ensures
|
||||||
|
components are reusable by other packages, like Fuel.
|
||||||
|
- System-wide registry of static content: XStatic brings a system-wide
|
||||||
|
registry of components, so that it is easy to check if one is missing. For
|
||||||
|
example, it can detect if there is no egg-info, or a broken package
|
||||||
|
dependency exists.
|
||||||
|
- No embedded content: The XStatic system helps us avoid embedding files that
|
||||||
|
are already available in the distribution, for example, libjs-* or fonts-*
|
||||||
|
packages. It even provides a compatibility layer for distributions. Not
|
||||||
|
every distribution places static files in the same position in the file
|
||||||
|
system. If you are packaging an XStatic package for your distribution, make
|
||||||
|
sure that you are using the static files provided by that specific
|
||||||
|
distribution. Having put together an XStatic package is *no* guarantee to
|
||||||
|
get it into a distribution. XStatic provides only the abstraction layer to
|
||||||
|
use distribution provided static files.
|
||||||
|
- Package build systems are disconnected from the outside network (for
|
||||||
|
several reasons). Other packaging systems download dependencies directly
|
||||||
|
from the internet without verifying that the downloaded file is intact,
|
||||||
|
matches a provided checksum, etc. With these other systems, there is no way
|
||||||
|
to provide a mirror, a proxy or a cache, making builds even more unstable
|
||||||
|
when minor networking issues are encountered.
|
||||||
|
|
||||||
|
The previous features are critical requirements of the Horizon packaging
|
||||||
|
system. Any new system *must* keep these features. Although XStatic may mean
|
||||||
|
a few additional steps from individual developers, those steps help maintain
|
||||||
|
consistency and prevent errors across the project.
|
||||||
|
|
||||||
|
|
||||||
Packaging Horizon for distributions
|
Packaging Horizon for distributions
|
||||||
-----------------------------------
|
-----------------------------------
|
||||||
|
|
||||||
Horizon is a python module. It will preferably be installed at default
|
Horizon is a Python module. Preferably, it is installed at the default
|
||||||
location for python; e.g in Fedora and openSUSE, this is
|
location for python. In Fedora and openSUSE, this is
|
||||||
/usr/lib/python2.7/site-packages/horizon, and in Debian/Ubuntu it is
|
/usr/lib/python2.7/site-packages/horizon, and in Debian/Ubuntu it is
|
||||||
/usr/lib/python2.7/dist-packages/horizon.
|
/usr/lib/python2.7/dist-packages/horizon.
|
||||||
|
|
||||||
Configuration files should live under /etc/openstack-dashboard; policy files
|
Configuration files should reside under /etc/openstack-dashboard. Policy
|
||||||
should be created and modified there as well.
|
files should be created and modified there as well.
|
||||||
|
|
||||||
It is expected that ``manage.py collectstatic`` will be executed during
|
It is expected that ``manage.py collectstatic`` will be run during package
|
||||||
package build.
|
build.
|
||||||
This is the `recommended way`_ for Django applications.
|
This is the `recommended way`_ for Django applications.
|
||||||
Depending on configuration, it might be required to ``manage.py compress``
|
Depending on configuration, it might be required to ``manage.py compress``
|
||||||
during package build, too.
|
during package build, too.
|
||||||
|
|||||||
Reference in New Issue
Block a user