diff --git a/doc/source/topics/packaging.rst b/doc/source/topics/packaging.rst index b1fc7ffc34..152b60b764 100644 --- a/doc/source/topics/packaging.rst +++ b/doc/source/topics/packaging.rst @@ -1,209 +1,215 @@ -Packaging generalities -====================== - -Packaging can easily be understood as: you are an engineer building a car. -Unfortunately, you don't have anything else, other than a manual and very -few tools. Any more specific tool you will require to actually build your car -has to be created, too. - -As example, if you are going to add a library named "foo", it has to be - - - Free Software - - All tools required to build it, have to be provided (as package) as well. - - actively maintained, i.e. we need an active and responsive upstream. - - It should NOT require a specific file system layout, FHS applies. - -Embedded copies ---------------- - -Imagine if all packages had a local copy of jQuery. If a security hole was -discovered in jQuery, then we would have to write more than 90 patches in -Debian (as more than 90 packages are depending on jQuery). This is simply -not practical. Therefore, it is not acceptable to purely copy code from -other repositories into any software code (and no, Horizon cannot be a -special case). This tends to become a fork, diverging from its real -upstream. The main reason for this is that it's not being maintained, and if a -bug is discovered in original upstream, it can't easily be fixed by updating -just a single package. - -Another reason not to simply copy a library into Horizon source code is, it -might have conflicting licenses. Distributing sources with conflicting -licenses in one tarball will revoke rights in best case. In worst case, -one might be liable. +Packaging Software +================== -Why did we decide to use XStatic? ---------------------------------- +Software packages +----------------- -There's a bunch of problems that are addressed by XStatic, which would -have to be addressed if we are to find a new solution. Non-exhaustively, -XStatic provides these things which are inherently not available by default -with systems like NPM, Grunt and such: +This section describes some general things that a developer should know about +packaging software. This content is mostly derived from best practices. - - Dependency checks: The general rule makes it possible to check that - dependencies (including fonts, JavaScript libs, etc.) are available in - downstream distributions. The current XStatic system enables these checks. - - Reusable components across projects: The XStatic system ensures - components are reusable by other packages (ie: non-Horizon things, - like Fuel for example). - - System-wide registry of static content: XStatic brings a system-wide - registry of components, so it is easy to check if one is missing (ie: - no egg-info, broken package dependency, etc.). Currently, NPM doesn't - offer this. - - No embedded contents: The XStatic system makes sure we aren't embedding - files which are already available in the distribution (ie: libjs-* or - fonts-* packages). It even provides a compatibility layer for - distributions. Not every distribution places static files in the same - position of the file system. - On the other side, if you are packaging an XStatic package for your - distribution, make sure, you are using distribution provided static files. - Having put together something as XStatic package is *no* guarantee to - get it into a distribution. Preferably, XStatic provides only the - abstraction layer to use distribution provided static files. - - Package build systems are disconnected from the outside network (for several - reasons). +A developer building a package is comparable to an engineer building a car +with only a manual and very few tools. If the engineer needs a specific tool +to build the car, he must create the tool, too. -If you want to redesign a new system to replace XStatic, please keep the -above 4 points in mind. The new system *must* keep these features: please -don't sacrifice them just because it's more convenient for Horizon -developers. Yes, XStatic are making your life harder, we understand that. -But we really need the above. +As a developer, if you are going to add a library named “foo”, the package +must adhere to the following standards: + + - Be a free package created with free software. + - Include all tools that are required to build the package. + - Have an active and responsive upstream to maintain the package. + - Adhere to Filesystem Hierarchy Standards (FHS). A specific file system + layout is not required. + + +Embedded copies not allowed +--------------------------- + +Imagine if all packages had a local copy of jQuery. If a security hole is +discovered in jQuery, we must write more than 90 patches in Debian, one for +each package that includes a copy. This is simply not practical. Therefore, +it is unacceptable for Horizon to copy code from other repositories when +creating a package. Copying code from another repository tends to create a +fork, diverging from the upstream code. The fork includes code that is not +being maintained, so if a bug is discovered in the original upstream, it +cannot easily be fixed by updating a single package. + +Another reason to avoid copying a library into Horizon source code is that +it might create conflicting licenses. Distributing sources with conflicting +licenses in one tarball revokes rights in best case. In the worst case, you +could be held legally responsible. Free software ------------- Red Hat, Debian, and SUSE distributions are made only of free software (free -as in Libre, or free speech). This means that not only the software we -include in our repository is free, but also all the tools that are used are -free as well, and also available in the distribution. As package maintainers -care about the quality of the packages they upload, they do run the unit -tests which are available from upstream repositories. This also qualifies test -requirements as build requirements, i.e. the same rules apply for building -the software as for the software itself. Build requirements not included in -the distribution are not allowed. +as in Libre, or free speech). The software that we include in our repository +is free. The tools are also free, and available in the distribution. -One famous example is Selenium. For a long time, it was only available from -the non-free repositories of Debian. The reason was that upstream included -some .xpi binaries. These .xpi included some Windows .dll and Linux .so -files. As they couldn't be rebuilt from source, the whole of python-selenium -was declared non-free. If we made Horizon build-depends on python-selenium, -this would mean Horizon wouldn't be in Debian main anymore (remember: -contrib and non-free are *not* considered part of Debian). Recently, the -package maintainer of python-selenium decided to remove the .xpi files from -python-selenium, and upload it to Debian Experimental (this time, in main, -not in non-free). So if it was possible for Horizon to use python-selenium -(without the non-free .xpi files), then we could run Selenium tests at -package build time. +Because package maintainers care about the quality of the packages we upload, +we run tests that are available from upstream repositories. This also +qualifies test requirements as build requirements. The same rules apply for +building the software as for the software itself. Special build requirements +that are not included in the overall distribution are not allowed. + +An example of historically limiting, non-free software is Selenium. For a +long time, Selenium was only available from the non-free repositories of +Debian. The reason was that upstream included some .xpi binaries. These .xpi +included some Windows .dll and Linux .so files. Because they could not be +rebuilt from the source, all of python-selenium was declared non-free. If we +made Horizon build-depends on python-selenium, this would mean Horizon +wouldn't be in Debian main anymore (contrib and non-free are *not* considered +part of Debian). Recently, the package maintainer of python-selenium decided +to remove the .xpi files from python-selenium, and upload it to Debian +Experimental (this time, in main, not in non-free). If at some point it is +possible for Horizon to use python-selenium (without the non-free .xpi files), +then we could run Selenium tests at package build time. Running unit tests at build time -------------------------------- -The build environment inside a distribution isn't exactly the same as the +The build environment inside a distribution is not exactly the same as the one in the OpenStack gate. For example, versions of a given library can be -slightly different from the one in the gate. And we do want to detect when -this is a problem, so it can be fixed. So whenever possible, try to make the -lives of package maintainer easier, and allow them (or help them) to run -unit tests when it is possible. +slightly different from the one in the gate. We want to detect when +problematic differences exist so that we can fix them. Whenever possible, try +to make the lives of the package maintainer easier, and allow them (or help +them) to run unit tests. Minified JavaScript policy -------------------------- -In all free software distribution which actively maintain OpenStack -packages (ie: at least RDO, Debian, and Ubuntu), minified JavaScript are -considered non-free. This means that they should *not* be present in -upstream source code, or at least, a non-minified version should be present -next to the minified version. Also, you should be aware of potential -security issues with minifiers. This `blog post`_ explains it very well. +In free software distributions that actively maintain OpenStack packages (such +as RDO, Debian, and Ubuntu), minified JavaScript is considered non-free. This +means that minified JavaScript should *not* be present in upstream source +code. At the very least, a non-minified version should be present next to the +minified version. Also, be aware of potential security issues with minifiers. +This `blog post`_ explains it very well. - .. _`blog post`: https://zyan.scripts.mit.edu/blog/backdooring-js/ +.. _`blog post`: https://zyan.scripts.mit.edu/blog/backdooring-js/ Component version ----------------- -One very important thing to take care about, is the version of all the -components you will use in your app. Since it is not acceptable to embed a -given component within Horizon, then we must use what's in the distribution -(including all fonts, JavaScript, etc.). This is where it becomes a bit -tricky. +Be careful about the version of all the components you use in your +application. Since it is not acceptable to embed a given component within +Horizon, we must use what is in the distribution, including all fonts, +JavaScript, etc. This is where it becomes a bit tricky. -In most distribution, it is not acceptable to have multiple version of the -same piece of software. - -In Red Hat systems, it is technically possible to install 2 versions of -one library at the same time, but a few restrictions apply, especially for -usage. However, package maintainers try to avoid this as much as possible. -For package dependency resolution, it might be necessary to provide packages -for depending packages as well. For example, if you had Django-1.4 and -Django-1.8 in the same release, you would have to provide Horizon built for -Django-1.4 and another package providing Horizon built for Django-1.8. This -is a high effort and needs to be evaluated carefully. +In most distributions, it is not acceptable to have multiple versions of the +same piece of software. In Red Hat systems, it is technically possible to +install 2 versions of one library at the same time, but a few restrictions +apply, especially for usage. However, package maintainers try to avoid +multiple versions as much as possible. For package dependency resolution, it +might be necessary to provide packages for depending packages as well. For +example, if you had Django-1.4 and Django-1.8 in the same release, you must +provide Horizon built for Django-1.4 and another package providing Horizon +built for Django-1.8. This is a large effort and needs to be evaluated +carefully. In Debian, it is generally forbidden to have multiple versions of the same -library in the same Debian release; there are very few specific exceptions -to that rule. +library in the same Debian release. Very few exceptions exist. -This has consequences for an upstream author willing to integrate their -software in a downstream distribution. The best situation is when it is -possible to support whatever version is currently available in the target -distributions, up to the latest version upstream. Declaring lower -and upper bounds within your requirements.txt doesn't solve the issue. It -allows all the tests to pass on gate because they are run against a narrow set -of versions in requirements.txt, while the downstream distribution may still -have some dependencies with versions outside of the range specified in -requirements.txt - which may lead to failures not caught in the OpenStack gate. +Component versioning has consequences for an upstream author willing to +integrate their software in a downstream distribution. The best situation +is when it is possible to support whatever version is currently available +in the target distributions, up to the latest version upstream. Declaring +lower and upper bounds within your requirements.txt does not solve the issue. +It allows all the tests to pass on gate because they are run against a narrow +set of versions in requirements.txt. The downstream distribution might still +have some dependencies with versions outside of the range that is specified +in requirements.txt. These dependencies may lead to failures that are not +caught in the OpenStack gate. -When it's not possible to support all versions of a library (because it would -be too much work, or when it would then be very hard to test in the gate), -then the best recommendation is to use whatever is available inside the -target distributions. For example, Horizon currently supports -jQuery >= 1.7.2, as this is what is currently available in Debian Jessie -and Ubuntu Trusty (the last LTS). +At times it might not be possible to support all versions of a library. It +might be too much work, or it might be very hard to test in the gate. In this +case, it is best to use whatever is available inside the target distributions. +For example, Horizon currently supports jQuery >= 1.7.2, as this is what is +currently available in Debian Jessie and Ubuntu Trusty (the last LTS). -One would search in a distribution for a piece of software foo using a command +You can search in a distribution for a piece of software foo using a command like ``dnf search foo``, or ``zypper se -s foo``. ``dnf info foo`` returns more detailed information about the package. - Filesystem Hierarchy Standards ------------------------------ -Every distribution has to comply with the FHS (Filesystem Hierarchy -Standards). This defines a set of rules which we *must* follow as package +Every distribution must comply with the Filesystem Hierarchy Standards (FHS). +The FHS defines a set of rules that we *must* follow as package maintainers. Some of the most important ones are: - - /usr should be considered as read only, and no software should write in it - at runtime (however, it is fine for a package post installation script - to write there). As a consequence, distributions had to write many - tricks to convince horizon to write in /var/lib only (for example: - writing symlinks to /var/lib/openstack-dashboard, or patch the default - local_settings.py to write the SECRET_KEY in /var). - - Configuration should always be in /etc, no matter what. As a consequence, - package maintainers had to place symlinks to - /etc/openstack-dashboard/local_settings in Red Hat based distributions - instead of using directly - /usr/share/openstack-dashboard/openstack_dashboard/local/local_settings.py - which Horizon expects. In Debian the configuration file is named - /etc/openstack-dashboard/local_settings.py +- /usr is considered read only. Software must not write in /usr at + runtime. However, it is fine for a package post-installation script to write + in /usr. When this rule was not followed, distributions had to write many + tricks to convince Horizon to write in /var/lib only. For example, + distributions wrote symlinks to /var/lib/openstack-dashboard, or patched + the default local_settings.py to write the SECRET_KEY in /var. +- Configuration must always be in /etc, no matter what. When this rule + was not followed, package maintainers had to place symlinks to + /etc/openstack-dashboard/local_settings in Red Hat based distributions + instead of using directly + /usr/share/openstack-dashboard/openstack_dashboard/local/local_settings.py + which Horizon expects. In Debian,the configuration file is named + /etc/openstack-dashboard/local_settings.py. + + +Packaging Horizon +================= + + +Why we use XStatic +------------------ + +XStatic provides the following features that are are not currently available +by default with systems like NPM and Grunt: + + - Dependency checks: XStatic checks that dependencies, such as fonts + and JavaScript libs, are available in downstream distributions. + - Reusable components across projects: The XStatic system ensures + components are reusable by other packages, like Fuel. + - System-wide registry of static content: XStatic brings a system-wide + registry of components, so that it is easy to check if one is missing. For + example, it can detect if there is no egg-info, or a broken package + dependency exists. + - No embedded content: The XStatic system helps us avoid embedding files that + are already available in the distribution, for example, libjs-* or fonts-* + packages. It even provides a compatibility layer for distributions. Not + every distribution places static files in the same position in the file + system. If you are packaging an XStatic package for your distribution, make + sure that you are using the static files provided by that specific + distribution. Having put together an XStatic package is *no* guarantee to + get it into a distribution. XStatic provides only the abstraction layer to + use distribution provided static files. + - Package build systems are disconnected from the outside network (for + several reasons). Other packaging systems download dependencies directly + from the internet without verifying that the downloaded file is intact, + matches a provided checksum, etc. With these other systems, there is no way + to provide a mirror, a proxy or a cache, making builds even more unstable + when minor networking issues are encountered. + +The previous features are critical requirements of the Horizon packaging +system. Any new system *must* keep these features. Although XStatic may mean +a few additional steps from individual developers, those steps help maintain +consistency and prevent errors across the project. Packaging Horizon for distributions ----------------------------------- -Horizon is a python module. It will preferably be installed at default -location for python; e.g in Fedora and openSUSE, this is +Horizon is a Python module. Preferably, it is installed at the default +location for python. In Fedora and openSUSE, this is /usr/lib/python2.7/site-packages/horizon, and in Debian/Ubuntu it is /usr/lib/python2.7/dist-packages/horizon. -Configuration files should live under /etc/openstack-dashboard; policy files -should be created and modified there as well. +Configuration files should reside under /etc/openstack-dashboard. Policy +files should be created and modified there as well. -It is expected that ``manage.py collectstatic`` will be executed during -package build. +It is expected that ``manage.py collectstatic`` will be run during package +build. This is the `recommended way`_ for Django applications. Depending on configuration, it might be required to ``manage.py compress`` during package build, too.