Merge "Add a spec for reducing tarball size"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
30a88d4bd8
282
specs/6.0/reduce-tarball-size.rst
Normal file
282
specs/6.0/reduce-tarball-size.rst
Normal file
@ -0,0 +1,282 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
=======================================================
|
||||
Reduce an OpenStack release size inside upgrade tarball
|
||||
=======================================================
|
||||
|
||||
https://blueprints.launchpad.net/fuel/+spec/reduce-tarball-size
|
||||
|
||||
Let's reduce size of our fuel upgrade tarball, since it's huge and
|
||||
uncomfortable for distribution.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
We need to reduce upgrade tarball's size in order to simplify distribution
|
||||
workflow, boost up downloading/unpacking speed and reduce used space
|
||||
on master node's file system.
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
Terms
|
||||
-----
|
||||
|
||||
*openstack bundle*
|
||||
An OpenStack bundle is a set of artifacts which together constitute
|
||||
an openstack release in terms of Nailgun. The bundle contains next
|
||||
artifacts:
|
||||
|
||||
- puppet modules and manifests
|
||||
- ubuntu and centos repos
|
||||
- openstack.yaml
|
||||
|
||||
*full repo*
|
||||
A full repo is a self-sufficient repo which contains all packages
|
||||
that are required for both master node and openstack deployment.
|
||||
|
||||
*diff-based repo*
|
||||
A diff-based repo is a kind of repo that contains only those packages
|
||||
that were changed regarding to some previous repo. Please note,
|
||||
that a previous repo may be either full or diff-based one.
|
||||
|
||||
*base repo*
|
||||
A base repo term is used only in context of diff-based repos, when
|
||||
we want to point the repo on which a diff-based repo based.
|
||||
|
||||
Changes
|
||||
-------
|
||||
|
||||
Since Fuel 5.1 an upgrade tarball may contain few OpenStack bundles and
|
||||
each OpenStack bundle includes a self-sufficient repo. The "self-sufficient"
|
||||
here means that the repo contains not only OpenStack packages, but system.
|
||||
|
||||
The system packages are rarely changed so in most cases we can distribute
|
||||
within OpenStack bundle a small repo which contains only those packages
|
||||
that were changed regarding to some base previous release.
|
||||
|
||||
The changes have to be detected by packages' checksum, since we can't rely
|
||||
on package versions due to the following issue -
|
||||
`LP1376694 <https://bugs.launchpad.net/fuel/+bug/1376694>`_.
|
||||
|
||||
To achieve this goal we need to resolve next points:
|
||||
|
||||
* A build system should have a ``make centos-diff-repo ubuntu-diff-repo``
|
||||
targets to create a diff-based mirror between current mirror and some
|
||||
previous one. As it was mentioned in previous paragraph, a diff-based
|
||||
mirror should be created by comparing packages' checksums from both
|
||||
mirrors.
|
||||
|
||||
* A Jenkins job should be added to create a diff-based repo and provide
|
||||
it as an artifact.
|
||||
|
||||
* The Jenkins Job which builds ISO/tarball for current version (6.0)
|
||||
should be able to include additional *openstack bundles* in order
|
||||
to deliver more than one openstack release.
|
||||
|
||||
* The Fuel Upgrade script has to be able to add new "OpenStack releases"
|
||||
with one, two or more repos. In other words, the script should detect
|
||||
whether we're dealing with a diff-based repo and if so to add a path
|
||||
to base repo too.
|
||||
|
||||
Information about base repo should be added through ``metadata.yaml``.
|
||||
The YAML should be generated by make system and should be in the
|
||||
following format::
|
||||
|
||||
diff_releases:
|
||||
2014.2-6.0: 2014.1.1-5.1
|
||||
|
||||
where *2014.2-6.0* is a release to be installed and *2014.1.1-5.1* is
|
||||
a base release for *2014.2-6.0*. The ``metadata.yaml`` should be located
|
||||
in ``upgrade/releases/metadata.yaml`` folder inside upgrade tarball.
|
||||
|
||||
There should be only one major Fuel master node upgrade chain. For example::
|
||||
|
||||
5.1 -> 6.0
|
||||
5.1.1 -> 6.0
|
||||
|
||||
In other words, there shouldn't be a situation when we upgrade to ``6.0``
|
||||
from ``5.0.x`` series.
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
We can distribute just a set of changed packages instead of a small repo.
|
||||
In that case we will need to create a new repo on fly during fuel upgrade
|
||||
procedure. Creating repos on fly is not trivial procedure and may fail
|
||||
time to time due to outside factors.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
A deployment info (a one that Nailgun sends to Astute and Astute saves it
|
||||
as ``astute.yaml``) should have a ``repo_metadata`` field, which is a JSON
|
||||
object (or Python dictionary).
|
||||
|
||||
The ``repo_metadata`` has next structure::
|
||||
|
||||
"repo_metadata": {
|
||||
"repo_name_1": "repo_path_1",
|
||||
"repo_name_2": "repo_path_2",
|
||||
}
|
||||
|
||||
and will be converted into following yum's repo file::
|
||||
|
||||
[repo_name_1]
|
||||
name=repo_name_1
|
||||
baseurl=repo_path_1
|
||||
gpgcheck=0
|
||||
|
||||
[repo_name_2]
|
||||
name=repo_name_2
|
||||
baseurl=repo_path_2
|
||||
gpgcheck=0
|
||||
|
||||
and into next apt's sources list::
|
||||
|
||||
deb repo_path_1
|
||||
deb repo_path_2
|
||||
|
||||
The changes that was described above are already implemented in both
|
||||
Nailgun and Astute, and therefore should be kept as is.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
Master node IP is not a constant value, so we need to support some sort of
|
||||
macro replacement in the release handler. In other words if we want to add
|
||||
a Nailgun's release with the following orchestrator data::
|
||||
|
||||
{
|
||||
"puppet_manifests_source":
|
||||
"rsync://{MASTER_IP}:/puppet/{OPENSTACK_VERSION}/manifests/",
|
||||
|
||||
"puppet_modules_source":
|
||||
"rsync://{MASTER_IP}:/puppet/{OPENSTACK_VERSION}/modules/",
|
||||
|
||||
"repo_metadata": { ... }
|
||||
}
|
||||
|
||||
the ``{MASTER_IP}`` and the ``{OPENSTACK_VERSION}`` will be replaced by
|
||||
Nailgun with an actual master node ip address and an openstack version
|
||||
respectively.
|
||||
|
||||
Upgrade impact
|
||||
--------------
|
||||
|
||||
The Fuel Upgrade script has to be able to add new "OpenStack releases"
|
||||
with one, two or more repos.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
* Unpacking time of tarball should be reduced.
|
||||
* Downloading time of tarball should be reduced.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
* Slave nodes may have more than one repo in the operating system.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
|
||||
* Igor Kalnitsky <ikalnitsky@mirantis.com>
|
||||
* Sergey Kulanov <skulanov@mirantis.com>
|
||||
* Vladimir Kozhukalov <vkozhukalov@mirantis.com>
|
||||
|
||||
Other contributors:
|
||||
|
||||
* Fuel OSCI
|
||||
* Fuel DevOps
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Add ``make centos-diff-repo ubuntu-diff-repo`` targets to build system.
|
||||
* Configure a Jenkins Job for providing diff-based mirrors as an artifact.
|
||||
* Add ability to existing Jenkins Job to build an upgrade tarball with
|
||||
few *openstack bundles*.
|
||||
* Add support of diff-based repos to fuel-upgrade script.
|
||||
* Resolve issue -
|
||||
`LP1376694 <https://bugs.launchpad.net/fuel/+bug/1376694>`_.
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Existing Fuel Upgrade / OpenStack patching tests are enough since it's about
|
||||
improvements, not about entirely new feature.
|
||||
|
||||
But next tests may be added:
|
||||
|
||||
* Test that there're more than one repo on slaves.
|
||||
* Test that an upgrade tarball contains only changed packages, not all.
|
||||
* Test that a diff-based release passes the same tests as the full release.
|
||||
|
||||
Acceptance criteria
|
||||
-------------------
|
||||
|
||||
* Build an upgrade tarball with a diff-based repo. The file
|
||||
``upgrade/releases/metadata.yaml`` must be exist.
|
||||
* After master node upgrade a diff-based release must have at least two
|
||||
repos in ``repo_metadata``.
|
||||
* A cluster (with a diff-based release) must be deployed successfully
|
||||
and with packages from a diff-based repo.
|
||||
* Each slave in that cluster must have at least two repos.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The documentation about release management should be added and it should
|
||||
resolve next questions:
|
||||
|
||||
* Which components includes a release?
|
||||
* Where are the components stored?
|
||||
* How releases reuse packages from older releases.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
* #fuel-dev on freenode
|
||||
Loading…
Reference in New Issue
Block a user