From e1eedec9b77a1bd672d88c72d2c72d12f5c01b40 Mon Sep 17 00:00:00 2001 From: Igor Kalnitsky Date: Mon, 15 Sep 2014 17:36:18 +0300 Subject: [PATCH] Add a spec for reducing tarball size Spec for the blueprint reduce-tarball-size Change-Id: I2a35586d8abc484e6654eae97833b7637ab2613a --- specs/6.0/reduce-tarball-size.rst | 282 ++++++++++++++++++++++++++++++ 1 file changed, 282 insertions(+) create mode 100644 specs/6.0/reduce-tarball-size.rst diff --git a/specs/6.0/reduce-tarball-size.rst b/specs/6.0/reduce-tarball-size.rst new file mode 100644 index 00000000..5bd2f242 --- /dev/null +++ b/specs/6.0/reduce-tarball-size.rst @@ -0,0 +1,282 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +======================================================= +Reduce an OpenStack release size inside upgrade tarball +======================================================= + +https://blueprints.launchpad.net/fuel/+spec/reduce-tarball-size + +Let's reduce size of our fuel upgrade tarball, since it's huge and +uncomfortable for distribution. + + +Problem description +=================== + +We need to reduce upgrade tarball's size in order to simplify distribution +workflow, boost up downloading/unpacking speed and reduce used space +on master node's file system. + + +Proposed change +=============== + +Terms +----- + +*openstack bundle* + An OpenStack bundle is a set of artifacts which together constitute + an openstack release in terms of Nailgun. The bundle contains next + artifacts: + + - puppet modules and manifests + - ubuntu and centos repos + - openstack.yaml + +*full repo* + A full repo is a self-sufficient repo which contains all packages + that are required for both master node and openstack deployment. + +*diff-based repo* + A diff-based repo is a kind of repo that contains only those packages + that were changed regarding to some previous repo. Please note, + that a previous repo may be either full or diff-based one. + +*base repo* + A base repo term is used only in context of diff-based repos, when + we want to point the repo on which a diff-based repo based. + +Changes +------- + +Since Fuel 5.1 an upgrade tarball may contain few OpenStack bundles and +each OpenStack bundle includes a self-sufficient repo. The "self-sufficient" +here means that the repo contains not only OpenStack packages, but system. + +The system packages are rarely changed so in most cases we can distribute +within OpenStack bundle a small repo which contains only those packages +that were changed regarding to some base previous release. + +The changes have to be detected by packages' checksum, since we can't rely +on package versions due to the following issue - +`LP1376694 `_. + +To achieve this goal we need to resolve next points: + +* A build system should have a ``make centos-diff-repo ubuntu-diff-repo`` + targets to create a diff-based mirror between current mirror and some + previous one. As it was mentioned in previous paragraph, a diff-based + mirror should be created by comparing packages' checksums from both + mirrors. + +* A Jenkins job should be added to create a diff-based repo and provide + it as an artifact. + +* The Jenkins Job which builds ISO/tarball for current version (6.0) + should be able to include additional *openstack bundles* in order + to deliver more than one openstack release. + +* The Fuel Upgrade script has to be able to add new "OpenStack releases" + with one, two or more repos. In other words, the script should detect + whether we're dealing with a diff-based repo and if so to add a path + to base repo too. + +Information about base repo should be added through ``metadata.yaml``. +The YAML should be generated by make system and should be in the +following format:: + + diff_releases: + 2014.2-6.0: 2014.1.1-5.1 + +where *2014.2-6.0* is a release to be installed and *2014.1.1-5.1* is +a base release for *2014.2-6.0*. The ``metadata.yaml`` should be located +in ``upgrade/releases/metadata.yaml`` folder inside upgrade tarball. + +There should be only one major Fuel master node upgrade chain. For example:: + + 5.1 -> 6.0 + 5.1.1 -> 6.0 + +In other words, there shouldn't be a situation when we upgrade to ``6.0`` +from ``5.0.x`` series. + + +Alternatives +------------ + +We can distribute just a set of changed packages instead of a small repo. +In that case we will need to create a new repo on fly during fuel upgrade +procedure. Creating repos on fly is not trivial procedure and may fail +time to time due to outside factors. + +Data model impact +----------------- + +A deployment info (a one that Nailgun sends to Astute and Astute saves it +as ``astute.yaml``) should have a ``repo_metadata`` field, which is a JSON +object (or Python dictionary). + +The ``repo_metadata`` has next structure:: + + "repo_metadata": { + "repo_name_1": "repo_path_1", + "repo_name_2": "repo_path_2", + } + +and will be converted into following yum's repo file:: + + [repo_name_1] + name=repo_name_1 + baseurl=repo_path_1 + gpgcheck=0 + + [repo_name_2] + name=repo_name_2 + baseurl=repo_path_2 + gpgcheck=0 + +and into next apt's sources list:: + + deb repo_path_1 + deb repo_path_2 + +The changes that was described above are already implemented in both +Nailgun and Astute, and therefore should be kept as is. + +REST API impact +--------------- + +Master node IP is not a constant value, so we need to support some sort of +macro replacement in the release handler. In other words if we want to add +a Nailgun's release with the following orchestrator data:: + + { + "puppet_manifests_source": + "rsync://{MASTER_IP}:/puppet/{OPENSTACK_VERSION}/manifests/", + + "puppet_modules_source": + "rsync://{MASTER_IP}:/puppet/{OPENSTACK_VERSION}/modules/", + + "repo_metadata": { ... } + } + +the ``{MASTER_IP}`` and the ``{OPENSTACK_VERSION}`` will be replaced by +Nailgun with an actual master node ip address and an openstack version +respectively. + +Upgrade impact +-------------- + +The Fuel Upgrade script has to be able to add new "OpenStack releases" +with one, two or more repos. + +Security impact +--------------- + +None + +Notifications impact +-------------------- + +None + +Other end user impact +--------------------- + +None + +Performance Impact +------------------ + +* Unpacking time of tarball should be reduced. +* Downloading time of tarball should be reduced. + +Other deployer impact +--------------------- + +* Slave nodes may have more than one repo in the operating system. + +Developer impact +---------------- + +None + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + +* Igor Kalnitsky +* Sergey Kulanov +* Vladimir Kozhukalov + +Other contributors: + +* Fuel OSCI +* Fuel DevOps + +Work Items +---------- + +* Add ``make centos-diff-repo ubuntu-diff-repo`` targets to build system. +* Configure a Jenkins Job for providing diff-based mirrors as an artifact. +* Add ability to existing Jenkins Job to build an upgrade tarball with + few *openstack bundles*. +* Add support of diff-based repos to fuel-upgrade script. +* Resolve issue - + `LP1376694 `_. + + +Dependencies +============ + +None + + +Testing +======= + +Existing Fuel Upgrade / OpenStack patching tests are enough since it's about +improvements, not about entirely new feature. + +But next tests may be added: + +* Test that there're more than one repo on slaves. +* Test that an upgrade tarball contains only changed packages, not all. +* Test that a diff-based release passes the same tests as the full release. + +Acceptance criteria +------------------- + +* Build an upgrade tarball with a diff-based repo. The file + ``upgrade/releases/metadata.yaml`` must be exist. +* After master node upgrade a diff-based release must have at least two + repos in ``repo_metadata``. +* A cluster (with a diff-based release) must be deployed successfully + and with packages from a diff-based repo. +* Each slave in that cluster must have at least two repos. + + +Documentation Impact +==================== + +The documentation about release management should be added and it should +resolve next questions: + +* Which components includes a release? +* Where are the components stored? +* How releases reuse packages from older releases. + + +References +========== + +* #fuel-dev on freenode