openstack/deb-sahara
Code Issues Proposed changes

Update links for plugin images

Browse Source 
* update link for vanilla images
* update link for hdp images
* small doc refactoring

Change-Id: Ie2d47e57238d00887be4df29f3e2e75b218b8a35
This commit is contained in:
Sergey Reshetnyak 2014-09-23 20:52:46 +04:00
parent 38a92ed5a7
commit 17b000eb6c
2 changed files with 92 additions and 41 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

88
doc/source/userdoc/hdp_plugin.rst
View File

@ -1,7 +1,13 @@
Hortonworks Data Platform Plugin
================================
The Hortonworks Data Platform (HDP) Sahara plugin provides a way to provision HDP clusters on OpenStack using templates in a single click and in an easily repeatable fashion. As seen from the architecture diagram below, the Sahara controller serves as the glue between Hadoop and OpenStack. The HDP plugin mediates between the Sahara controller and Apache Ambari in order to deploy and configure Hadoop on OpenStack. Core to the HDP Plugin is Apache Ambari which is used as the orchestrator for deploying HDP on OpenStack.
The Hortonworks Data Platform (HDP) Sahara plugin provides a way to provision
HDP clusters on OpenStack using templates in a single click and in an easily
repeatable fashion. As seen from the architecture diagram below, the Sahara
controller serves as the glue between Hadoop and OpenStack. The HDP plugin
mediates between the Sahara controller and Apache Ambari in order to deploy
and configure Hadoop on OpenStack. Core to the HDP Plugin is Apache Ambari
which is used as the orchestrator for deploying HDP on OpenStack.
.. image:: ../images/hdp-plugin-architecture.png
:width: 800 px
@ -12,53 +18,93 @@ The HDP plugin can make use of Ambari Blueprints for cluster provisioning.
Apache Ambari Blueprints
------------------------
Apache Ambari Blueprints is a portable document definition, which provides a complete definition for an Apache Hadoop cluster, including cluster topology, components, services and their configurations. Ambari Blueprints can be consumed by the HDP plugin to instantiate a Hadoop cluster on OpenStack. The benefits of this approach is that it allows for Hadoop clusters to be configured and deployed using an Ambari native format that can be used with as well as outside of OpenStack allowing for clusters to be re-instantiated in a variety of environments.
Apache Ambari Blueprints is a portable document definition, which provides
a complete definition for an Apache Hadoop cluster, including cluster topology,
components, services and their configurations. Ambari Blueprints can be
consumed by the HDP plugin to instantiate a Hadoop cluster on OpenStack.
The benefits of this approach is that it allows for Hadoop clusters to be
configured and deployed using an Ambari native format that can be used with as
well as outside of OpenStack allowing for clusters to be re-instantiated in a
variety of environments.
For more information about Apache Ambari Blueprints, refer to: https://issues.apache.org/jira/browse/AMBARI-1783. Note that Apache Ambari Blueprints are not yet finalized.
For more information about Apache Ambari Blueprints, refer to:
https://issues.apache.org/jira/browse/AMBARI-1783. Note that Apache Ambari
Blueprints are not yet finalized.
Operation
---------
The HDP Plugin performs the following four primary functions during cluster creation:
The HDP Plugin performs the following four primary functions during cluster
creation:
1. Software deployment - the plugin orchestrates the deployment of the required software to the target VMs
2. Services Installation - the Hadoop services configured for the node groups within the cluster are installed on the associated VMs
3. Services Configuration - the plugin merges the default configuration values and user provided configurations for each installed service to the cluster
4. Services Start - the plugin invokes the appropriate APIs to indicate to the Ambari Server that the cluster services should be started
1. Software deployment - the plugin orchestrates the deployment of the
required software to the target VMs
2. Services Installation - the Hadoop services configured for the node groups
within the cluster are installed on the associated VMs
3. Services Configuration - the plugin merges the default configuration values
and user provided configurations for each installed service to the cluster
4. Services Start - the plugin invokes the appropriate APIs to indicate to the
Ambari Server that the cluster services should be started
Images
------
The Sahara HDP plugin can make use of either minimal (operating system only) images or pre-populated HDP images. The base requirement for both is that the image is cloud-init enabled and contains a supported operating system (see http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.4/bk_hdp1-system-admin-guide/content/sysadminguides_ha_chap2_3.html).
The Sahara HDP plugin can make use of either minimal (operating system only)
images or pre-populated HDP images. The base requirement for both is that the
image is cloud-init enabled and contains a supported operating system
(see http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.4/bk_hdp1-system-admin-guide/content/sysadminguides_ha_chap2_3.html).
The advantage of a pre-populated image is that provisioning time is reduced, as packages do not need to be downloaded and installed which make up the majority of the time spent in the provisioning cycle. In addition, provisioning large clusters will put a burden on the network as packages for all nodes need to be downloaded from the package repository.
The advantage of a pre-populated image is that provisioning time is reduced,
as packages do not need to be downloaded and installed which make up the
majority of the time spent in the provisioning cycle. In addition, provisioning
large clusters will put a burden on the network as packages for all nodes need
to be downloaded from the package repository.
For more information about HDP images, refer to https://github.com/openstack/sahara-image-elements.
For more information about HDP images, refer to
https://github.com/openstack/sahara-image-elements.
There are three VM images provided for use with the HDP Plugin, that can also be built using the tools available in sahara-image-elemnts:
There are three VM images provided for use with the HDP Plugin, that can also
be built using the tools available in sahara-image-elemnts:
1. `centos-6_64-hdp-1.3.qcow2 <http://public-repo-1.hortonworks.com/sahara/images/centos-6_4-64-hdp-1.3.qcow2>`_: This image contains most of the requisite packages necessary for HDP deployment. The packages contained herein correspond to the HDP 1.3 release. The operating system is a minimal CentOS 6.5 cloud-init enabled install. This image can only be used to provision HDP 1.3 hadoop clusters.
2. `centos-6_64-hdp-2.0.6.qcow2 <https://s3.amazonaws.com/public-repo-1.hortonworks.com/sahara/images/centos-6_4-64-hdp-2_0_6.qcow2>`_: This image contains most of the requisite packages necessary for HDP deployment. The packages contained herein correspond to the HDP 2.0.6 release. The operating system is a minimal CentOS 6.5 cloud-init enabled install. This image can only be used to provision HDP 2.0.6 hadoop clusters.
3. `centos-6-64-hdp-vanilla.qcow2 <http://public-repo-1.hortonworks.com/sahara/images/centos-6_4-64-vanilla.qcow2>`_: This image provides only a minimal install of CentOS 6.5 and is cloud-init enabled. This image can be used to provision any versions of HDP supported by Sahara.
1. `sahara-juno-hdp-1.3.2-centos-6.5.qcow2 <http://sahara-files.mirantis.com/sahara-juno-hdp-1.3.2-centos-6.5.qcow2>`_:
This image contains most of the requisite packages necessary for HDP
deployment. The packages contained herein correspond to the HDP 1.3 release.
The operating system is a minimal CentOS 6.5 cloud-init enabled install.
This image can only be used to provision HDP 1.3 hadoop clusters.
2. `sahara-juno-hdp-2.0.6-centos-6.5.qcow2 <http://sahara-files.mirantis.com/sahara-juno-hdp-2.0.6-centos-6.5.qcow2>`_:
This image contains most of the requisite packages necessary for HDP
deployment. The packages contained herein correspond to the HDP 2.0.6
release. The operating system is a minimal CentOS 6.5 cloud-init enabled
install. This image can only be used to provision HDP 2.0.6 hadoop clusters.
3. `sahara-juno-hdp-plain-centos-6.5.qcow2 <http://sahara-files.mirantis.com/sahara-juno-hdp-plain-centos-6.5.qcow2>`_:
This image provides only a minimal install of CentOS 6.5 and is cloud-init
enabled. This image can be used to provision any versions of HDP supported
by Sahara.
HDP plugin requires an image to be tagged in Sahara Image Registry with two tags: 'hdp' and '<hdp version>' (e.g. '1.3.2').
HDP plugin requires an image to be tagged in Sahara Image Registry with two
tags: 'hdp' and '<hdp version>' (e.g. '1.3.2').
Also in the Image Registry you will need to specify username for an image. The username specified should be 'root'.
Also in the Image Registry you will need to specify username for an image.
The username specified should be 'cloud-user'.
Limitations
-----------
The HDP plugin currently has the following limitations:
* It is not possible to decrement the number of node-groups or hosts per node group in a Sahara generated cluster.
* It is not possible to decrement the number of node-groups or hosts per node
group in a Sahara generated cluster.
HDP Version Support
-------------------
The HDP plugin currently supports HDP 1.3.2 and HDP 2.0.6. Support for future version of HDP will be provided shortly after software is generally available.
The HDP plugin currently supports HDP 1.3.2 and HDP 2.0.6. Support for future
version of HDP will be provided shortly after software is generally available.
Cluster Validation
------------------
Prior to Hadoop cluster creation, the HDP plugin will perform the following validation checks to ensure a successful Hadoop deployment:
Prior to Hadoop cluster creation, the HDP plugin will perform the following
validation checks to ensure a successful Hadoop deployment:
* Ensure the existence of a NAMENODE process in the cluster
* Ensure the existence of a JOBTRACKER should any TASKTRACKER be deployed to the cluster
* Ensure the existence of a JOBTRACKER should any TASKTRACKER be deployed to
the cluster
* Ensure the deployment of one Ambari Server instance to the cluster
* Ensure that each defined node group had an associated Ambari Agent configured

45
doc/source/userdoc/vanilla_plugin.rst
View File

@ -1,34 +1,36 @@
Vanilla Plugin
==============
Vanilla plugin is a reference plugin implementation which allows to operate with cluster with Apache Hadoop.
The vanilla plugin is a reference implementation which allows users to operate
a cluster with Apache Hadoop.
For cluster provisioning prepared images should be used. They already have
Apache Hadoop 1.2.1 and Apache Hadoop 2.3.0 installed. Here you can find
prepared images:
Apache Hadoop 1.2.1 and Apache Hadoop 2.4.1 installed. Prepared images
can be found at the following locations:
* http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-ubuntu-13.10.qcow2
* http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-fedora-20.qcow2
* http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-centos-6.5.qcow2
* http://sahara-files.mirantis.com/sahara-juno-vanilla-1.2.1-ubuntu-14.04.qcow2
* http://sahara-files.mirantis.com/sahara-juno-vanilla-1.2.1-centos-6.5.qcow2
* http://sahara-files.mirantis.com/sahara-icehouse-vanilla-2.3.0-ubuntu-13.10.qcow2
* http://sahara-files.mirantis.com/sahara-icehouse-vanilla-2.3.0-fedora-20.qcow2
* http://sahara-files.mirantis.com/sahara-icehouse-vanilla-2.3.0-centos-6.5.qcow2
* http://sahara-files.mirantis.com/sahara-juno-vanilla-2.4.1-ubuntu-14.04.qcow2
* http://sahara-files.mirantis.com/sahara-juno-vanilla-2.4.1-centos-6.5.qcow2
Besides, you may build images by yourself using :doc:`diskimagebuilder`.
Keep in mind that if you want to use "Swift Integration" feature ( :doc:`features`),
Hadoop 1.2.1 must be patched with implementation of Swift File System.
For more information about patching required by "Swift Integration" feature see :doc:`hadoop-swift`.
Additionally, you may build images by yourself using :doc:`diskimagebuilder`.
Keep in mind that if you want to use the Swift Integration feature
( :doc:`features`),
Hadoop 1.2.1 must be patched with an implementation of Swift File System.
For more information about patching required by the Swift Integration feature
see :doc:`hadoop-swift`.
Vanilla plugin requires an image to be tagged in Sahara Image Registry with
two tags: 'vanilla' and '<hadoop version>' (e.g. '1.2.1').
Also you should specify username of default cloud-user used in the Image:
The default username specified for these images is different
for each distribution:
+--------------+------------+
| OS | username |
+==============+============+
| Ubuntu 13.10 | ubuntu |
| Ubuntu 14.04 | ubuntu |
+--------------+------------+
| Fedora 20 | fedora |
+--------------+------------+
@ -42,13 +44,15 @@ Cluster Validation
When user creates or scales a Hadoop cluster using a Vanilla plugin,
the cluster topology requested by user is verified for consistency.
Currently there are the following limitations in cluster topology for Vanilla plugin:
Currently there are the following limitations in cluster topology for Vanilla
plugin:
For Vanilla Hadoop version 1.X.X:
+ Cluster must contain exactly one namenode
+ Cluster can contain at most one jobtracker
+ Cluster can contain at most one oozie and this process is also required for EDP
+ Cluster can contain at most one oozie and this process is also required
for EDP
+ Cluster can't contain oozie without jobtraker
+ Cluster can't have tasktracker nodes if it doesn't have jobtracker
@ -57,7 +61,8 @@ For Vanilla Hadoop version 2.X.X:
+ Cluster must contain exactly one namenode
+ Cluster can contain at most one resourcemanager
+ Cluster can contain at most one historyserver
+ Cluster can contain at most one oozie and this process is also required for EDP
+ Cluster can't contain oozie without resourcemanager and without historyserver
+ Cluster can contain at most one oozie and this process is also required
for EDP
+ Cluster can't contain oozie without resourcemanager and without
historyserver
+ Cluster can't have nodemanager nodes if it doesn't have resourcemanager