Merge "Modify HDP plugin doc for Ambari plugin"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
aa5b77ae73
Binary file not shown.
|
Before Width: | Height: | Size: 156 KiB |
@ -49,7 +49,7 @@ User guide
|
|||||||
|
|
||||||
userdoc/plugins
|
userdoc/plugins
|
||||||
userdoc/vanilla_plugin
|
userdoc/vanilla_plugin
|
||||||
userdoc/hdp_plugin
|
userdoc/ambari_plugin
|
||||||
userdoc/spark_plugin
|
userdoc/spark_plugin
|
||||||
userdoc/cdh_plugin
|
userdoc/cdh_plugin
|
||||||
userdoc/mapr_plugin
|
userdoc/mapr_plugin
|
||||||
|
|||||||
67
doc/source/userdoc/ambari_plugin.rst
Normal file
67
doc/source/userdoc/ambari_plugin.rst
Normal file
@ -0,0 +1,67 @@
|
|||||||
|
|
||||||
|
Ambari Plugin
|
||||||
|
=============
|
||||||
|
The Ambari sahara plugin provides a way to provision
|
||||||
|
clusters with Hortonworks Data Platform on OpenStack using templates in a
|
||||||
|
single click and in an easily repeatable fashion. The sahara controller serves
|
||||||
|
as the glue between Hadoop and OpenStack. The Ambari plugin mediates between
|
||||||
|
the sahara controller and Apache Ambari in order to deploy and configure Hadoop
|
||||||
|
on OpenStack. Core to the HDP Plugin is Apache Ambari
|
||||||
|
which is used as the orchestrator for deploying HDP on OpenStack. The Ambari
|
||||||
|
plugin uses Ambari Blueprints for cluster provisioning.
|
||||||
|
|
||||||
|
Apache Ambari Blueprints
|
||||||
|
------------------------
|
||||||
|
Apache Ambari Blueprints is a portable document definition, which provides a
|
||||||
|
complete definition for an Apache Hadoop cluster, including cluster topology,
|
||||||
|
components, services and their configurations. Ambari Blueprints can be
|
||||||
|
consumed by the Ambari plugin to instantiate a Hadoop cluster on OpenStack. The
|
||||||
|
benefits of this approach is that it allows for Hadoop clusters to be
|
||||||
|
configured and deployed using an Ambari native format that can be used with as
|
||||||
|
well as outside of OpenStack allowing for clusters to be re-instantiated in a
|
||||||
|
variety of environments.
|
||||||
|
|
||||||
|
Images
|
||||||
|
------
|
||||||
|
The sahara Ambari plugin is using minimal (operating system only) images.
|
||||||
|
|
||||||
|
For more information about Ambari images, refer to
|
||||||
|
https://github.com/openstack/sahara-image-elements.
|
||||||
|
|
||||||
|
You could download well tested and up-to-date prepared images from
|
||||||
|
http://sahara-files.mirantis.com/images/upstream/
|
||||||
|
|
||||||
|
HDP plugin requires an image to be tagged in sahara Image Registry with two
|
||||||
|
tags: 'ambari' and '<plugin version>' (e.g. '2.3').
|
||||||
|
|
||||||
|
Also in the Image Registry you will need to specify username for an image.
|
||||||
|
The username specified should be 'cloud-user' in case of CentOS 6.x image,
|
||||||
|
'centos' for CentOS 7 images and 'ubuntu' for Ubuntu images.
|
||||||
|
|
||||||
|
High Availability for HDFS and YARN
|
||||||
|
-----------------------------------
|
||||||
|
High Availability (Using the Quorum Journal Manager) can be
|
||||||
|
deployed automatically with the Ambari plugin. You can deploy High Avaliable
|
||||||
|
cluster through UI by selecting ``NameNode HA`` and/or ``ResourceManager HA``
|
||||||
|
options in general configs of cluster template.
|
||||||
|
|
||||||
|
The NameNode High Availability is deployed using 2 NameNodes, one active and
|
||||||
|
one standby. The NameNodes use a set of JournalNodes and Zookepeer Servers to
|
||||||
|
ensure the necessary synchronization. In case of ResourceManager HA 2
|
||||||
|
ResourceManagers should be enabled in addition.
|
||||||
|
|
||||||
|
A typical Highly available Ambari cluster uses 2 separate NameNodes, 2 separate
|
||||||
|
ResourceManagers and at least 3 JournalNodes and at least 3 Zookeeper Servers.
|
||||||
|
|
||||||
|
HDP Version Support
|
||||||
|
-------------------
|
||||||
|
The HDP plugin currently supports deployment of HDP 2.3.
|
||||||
|
|
||||||
|
Cluster Validation
|
||||||
|
------------------
|
||||||
|
Prior to Hadoop cluster creation, the HDP plugin will perform the following
|
||||||
|
validation checks to ensure a successful Hadoop deployment:
|
||||||
|
|
||||||
|
* Ensure the existence of Ambari Server process in the cluster;
|
||||||
|
* Ensure the existence of a NameNode, Zookeeper, ResourceManagers processes
|
||||||
|
HistoryServer and App TimeLine Server in the cluster
|
||||||
@ -1,127 +0,0 @@
|
|||||||
|
|
||||||
Hortonworks Data Platform Plugin
|
|
||||||
================================
|
|
||||||
The Hortonworks Data Platform (HDP) sahara plugin provides a way to provision
|
|
||||||
HDP clusters on OpenStack using templates in a single click and in an easily
|
|
||||||
repeatable fashion. As seen from the architecture diagram below, the sahara
|
|
||||||
controller serves as the glue between Hadoop and OpenStack. The HDP plugin
|
|
||||||
mediates between the sahara controller and Apache Ambari in order to deploy
|
|
||||||
and configure Hadoop on OpenStack. Core to the HDP Plugin is Apache Ambari
|
|
||||||
which is used as the orchestrator for deploying HDP on OpenStack.
|
|
||||||
|
|
||||||
.. image:: ../images/hdp-plugin-architecture.png
|
|
||||||
:width: 800 px
|
|
||||||
:scale: 80 %
|
|
||||||
:align: center
|
|
||||||
|
|
||||||
The HDP plugin can make use of Ambari Blueprints for cluster provisioning.
|
|
||||||
|
|
||||||
Apache Ambari Blueprints
|
|
||||||
------------------------
|
|
||||||
Apache Ambari Blueprints is a portable document definition, which provides a
|
|
||||||
complete definition for an Apache Hadoop cluster, including cluster topology,
|
|
||||||
components, services and their configurations. Ambari Blueprints can be
|
|
||||||
consumed by the HDP plugin to instantiate a Hadoop cluster on OpenStack. The
|
|
||||||
benefits of this approach is that it allows for Hadoop clusters to be
|
|
||||||
configured and deployed using an Ambari native format that can be used with as
|
|
||||||
well as outside of OpenStack allowing for clusters to be re-instantiated in a
|
|
||||||
variety of environments.
|
|
||||||
|
|
||||||
For more information about Apache Ambari Blueprints, refer to:
|
|
||||||
https://issues.apache.org/jira/browse/AMBARI-1783. Note that Apache Ambari
|
|
||||||
Blueprints are not yet finalized.
|
|
||||||
|
|
||||||
Operation
|
|
||||||
---------
|
|
||||||
The HDP Plugin performs the following four primary functions during cluster
|
|
||||||
creation:
|
|
||||||
|
|
||||||
1. Software deployment - the plugin orchestrates the deployment of the
|
|
||||||
required software to the target VMs
|
|
||||||
2. Services Installation - the Hadoop services configured for the node groups
|
|
||||||
within the cluster are installed on the associated VMs
|
|
||||||
3. Services Configuration - the plugin merges the default configuration values
|
|
||||||
and user provided configurations for each installed service to the cluster
|
|
||||||
4. Services Start - the plugin invokes the appropriate APIs to indicate to the
|
|
||||||
Ambari Server that the cluster services should be started
|
|
||||||
|
|
||||||
Images
|
|
||||||
------
|
|
||||||
The sahara HDP plugin can make use of either minimal (operating system only)
|
|
||||||
images or pre-populated HDP images. The base requirement for both is that the
|
|
||||||
image is cloud-init enabled and contains a supported operating system (see
|
|
||||||
http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.4/bk_hdp1-system-admin-guide/content/sysadminguides_ha_chap2_3.html).
|
|
||||||
|
|
||||||
The advantage of a pre-populated image is that provisioning time is reduced,
|
|
||||||
as packages do not need to be downloaded and installed which make up the
|
|
||||||
majority of the time spent in the provisioning cycle. In addition, provisioning
|
|
||||||
large clusters will put a burden on the network as packages for all nodes need
|
|
||||||
to be downloaded from the package repository.
|
|
||||||
|
|
||||||
For more information about HDP images, refer to
|
|
||||||
https://github.com/openstack/sahara-image-elements.
|
|
||||||
|
|
||||||
You could download well tested and up-to-date prepared images from
|
|
||||||
http://sahara-files.mirantis.com/images/upstream/liberty/
|
|
||||||
|
|
||||||
HDP plugin requires an image to be tagged in sahara Image Registry with two
|
|
||||||
tags: 'hdp' and '<hdp version>' (e.g. '2.0.6').
|
|
||||||
|
|
||||||
Also in the Image Registry you will need to specify username for an image.
|
|
||||||
The username specified should be 'cloud-user'.
|
|
||||||
|
|
||||||
HDFS NameNode High Availability
|
|
||||||
-------------------------------
|
|
||||||
HDFS NameNode High Availability (Using the Quorum Journal Manager) can be
|
|
||||||
deployed automatically with HDP 2.0.6. Currently the only way to deploy it is
|
|
||||||
through the command line client (python-saharaclient) or sahara REST API by
|
|
||||||
simply adding the following cluster_configs parameter in the cluster's JSON :
|
|
||||||
|
|
||||||
.. sourcecode:: cfg
|
|
||||||
|
|
||||||
"cluster_configs": {
|
|
||||||
"HDFSHA": {
|
|
||||||
"hdfs.nnha": true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
The NameNode High Availability is deployed using 2 NameNodes, one active and
|
|
||||||
one standby. The NameNodes use a set of JOURNALNODES and ZOOKEEPER_SERVERS to
|
|
||||||
ensure the necessary synchronization.
|
|
||||||
|
|
||||||
A typical Highly available HDP 2.0.6 cluster uses 2 separate NameNodes, at
|
|
||||||
least 3 JOURNALNODES and at least 3 ZOOKEEPER_SERVERS.
|
|
||||||
|
|
||||||
When HDFS NameNode High Availability is enabled, the plugin will perform the
|
|
||||||
following additional validations:
|
|
||||||
|
|
||||||
* Ensure the existence of 2 NAMENODES processes in the cluster
|
|
||||||
* Ensure the existence of at least 3 JOURNALNODES processes in the cluster
|
|
||||||
* Ensure the existence of at least 3 ZOOKEEPER_SERVERS processes in the cluster
|
|
||||||
|
|
||||||
|
|
||||||
Limitations
|
|
||||||
-----------
|
|
||||||
The HDP plugin currently has the following limitations:
|
|
||||||
|
|
||||||
* It is not possible to decrement the number of node-groups or hosts per node
|
|
||||||
group in a sahara generated cluster.
|
|
||||||
|
|
||||||
HDP Version Support
|
|
||||||
-------------------
|
|
||||||
The HDP plugin currently supports HDP 2.0.6.
|
|
||||||
|
|
||||||
Cluster Validation
|
|
||||||
------------------
|
|
||||||
Prior to Hadoop cluster creation, the HDP plugin will perform the following
|
|
||||||
validation checks to ensure a successful Hadoop deployment:
|
|
||||||
|
|
||||||
* Ensure the existence of a NAMENODE process in the cluster
|
|
||||||
* Ensure the existence of a JOBTRACKER should any TASKTRACKER be deployed to
|
|
||||||
the cluster
|
|
||||||
* Ensure the deployment of one Ambari Server instance to the cluster
|
|
||||||
* Ensure that each defined node group had an associated Ambari Agent configured
|
|
||||||
|
|
||||||
The HDP Plugin and sahara support
|
|
||||||
---------------------------------
|
|
||||||
For more information, please contact Hortonworks.
|
|
||||||
@ -7,7 +7,7 @@ Hadoop) or distribution, and allows configuration of topology and
|
|||||||
management/monitoring tools.
|
management/monitoring tools.
|
||||||
|
|
||||||
* :doc:`vanilla_plugin` - deploys Vanilla Apache Hadoop
|
* :doc:`vanilla_plugin` - deploys Vanilla Apache Hadoop
|
||||||
* :doc:`hdp_plugin` - deploys Hortonworks Data Platform
|
* :doc:`ambari_plugin` - deploys Hortonworks Data Platform
|
||||||
* :doc:`spark_plugin` - deploys Apache Spark with Cloudera HDFS
|
* :doc:`spark_plugin` - deploys Apache Spark with Cloudera HDFS
|
||||||
* :doc:`mapr_plugin` - deploys MapR plugin with MapR File System
|
* :doc:`mapr_plugin` - deploys MapR plugin with MapR File System
|
||||||
* :doc:`cdh_plugin` - deploys Cloudera Hadoop
|
* :doc:`cdh_plugin` - deploys Cloudera Hadoop
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user