Update Sahara Dev Quickstart Guide

Clean-up of old references generalized version

Change-Id: Ie4e6554cd283cd81f56e268f3a99da3120aa4d19
Closes-Bug: #1485648
This commit is contained in:
Ashish Billore 2015-11-12 19:26:41 +05:30 committed by Vitaly Gridnev
parent a0162c8639
commit b0c1d8ba0a

View File

@ -12,10 +12,9 @@ of OpenStack command line tools and the sahara :doc:`REST API <../restapi>`.
OR
* If you just want to install and use Sahara follow
* If you just want to install and use sahara follow
:doc:`../userdoc/installation.guide`.
2. Identity service configuration
---------------------------------
@ -39,7 +38,6 @@ token using the ``keystone`` command line client as follows:
$ keystone token-get
If authentication succeeds, the output will be as follows:
.. sourcecode:: console
@ -47,10 +45,10 @@ If authentication succeeds, the output will be as follows:
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2013-07-08T15:21:18Z |
| id | dd92e3cdb4e1462690cd444d6b01b746 |
| tenant_id | 62bd2046841e4e94a87b4a22aa886c13 |
| user_id | 720fb87141a14fd0b204f977f5f02512 |
| expires | 2015-09-03T13:37:32Z |
| id | 2542c427092a4b09a07ee7612c3d99ae |
| tenant_id | c82e4bce56ce4cf9b90bd15dfdef699d |
| user_id | 7f5becaaa38b4c9e850ccd11672a4c96 |
+-----------+----------------------------------+
The ``id`` and ``tenant_id`` values will be used for creating REST calls
@ -61,40 +59,50 @@ variables for ease of use later.
.. sourcecode:: console
$ export AUTH_TOKEN="dd92e3cdb4e1462690cd444d6b01b746"
$ export TENANT_ID="62bd2046841e4e94a87b4a22aa886c13"
$ export AUTH_TOKEN="2542c427092a4b09a07ee7612c3d99ae"
$ export TENANT_ID="c82e4bce56ce4cf9b90bd15dfdef699d"
Alternatively, if a devstack environment is used, these values are available
through "openrc" file under the "devstack_install_root" directory and can be
configured as:
.. sourcecode:: console
$ source <devstack_install_root>/openrc
3. Upload an image to the Image service
---------------------------------------
You will need to upload a virtual machine image to the OpenStack Image
service. You can download pre-built images with vanilla Apache Hadoop
installed, or build the images yourself:
installed, or build the images yourself. This guide uses the latest available
Ubuntu upstream image, referred to as ``sahara-vanilla-latest-ubuntu.qcow2``
and the latest version of vanilla plugin as an example.
Sample images are available here:
* Download and install a pre-built image with Ubuntu 13.10
`Sample Images <http://sahara-files.mirantis.com/images/upstream/>`_
* Download a pre-built image
**Note:** For the steps below, substitute ``<openstack_release>`` with the
appropriate OpenStack release and ``<sahara_image>`` with the image of your
choice.
.. sourcecode:: console
$ ssh user@hostname
$ wget http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-ubuntu-13.10.qcow2
$ glance image-create --name=sahara-icehouse-vanilla-1.2.1-ubuntu-13.10 \
--disk-format=qcow2 --container-format=bare < ./sahara-icehouse-vanilla-1.2.1-ubuntu-13.10.qcow2
$ wget http://sahara-files.mirantis.com/images/upstream/<openstack_release>/<sahara_image>.qcow2
OR
* with Fedora 20
Upload the above downloaded image into the OpenStack Image service:
.. sourcecode:: console
$ ssh user@hostname
$ wget http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-fedora-20.qcow2
$ glance image-create --name=sahara-icehouse-vanilla-1.2.1-fedora-20 \
--disk-format=qcow2 --container-format=bare < ./sahara-icehouse-vanilla-1.2.1-fedora-20.qcow2
$ glance image-create --name=sahara-vanilla-latest-ubuntu \
--disk-format=qcow2 --container-format=bare < ./sahara-vanilla-latest-ubuntu.qcow2
OR
* build the image using :doc:`../userdoc/diskimagebuilder`.
* Build the image using: `diskimage-builder script <https://github.com/openstack/sahara-image-elements/blob/master/diskimage-create/README.rst>`_
Save the image id, this will be used during the image registration with
sahara. You can get the image id using the ``glance`` command line tool
@ -102,15 +110,14 @@ as follows:
.. sourcecode:: console
$ glance image-list --name sahara-icehouse-vanilla-1.2.1-ubuntu-13.10
+--------------------------------------+---------------------------------------------+
$ glance image-list --name sahara-vanilla-latest-ubuntu
+--------------------------------------+-------------------------------------+
| ID | Name |
+--------------------------------------+---------------------------------------------+
| 3f9fc974-b484-4756-82a4-bff9e116919b | sahara-icehouse-vanilla-1.2.1-ubuntu-13.10 |
+--------------------------------------+---------------------------------------------+
$ export IMAGE_ID="3f9fc974-b484-4756-82a4-bff9e116919b"
+--------------------------------------+-------------------------------------+
| c119f99c-67f2-4404-9cff-f30e4b185036 | sahara-vanilla-latest-ubuntu |
+--------------------------------------+-------------------------------------+
$ export IMAGE_ID="c119f99c-67f2-4404-9cff-f30e4b185036"
4. Register the image with the sahara image registry
----------------------------------------------------
@ -126,12 +133,16 @@ will vary depending on the source image used, for more please see*
$ sahara image-register --id $IMAGE_ID --username ubuntu
Tag the image to inform sahara about the plugin with which it shall be used:
Tag the image to inform sahara about the plugin and the version with which
it shall be used.
**Note:** For the steps below and the rest of this guide, substitute
``<plugin_version>`` with the appropriate version of your plugin.
.. sourcecode:: console
$ sahara image-add-tag --id $IMAGE_ID --tag vanilla
$ sahara image-add-tag --id $IMAGE_ID --tag 1.2.1
$ sahara image-add-tag --id $IMAGE_ID --tag <plugin_version>
Ensure that the image is registered correctly by querying sahara. If
registered successfully, the image will appear in the output as follows:
@ -139,48 +150,63 @@ registered successfully, the image will appear in the output as follows:
.. sourcecode:: console
$ sahara image-list
+--------------------------------------------+---------------------------------------+----------+----------------+-------------+
+------------------------------+--------------------------------------+----------+---------------------------+-------------+
| name | id | username | tags | description |
+--------------------------------------------+---------------------------------------+----------+----------------+-------------+
| sahara-icehouse-vanilla-1.2.1-ubuntu-13.10 | 3f9fc974-b484-4756-82a4-bff9e116919b | ubuntu | vanilla, 1.2.1 | None |
+--------------------------------------------+---------------------------------------+----------+----------------+-------------+
+------------------------------+--------------------------------------+----------+---------------------------+-------------+
| sahara-vanilla-latest-ubuntu | c119f99c-67f2-4404-9cff-f30e4b185036 | ubuntu | vanilla, <plugin_version> | None |
+------------------------------+--------------------------------------+----------+---------------------------+-------------+
5. Create node group templates
------------------------------
Node groups are the building blocks of clusters in sahara. Before you can
begin provisioning clusters you must define a few node group templates to
describe their configurations.
describe node group configurations.
*Note, these templates assume that floating IP addresses are not being
used, for more information please see* :ref:`floating_ip_management`
*Note, these templates assume that floating IP addresses are being used. For
more details on floating IP please see* :ref:`floating_ip_management`
Create a file named ``ng_master_template_create.json`` with the following
If your environment does not use floating IP, omit defining floating IP in
the template below.
Sample templates can be found here:
`Sample Templates <https://github.com/openstack/sahara/tree/master/sahara/plugins/default_templates/>`_
Create a file named ``my_master_template_create.json`` with the following
content:
.. sourcecode:: json
{
"name": "test-master-tmpl",
"flavor_id": "2",
"plugin_name": "vanilla",
"hadoop_version": "1.2.1",
"node_processes": ["jobtracker", "namenode"],
"hadoop_version": "<plugin_version>",
"node_processes": [
"namenode",
"resourcemanager",
"hiveserver"
],
"name": "vanilla-default-master",
"floating_ip_pool": "public",
"flavor_id": "2",
"auto_security_group": true
}
Create a file named ``ng_worker_template_create.json`` with the following
Create a file named ``my_worker_template_create.json`` with the following
content:
.. sourcecode:: json
{
"name": "test-worker-tmpl",
"flavor_id": "2",
"plugin_name": "vanilla",
"hadoop_version": "1.2.1",
"node_processes": ["tasktracker", "datanode"],
"hadoop_version": "<plugin_version>",
"node_processes": [
"nodemanager",
"datanode"
],
"name": "vanilla-default-worker",
"floating_ip_pool": "public",
"flavor_id": "2",
"auto_security_group": true
}
@ -188,8 +214,8 @@ Use the ``sahara`` client to upload the node group templates:
.. sourcecode:: console
$ sahara node-group-template-create --json ng_master_template_create.json
$ sahara node-group-template-create --json ng_worker_template_create.json
$ sahara node-group-template-create --json my_master_template_create.json
$ sahara node-group-template-create --json my_worker_template_create.json
List the available node group templates to ensure that they have been
added properly:
@ -197,19 +223,19 @@ added properly:
.. sourcecode:: console
$ sahara node-group-template-list
+------------------+--------------------------------------+-------------+-----------------------+-------------+
+------------------------+--------------------------------------+-------------+---------------------------------------+-------------+
| name | id | plugin_name | node_processes | description |
+------------------+--------------------------------------+-------------+-----------------------+-------------+
| test-master-tmpl | b38227dc-64fe-42bf-8792-d1456b453ef3 | vanilla | jobtracker, namenode | None |
| test-worker-tmpl | 634827b9-6a18-4837-ae15-5371d6ecf02c | vanilla | datanode, nodemanager | None |
+------------------+--------------------------------------+-------------+-----------------------+-------------+
+------------------------+--------------------------------------+-------------+---------------------------------------+-------------+
| vanilla-default-master | 9d3b5b2c-d5d5-4d16-8a93-a568d29c6569 | vanilla | namenode, resourcemanager, hiveserver | None |
| vanilla-default-worker | 1aa4a397-cb1e-4f38-be18-7f65fa0cc2eb | vanilla | nodemanager, datanode | None |
+------------------------+--------------------------------------+-------------+---------------------------------------+-------------+
Save the id for the master and worker node group templates as they will be
used during cluster template creation. For example:
* Master node group template id: ``b38227dc-64fe-42bf-8792-d1456b453ef3``
* Worker node group template id: ``634827b9-6a18-4837-ae15-5371d6ecf02c``
used during cluster template creation.
For example:
* Master node group template id: ``9d3b5b2c-d5d5-4d16-8a93-a568d29c6569``
* Worker node group template id: ``1aa4a397-cb1e-4f38-be18-7f65fa0cc2eb``
6. Create a cluster template
----------------------------
@ -217,82 +243,82 @@ used during cluster template creation. For example:
The last step before provisioning the cluster is to create a template
that describes the node groups of the cluster.
Create a file named ``cluster_template_create.json`` with the following
Create a file named ``my_cluster_template_create.json`` with the following
content:
.. sourcecode:: json
{
"name": "demo-cluster-template",
"plugin_name": "vanilla",
"hadoop_version": "1.2.1",
"hadoop_version": "<plugin_version>",
"node_groups": [
{
"name": "master",
"node_group_template_id": "b38227dc-64fe-42bf-8792-d1456b453ef3",
"count": 1
"name": "worker",
"count": 2,
"node_group_template_id": "1aa4a397-cb1e-4f38-be18-7f65fa0cc2eb"
},
{
"name": "workers",
"node_group_template_id": "634827b9-6a18-4837-ae15-5371d6ecf02c",
"count": 2
"name": "master",
"count": 1,
"node_group_template_id": "9d3b5b2c-d5d5-4d16-8a93-a568d29c6569"
}
]
],
"name": "vanilla-default-cluster",
"cluster_configs": {}
}
Upload the Cluster template using the ``sahara`` command line tool:
.. sourcecode:: console
$ sahara cluster-template-create --json cluster_template_create.json
$ sahara cluster-template-create --json my_cluster_template_create.json
Save the template id for use in the cluster provisioning command. The
cluster id can be found in the output of the creation command or by
listing the cluster templates as follows:
Save the cluster template id for use in the cluster provisioning command. The
cluster id can be found in the output of the creation command or by listing
the cluster templates as follows:
.. sourcecode:: console
$ sahara cluster-template-list
+-----------------------+--------------------------------------+-------------+-----------------------+-------------+
+-------------------------+--------------------------------------+-------------+----------------------+-------------+
| name | id | plugin_name | node_groups | description |
+-----------------------+--------------------------------------+-------------+-----------------------+-------------+
| demo-cluster-template | c0609da7-faac-4dcf-9cbc-858a3aa130cd | vanilla | master: 1, workers: 2 | None |
+-----------------------+--------------------------------------+-------------+-----------------------+-------------+
+-------------------------+--------------------------------------+-------------+----------------------+-------------+
| vanilla-default-cluster | 74add4df-07c2-4053-931f-d5844712727f | vanilla | master: 1, worker: 2 | None |
+-------------------------+--------------------------------------+-------------+----------------------+-------------+
7. Create cluster
-----------------
Now you will provision the cluster. This step requires a few pieces of
information that will be found by querying various OpenStack services.
Now you are ready to provision the cluster. This step requires a few pieces of
information that can be found by querying various OpenStack services.
Create a file named ``cluster_create.json`` with the following content:
Create a file named ``my_cluster_create.json`` with the following content:
.. sourcecode:: json
{
"name": "cluster-1",
"name": "my-cluster-1",
"plugin_name": "vanilla",
"hadoop_version": "1.2.1",
"cluster_template_id" : "c0609da7-faac-4dcf-9cbc-858a3aa130cd",
"user_keypair_id": "stack",
"default_image_id": "3f9fc974-b484-4756-82a4-bff9e116919b"
"hadoop_version": "<plugin_version>",
"cluster_template_id" : "74add4df-07c2-4053-931f-d5844712727f",
"user_keypair_id": "my_stack",
"default_image_id": "c119f99c-67f2-4404-9cff-f30e4b185036",
"neutron_management_network": "8cccf998-85e4-4c5f-8850-63d33c1c6916"
}
The parameter ``user_keypair_id`` with the value ``stack`` is generated by
The parameter ``user_keypair_id`` with the value ``my_stack`` is generated by
creating a keypair. You can create your own keypair in the OpenStack
Dashboard, or through the ``nova`` command line client as follows:
.. sourcecode:: console
$ nova keypair-add stack --pub-key $PATH_TO_PUBLIC_KEY
$ nova keypair-add my_stack --pub-key $PATH_TO_PUBLIC_KEY
If sahara is configured to use neutron for networking, you will also need to
include the ``neutron_management_network`` parameter in
``cluster_create.json``. Cluster instances will get fixed IP addresses in
this network. You can determine the neutron network id with the following
command:
``my_cluster_create.json``. If your environment does not use neutron, you can
omit ``neutron_management_network`` above. You can determine the neutron
network id with the following command:
.. sourcecode:: console
@ -302,62 +328,244 @@ Create and start the cluster:
.. sourcecode:: console
$ sahara cluster-create --json cluster_create.json
$ sahara cluster-create --json my_cluster_create.json
+----------------------------+-------------------------------------------------+
| Property | Value |
+----------------------------+-------------------------------------------------+
| status | Validating |
| neutron_management_network | 8cccf998-85e4-4c5f-8850-63d33c1c6916 |
| status | Active |
| neutron_management_network | None |
| is_transient | False |
| description | None |
| user_keypair_id | stack |
| updated_at | 2013-07-07T19:01:51 |
| user_keypair_id | my_stack |
| updated_at | 2015-09-02T10:58:02 |
| plugin_name | vanilla |
| provision_progress | [{u'successful': True, u'tenant_id': |
| | u'c82e4bce56ce4cf9b90bd15dfdef699d', |
| | u'created_at': u'2015-09-02T10:41:07', |
| | u'step_type': u'Engine: create cluster', |
| | u'updated_at': u'2015-09-02T10:41:12', |
| | u'cluster_id': u'9b094131-a858-4ddb- |
| | 81a8-b71597417cad', u'step_name': u'Wait for |
| | instances to become active', u'total': 3, |
| | u'id': u'34b4b23e- |
| | dc94-4253-bb36-d343a4ec1e57'}, {u'successful': |
| | True, u'tenant_id': |
| | u'c82e4bce56ce4cf9b90bd15dfdef699d', |
| | u'created_at': u'2015-09-02T10:41:05', |
| | u'step_type': u'Engine: create cluster', |
| | u'updated_at': u'2015-09-02T10:41:07', |
| | u'cluster_id': u'9b094131-a858-4ddb- |
| | 81a8-b71597417cad', u'step_name': u'Run |
| | instances', u'total': 3, u'id': u'401f6812 |
| | -d92c-44f0-acfe-f22f4dc1c3fe'}, {u'successful': |
| | True, u'tenant_id': |
| | u'c82e4bce56ce4cf9b90bd15dfdef699d', |
| | u'created_at': u'2015-09-02T10:52:12', |
| | u'step_type': u'Plugin: start cluster', |
| | u'updated_at': u'2015-09-02T10:55:02', |
| | u'cluster_id': u'9b094131-a858-4ddb- |
| | 81a8-b71597417cad', u'step_name': u'Await |
| | DataNodes start up', u'total': 1, u'id': u |
| | '407379af-94a4-4821-9952-14a21be06ebc'}, |
| | {u'successful': True, u'tenant_id': |
| | u'c82e4bce56ce4cf9b90bd15dfdef699d', |
| | u'created_at': u'2015-09-02T10:41:13', |
| | u'step_type': u'Engine: create cluster', |
| | u'updated_at': u'2015-09-02T10:48:21', |
| | u'cluster_id': u'9b094131-a858-4ddb- |
| | 81a8-b71597417cad', u'step_name': u'Wait for |
| | instance accessibility', u'total': 3, u'id': |
| | u'534a3a7b-2678-44f4-9562-f859fef00b1f'}, |
| | {u'successful': True, u'tenant_id': |
| | u'c82e4bce56ce4cf9b90bd15dfdef699d', |
| | u'created_at': u'2015-09-02T10:51:43', |
| | u'step_type': u'Plugin: start cluster', |
| | u'updated_at': u'2015-09-02T10:52:12', |
| | u'cluster_id': u'9b094131-a858-4ddb- |
| | 81a8-b71597417cad', u'step_name': u'Start the |
| | following process(es): DataNodes, |
| | NodeManagers', u'total': 2, u'id': u'628a995c- |
| | 316c-4eed-acbf-17076ffa34db'}, {u'successful': |
| | True, u'tenant_id': |
| | u'c82e4bce56ce4cf9b90bd15dfdef699d', |
| | u'created_at': u'2015-09-02T10:48:21', |
| | u'step_type': u'Engine: create cluster', |
| | u'updated_at': u'2015-09-02T10:48:33', |
| | u'cluster_id': u'9b094131-a858-4ddb- |
| | 81a8-b71597417cad', u'step_name': u'Configure |
| | instances', u'total': 3, u'id': u'7fa3987a- |
| | 636f-48a5-a34c-7a6ecd6b5a44'}, {u'successful': |
| | True, u'tenant_id': |
| | u'c82e4bce56ce4cf9b90bd15dfdef699d', |
| | u'created_at': u'2015-09-02T10:50:26', |
| | u'step_type': u'Plugin: start cluster', |
| | u'updated_at': u'2015-09-02T10:51:30', |
| | u'cluster_id': u'9b094131-a858-4ddb- |
| | 81a8-b71597417cad', u'step_name': u'Start the |
| | following process(es): NameNode', u'total': 1, |
| | u'id': u'8988c41f-9bef-484a- |
| | bd93-58700f55f82b'}, {u'successful': True, |
| | u'tenant_id': |
| | u'c82e4bce56ce4cf9b90bd15dfdef699d', |
| | u'created_at': u'2015-09-02T10:50:14', |
| | u'step_type': u'Plugin: configure cluster', |
| | u'updated_at': u'2015-09-02T10:50:25', |
| | u'cluster_id': u'9b094131-a858-4ddb- |
| | 81a8-b71597417cad', u'step_name': u'Configure |
| | topology data', u'total': 1, u'id': |
| | u'bc20afb9-c44a-4825-9ac2-8bd69bf7efcc'}, |
| | {u'successful': True, u'tenant_id': |
| | u'c82e4bce56ce4cf9b90bd15dfdef699d', |
| | u'created_at': u'2015-09-02T10:48:33', |
| | u'step_type': u'Plugin: configure cluster', |
| | u'updated_at': u'2015-09-02T10:50:14', |
| | u'cluster_id': u'9b094131-a858-4ddb- |
| | 81a8-b71597417cad', u'step_name': u'Configure |
| | instances', u'total': 3, u'id': u'c0a3f2ac- |
| | 508f-4ef4-ac87-db82a4999795'}, {u'successful': |
| | True, u'tenant_id': |
| | u'c82e4bce56ce4cf9b90bd15dfdef699d', |
| | u'created_at': u'2015-09-02T10:55:02', |
| | u'step_type': u'Plugin: start cluster', |
| | u'updated_at': u'2015-09-02T10:58:01', |
| | u'cluster_id': u'9b094131-a858-4ddb- |
| | 81a8-b71597417cad', u'step_name': u'Start the |
| | following process(es): HiveServer', u'total': |
| | 1, u'id': u'd5ab5d4c-b8e7-4fe0-b36f- |
| | 116861bdfcb3'}, {u'successful': True, |
| | u'tenant_id': |
| | u'c82e4bce56ce4cf9b90bd15dfdef699d', |
| | u'created_at': u'2015-09-02T10:41:13', |
| | u'step_type': u'Engine: create cluster', |
| | u'updated_at': u'2015-09-02T10:41:13', |
| | u'cluster_id': u'9b094131-a858-4ddb- |
| | 81a8-b71597417cad', u'step_name': u'Assign |
| | IPs', u'total': 3, u'id': |
| | u'd6848957-6206-4116-a310-ec458e651c12'}, |
| | {u'successful': True, u'tenant_id': |
| | u'c82e4bce56ce4cf9b90bd15dfdef699d', |
| | u'created_at': u'2015-09-02T10:51:30', |
| | u'step_type': u'Plugin: start cluster', |
| | u'updated_at': u'2015-09-02T10:51:43', |
| | u'cluster_id': u'9b094131-a858-4ddb- |
| | 81a8-b71597417cad', u'step_name': u'Start the |
| | following process(es): ResourceManager', |
| | u'total': 1, u'id': u'dcd433e3-017a- |
| | 430a-8217-94cae4b813c2'}] |
| use_autoconfig | True |
| anti_affinity | [] |
| node_groups | [{u'count': 1, u'name': u'master', |
| | u'instances': [], u'volume_mount_prefix': |
| | u'/volumes/disk', u'created_at': u'2015-03-17 |
| | 18:33:42', u'updated_at': None, |
| | u'floating_ip_pool': u'70b8c139-096b-4b3b-b29f- |
| | f42b16316758', u'image_id': None, |
| | u'volumes_size': 0, u'node_configs': {}, |
| | u'node_group_template_id': u'09946a01-7973-4f63 |
| | -9aca-7fc6d498d8a6', u'volumes_per_node': 0, |
| | u'node_processes': [u'jobtracker', |
| | u'namenode'], u'auto_security_group': True, |
| | u'security_groups': None, u'flavor_id': u'2'}, |
| | {u'count': 2, u'name': u'workers', |
| | u'instances': [], u'volume_mount_prefix': |
| | u'/volumes/disk', u'created_at': u'2015-03-17 |
| | 18:33:42', u'updated_at': None, |
| | u'floating_ip_pool': u'70b8c139-096b-4b3b-b29f- |
| | f42b16316758', u'image_id': None, |
| | u'volumes_size': 0, u'node_configs': {}, |
| | u'node_group_template_id': u'ceb017bd-0568-42e9 |
| | -890b-03eb298dc99f', u'volumes_per_node': 0, |
| | u'node_processes': [u'tasktracker', |
| | u'datanode'], u'auto_security_group': True, |
| | u'security_groups': None, u'flavor_id': u'2'}] |
| management_public_key | ssh-rsa BBBBB3NzaB1yc2EAAAADAQABAAABAQCziEF+3oJ |
| | ki6Fd1rvuiducJ470DN9ZFagiFbLfcwqu7TNKee10uice5P |
| | KmvpusXMaL5LiZFTHafbFJfNUlah90yGpfsYqbcx2dMNqoU |
| | EF4ZvEVO7RVU8jCe7DXBEkBFGQ1x/v17vyaxIJ8AqnFVSuu |
| | FgfcHuihLAC250ZlfNWMcoFhUy6MsBocoxCF6MVal5Xt8nw |
| | Y8o8xTQwd/f4wbAeAE3P0TaOCpXpMxxLL/hMDALekdxs1Gh |
| | Mk0k5rbj4oD9AKx8+/jucIxS6mmwqWwwqo7jmy2jIsukOGZ |
| | 1LdeNe0ctOX56k1LoZybzMzT6NbgUwfuIRbOwuryy2QbWwV |
| | gX6t Generated by Sahara |
| node_groups | [{u'volume_local_to_instance': False, |
| | u'availability_zone': None, u'updated_at': |
| | u'2015-09-02T10:41:06', u'instances': |
| | [{u'instance_id': u'949da8aa-7c9e-48b3-882e- |
| | 0c7a0049100e', u'created_at': |
| | u'2015-09-02T10:41:06', u'updated_at': |
| | u'2015-09-02T10:41:13', u'instance_name': |
| | u'cluster-3-master-001', u'management_ip': |
| | u'192.168.1.134', u'internal_ip': |
| | u'172.24.17.2', u'id': u'e27503e8-a118-4c3e- |
| | a7d7-ee64fcd4568a'}], |
| | u'node_group_template_id': u'9d3b5b2c- |
| | d5d5-4d16-8a93-a568d29c6569', |
| | u'volumes_per_node': 0, u'id': u'6a53f95a-c2aa- |
| | 48d7-b43a-62d149c656af', u'security_groups': |
| | [6], u'shares': None, u'node_configs': |
| | {u'MapReduce': {u'mapreduce.map.memory.mb': |
| | 256, u'mapreduce.reduce.memory.mb': 512, |
| | u'yarn.app.mapreduce.am.command-opts': |
| | u'-Xmx204m', u'mapreduce.reduce.java.opts': |
| | u'-Xmx409m', |
| | u'yarn.app.mapreduce.am.resource.mb': 256, |
| | u'mapreduce.map.java.opts': u'-Xmx204m', |
| | u'mapreduce.task.io.sort.mb': 102}, u'YARN': |
| | {u'yarn.scheduler.minimum-allocation-mb': 256, |
| | u'yarn.scheduler.maximum-allocation-mb': 2048, |
| | u'yarn.nodemanager.vmem-check-enabled': |
| | u'false', u'yarn.nodemanager.resource.memory- |
| | mb': 2048}}, u'auto_security_group': True, |
| | u'volumes_availability_zone': None, |
| | u'volume_mount_prefix': u'/volumes/disk', |
| | u'floating_ip_pool': u'public', u'image_id': |
| | None, u'volumes_size': 0, u'is_proxy_gateway': |
| | False, u'count': 1, u'name': u'master', |
| | u'created_at': u'2015-09-02T10:41:02', |
| | u'volume_type': None, u'node_processes': |
| | [u'namenode', u'resourcemanager', |
| | u'hiveserver'], u'flavor_id': u'2', |
| | u'use_autoconfig': True}, |
| | {u'volume_local_to_instance': False, |
| | u'availability_zone': None, u'updated_at': |
| | u'2015-09-02T10:41:07', u'instances': |
| | [{u'instance_id': u'47f97841-4a17-4e18-a8eb- |
| | b4ff7dd4c3d8', u'created_at': |
| | u'2015-09-02T10:41:06', u'updated_at': |
| | u'2015-09-02T10:41:13', u'instance_name': |
| | u'cluster-3-worker-001', u'management_ip': |
| | u'192.168.1.135', u'internal_ip': |
| | u'172.24.17.3', u'id': u'c4a02678-113b-432e- |
| | 8f91-927b8e7cfe83'}, {u'instance_id': |
| | u'a02aea39-cc1f-4a1f-8232-2470ab6e8478', |
| | u'created_at': u'2015-09-02T10:41:07', |
| | u'updated_at': u'2015-09-02T10:41:13', |
| | u'instance_name': u'cluster-3-worker-002', |
| | u'management_ip': u'192.168.1.130', |
| | u'internal_ip': u'172.24.17.4', u'id': u |
| | 'b7b2d6db-cd50-484b-8036-09820d2623f2'}], |
| | u'node_group_template_id': u'1aa4a397-cb1e- |
| | 4f38-be18-7f65fa0cc2eb', u'volumes_per_node': |
| | 0, u'id': u'b666103f-a44b-4cf8-b3ae- |
| | 7d2623c6cd18', u'security_groups': [7], |
| | u'shares': None, u'node_configs': |
| | {u'MapReduce': {u'mapreduce.map.memory.mb': |
| | 256, u'mapreduce.reduce.memory.mb': 512, |
| | u'yarn.app.mapreduce.am.command-opts': |
| | u'-Xmx204m', u'mapreduce.reduce.java.opts': |
| | u'-Xmx409m', |
| | u'yarn.app.mapreduce.am.resource.mb': 256, |
| | u'mapreduce.map.java.opts': u'-Xmx204m', |
| | u'mapreduce.task.io.sort.mb': 102}, u'YARN': |
| | {u'yarn.scheduler.minimum-allocation-mb': 256, |
| | u'yarn.scheduler.maximum-allocation-mb': 2048, |
| | u'yarn.nodemanager.vmem-check-enabled': |
| | u'false', u'yarn.nodemanager.resource.memory- |
| | mb': 2048}}, u'auto_security_group': True, |
| | u'volumes_availability_zone': None, |
| | u'volume_mount_prefix': u'/volumes/disk', |
| | u'floating_ip_pool': u'public', u'image_id': |
| | None, u'volumes_size': 0, u'is_proxy_gateway': |
| | False, u'count': 2, u'name': u'worker', |
| | u'created_at': u'2015-09-02T10:41:02', |
| | u'volume_type': None, u'node_processes': |
| | [u'nodemanager', u'datanode'], u'flavor_id': |
| | u'2', u'use_autoconfig': True}] |
| is_public | False |
| management_public_key | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDiFXlWNVD |
| | 6gJT74wherHWtgchqpvgi2aJ4fPWXP+WgB4GEKpfD7a/dWu |
| | Qg9eDBQIrWvVsKgG1i9YgRTHOQ7DdwoSKUAcpEewgw927ER |
| | wdJ3IV7EDu0xENUgrUgp+CwPdk94SXPg1G4oHOCbOvJYcW6 |
| | /b8Ci86vH9A7Uyu2T7tbVS4ciMKfwI0Z47lzcp2qDV6W8M7 |
| | neghC1mNT4k29ghgcYOzY4SxQjxp1a5Iu6RtnJ2fvHbLeMS |
| | 0hgeobSZ8heQzLImrp2dbyZy74goOcwKtk9dDPV853aZrjL |
| | yOsc78EgW6n2Gugu7Ks12v9QEDr4H3yTt3DNTrB5Y8tt468 |
| | k2n1 Generated-by-Sahara |
| status_description | |
| hadoop_version | 1.2.1 |
| id | c5e755a2-b3f9-417b-948b-e99ed7fbf1e3 |
| hadoop_version | <plugin_version> |
| id | 9b094131-a858-4ddb-81a8-b71597417cad |
| trust_id | None |
| info | {} |
| cluster_template_id | c0609da7-faac-4dcf-9cbc-858a3aa130cd |
| name | cluster-1 |
| cluster_configs | {} |
| created_at | 2013-07-07T19:01:51 |
| default_image_id | 3f9fc974-b484-4756-82a4-bff9e116919b |
| tenant_id | 3fd7266fb3b547b1a45307b481bcadfd |
| info | {u'HDFS': {u'NameNode': |
| | u'hdfs://cluster-3-master-001:9000', u'Web UI': |
| | u'http://192.168.1.134:50070'}, u'YARN': {u'Web |
| | UI': u'http://192.168.1.134:8088', |
| | u'ResourceManager': |
| | u'http://192.168.1.134:8032'}} |
| cluster_template_id | 74add4df-07c2-4053-931f-d5844712727f |
| name | my-cluster-1 |
| cluster_configs | {u'HDFS': {u'dfs.replication': 2}} |
| created_at | 2015-09-02T10:41:02 |
| default_image_id | c119f99c-67f2-4404-9cff-f30e4b185036 |
| shares | None |
| is_protected | False |
| tenant_id | c82e4bce56ce4cf9b90bd15dfdef699d |
+----------------------------+-------------------------------------------------+
Verify the cluster launched successfully by using the ``sahara`` command
@ -366,27 +574,27 @@ line tool as follows:
.. sourcecode:: console
$ sahara cluster-list
+-----------+--------------------------------------+--------+------------+
+--------------+--------------------------------------+--------+------------+
| name | id | status | node_count |
+-----------+--------------------------------------+--------+------------+
| cluster-1 | c5e755a2-b3f9-417b-948b-e99ed7fbf1e3 | Active | 3 |
+-----------+--------------------------------------+--------+------------+
+--------------+--------------------------------------+--------+------------+
| my-cluster-1 | 9b094131-a858-4ddb-81a8-b71597417cad | Active | 3 |
+--------------+--------------------------------------+--------+------------+
The cluster creation operation may take several minutes to complete. During
this time the "status" returned from the previous command may show states
other than "Active".
8. Run a MapReduce job
----------------------
8. Run a MapReduce job to check Hadoop installation
---------------------------------------------------
Check that your Hadoop installation is working properly by running an
example job on the cluster manually.
* Login to the NameNode via ssh:
* Login to NameNode (usually master node) via ssh with ssh-key used above:
.. sourcecode:: console
$ ssh ubuntu@<namenode_ip>
$ ssh -i my_stack.pem ubuntu@<namenode_ip>
* Switch to the hadoop user:
@ -398,8 +606,8 @@ example job on the cluster manually.
.. sourcecode:: console
$ cd /usr/share/hadoop
$ hadoop jar hadoop-examples-1.2.1.jar pi 10 100
$ cd /opt/hadoop-<plugin_version>/share/hadoop/mapreduce
$ /opt/hadoop-<plugin_version>/bin/hadoop jar hadoop-mapreduce-examples-<plugin_version>.jar pi 10 100
Congratulations! Now you have the Hadoop cluster ready on the OpenStack
cloud.
Congratulations! Your Hadoop cluster is ready to use, running on your
OpenStack cloud.