From b0c1d8ba0ab0329d1bef2d5b8bf2745ddb3738d0 Mon Sep 17 00:00:00 2001 From: Ashish Billore Date: Thu, 12 Nov 2015 19:26:41 +0530 Subject: [PATCH] Update Sahara Dev Quickstart Guide Clean-up of old references generalized version Change-Id: Ie4e6554cd283cd81f56e268f3a99da3120aa4d19 Closes-Bug: #1485648 --- doc/source/devref/quickstart.rst | 546 +++++++++++++++++++++---------- 1 file changed, 377 insertions(+), 169 deletions(-) diff --git a/doc/source/devref/quickstart.rst b/doc/source/devref/quickstart.rst index 8c4832fb..4e80916a 100644 --- a/doc/source/devref/quickstart.rst +++ b/doc/source/devref/quickstart.rst @@ -12,10 +12,9 @@ of OpenStack command line tools and the sahara :doc:`REST API <../restapi>`. OR -* If you just want to install and use Sahara follow +* If you just want to install and use sahara follow :doc:`../userdoc/installation.guide`. - 2. Identity service configuration --------------------------------- @@ -27,18 +26,17 @@ whose password is ``nova``: .. sourcecode:: console - $ export OS_AUTH_URL=http://127.0.0.1:5000/v2.0/ - $ export OS_TENANT_NAME=admin - $ export OS_USERNAME=admin - $ export OS_PASSWORD=nova + $ export OS_AUTH_URL=http://127.0.0.1:5000/v2.0/ + $ export OS_TENANT_NAME=admin + $ export OS_USERNAME=admin + $ export OS_PASSWORD=nova With these environment variables set you can get an authentication token using the ``keystone`` command line client as follows: .. sourcecode:: console - $ keystone token-get - + $ keystone token-get If authentication succeeds, the output will be as follows: @@ -47,10 +45,10 @@ If authentication succeeds, the output will be as follows: +-----------+----------------------------------+ | Property | Value | +-----------+----------------------------------+ - | expires | 2013-07-08T15:21:18Z | - | id | dd92e3cdb4e1462690cd444d6b01b746 | - | tenant_id | 62bd2046841e4e94a87b4a22aa886c13 | - | user_id | 720fb87141a14fd0b204f977f5f02512 | + | expires | 2015-09-03T13:37:32Z | + | id | 2542c427092a4b09a07ee7612c3d99ae | + | tenant_id | c82e4bce56ce4cf9b90bd15dfdef699d | + | user_id | 7f5becaaa38b4c9e850ccd11672a4c96 | +-----------+----------------------------------+ The ``id`` and ``tenant_id`` values will be used for creating REST calls @@ -61,40 +59,50 @@ variables for ease of use later. .. sourcecode:: console - $ export AUTH_TOKEN="dd92e3cdb4e1462690cd444d6b01b746" - $ export TENANT_ID="62bd2046841e4e94a87b4a22aa886c13" + $ export AUTH_TOKEN="2542c427092a4b09a07ee7612c3d99ae" + $ export TENANT_ID="c82e4bce56ce4cf9b90bd15dfdef699d" +Alternatively, if a devstack environment is used, these values are available +through "openrc" file under the "devstack_install_root" directory and can be +configured as: + +.. sourcecode:: console + + $ source /openrc 3. Upload an image to the Image service --------------------------------------- You will need to upload a virtual machine image to the OpenStack Image service. You can download pre-built images with vanilla Apache Hadoop -installed, or build the images yourself: +installed, or build the images yourself. This guide uses the latest available +Ubuntu upstream image, referred to as ``sahara-vanilla-latest-ubuntu.qcow2`` +and the latest version of vanilla plugin as an example. +Sample images are available here: -* Download and install a pre-built image with Ubuntu 13.10 +`Sample Images `_ + +* Download a pre-built image + +**Note:** For the steps below, substitute ```` with the +appropriate OpenStack release and ```` with the image of your +choice. .. sourcecode:: console - $ ssh user@hostname - $ wget http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-ubuntu-13.10.qcow2 - $ glance image-create --name=sahara-icehouse-vanilla-1.2.1-ubuntu-13.10 \ - --disk-format=qcow2 --container-format=bare < ./sahara-icehouse-vanilla-1.2.1-ubuntu-13.10.qcow2 + $ ssh user@hostname + $ wget http://sahara-files.mirantis.com/images/upstream//.qcow2 -OR - -* with Fedora 20 +Upload the above downloaded image into the OpenStack Image service: .. sourcecode:: console - $ ssh user@hostname - $ wget http://sahara-files.mirantis.com/sahara-icehouse-vanilla-1.2.1-fedora-20.qcow2 - $ glance image-create --name=sahara-icehouse-vanilla-1.2.1-fedora-20 \ - --disk-format=qcow2 --container-format=bare < ./sahara-icehouse-vanilla-1.2.1-fedora-20.qcow2 + $ glance image-create --name=sahara-vanilla-latest-ubuntu \ + --disk-format=qcow2 --container-format=bare < ./sahara-vanilla-latest-ubuntu.qcow2 OR -* build the image using :doc:`../userdoc/diskimagebuilder`. +* Build the image using: `diskimage-builder script `_ Save the image id, this will be used during the image registration with sahara. You can get the image id using the ``glance`` command line tool @@ -102,15 +110,14 @@ as follows: .. sourcecode:: console - $ glance image-list --name sahara-icehouse-vanilla-1.2.1-ubuntu-13.10 - +--------------------------------------+---------------------------------------------+ - | ID | Name | - +--------------------------------------+---------------------------------------------+ - | 3f9fc974-b484-4756-82a4-bff9e116919b | sahara-icehouse-vanilla-1.2.1-ubuntu-13.10 | - +--------------------------------------+---------------------------------------------+ - - $ export IMAGE_ID="3f9fc974-b484-4756-82a4-bff9e116919b" + $ glance image-list --name sahara-vanilla-latest-ubuntu + +--------------------------------------+-------------------------------------+ + | ID | Name | + +--------------------------------------+-------------------------------------+ + | c119f99c-67f2-4404-9cff-f30e4b185036 | sahara-vanilla-latest-ubuntu | + +--------------------------------------+-------------------------------------+ + $ export IMAGE_ID="c119f99c-67f2-4404-9cff-f30e4b185036" 4. Register the image with the sahara image registry ---------------------------------------------------- @@ -124,63 +131,82 @@ will vary depending on the source image used, for more please see* .. sourcecode:: console - $ sahara image-register --id $IMAGE_ID --username ubuntu + $ sahara image-register --id $IMAGE_ID --username ubuntu -Tag the image to inform sahara about the plugin with which it shall be used: +Tag the image to inform sahara about the plugin and the version with which +it shall be used. + +**Note:** For the steps below and the rest of this guide, substitute +```` with the appropriate version of your plugin. .. sourcecode:: console - $ sahara image-add-tag --id $IMAGE_ID --tag vanilla - $ sahara image-add-tag --id $IMAGE_ID --tag 1.2.1 + $ sahara image-add-tag --id $IMAGE_ID --tag vanilla + $ sahara image-add-tag --id $IMAGE_ID --tag Ensure that the image is registered correctly by querying sahara. If registered successfully, the image will appear in the output as follows: .. sourcecode:: console - $ sahara image-list - +--------------------------------------------+---------------------------------------+----------+----------------+-------------+ - | name | id | username | tags | description | - +--------------------------------------------+---------------------------------------+----------+----------------+-------------+ - | sahara-icehouse-vanilla-1.2.1-ubuntu-13.10 | 3f9fc974-b484-4756-82a4-bff9e116919b | ubuntu | vanilla, 1.2.1 | None | - +--------------------------------------------+---------------------------------------+----------+----------------+-------------+ - + $ sahara image-list + +------------------------------+--------------------------------------+----------+---------------------------+-------------+ + | name | id | username | tags | description | + +------------------------------+--------------------------------------+----------+---------------------------+-------------+ + | sahara-vanilla-latest-ubuntu | c119f99c-67f2-4404-9cff-f30e4b185036 | ubuntu | vanilla, | None | + +------------------------------+--------------------------------------+----------+---------------------------+-------------+ 5. Create node group templates ------------------------------ Node groups are the building blocks of clusters in sahara. Before you can begin provisioning clusters you must define a few node group templates to -describe their configurations. +describe node group configurations. -*Note, these templates assume that floating IP addresses are not being -used, for more information please see* :ref:`floating_ip_management` +*Note, these templates assume that floating IP addresses are being used. For +more details on floating IP please see* :ref:`floating_ip_management` -Create a file named ``ng_master_template_create.json`` with the following +If your environment does not use floating IP, omit defining floating IP in +the template below. + +Sample templates can be found here: + +`Sample Templates `_ + +Create a file named ``my_master_template_create.json`` with the following content: .. sourcecode:: json { - "name": "test-master-tmpl", - "flavor_id": "2", "plugin_name": "vanilla", - "hadoop_version": "1.2.1", - "node_processes": ["jobtracker", "namenode"], + "hadoop_version": "", + "node_processes": [ + "namenode", + "resourcemanager", + "hiveserver" + ], + "name": "vanilla-default-master", + "floating_ip_pool": "public", + "flavor_id": "2", "auto_security_group": true } -Create a file named ``ng_worker_template_create.json`` with the following +Create a file named ``my_worker_template_create.json`` with the following content: .. sourcecode:: json { - "name": "test-worker-tmpl", - "flavor_id": "2", "plugin_name": "vanilla", - "hadoop_version": "1.2.1", - "node_processes": ["tasktracker", "datanode"], + "hadoop_version": "", + "node_processes": [ + "nodemanager", + "datanode" + ], + "name": "vanilla-default-worker", + "floating_ip_pool": "public", + "flavor_id": "2", "auto_security_group": true } @@ -188,28 +214,28 @@ Use the ``sahara`` client to upload the node group templates: .. sourcecode:: console - $ sahara node-group-template-create --json ng_master_template_create.json - $ sahara node-group-template-create --json ng_worker_template_create.json + $ sahara node-group-template-create --json my_master_template_create.json + $ sahara node-group-template-create --json my_worker_template_create.json List the available node group templates to ensure that they have been added properly: .. sourcecode:: console - $ sahara node-group-template-list - +------------------+--------------------------------------+-------------+-----------------------+-------------+ - | name | id | plugin_name | node_processes | description | - +------------------+--------------------------------------+-------------+-----------------------+-------------+ - | test-master-tmpl | b38227dc-64fe-42bf-8792-d1456b453ef3 | vanilla | jobtracker, namenode | None | - | test-worker-tmpl | 634827b9-6a18-4837-ae15-5371d6ecf02c | vanilla | datanode, nodemanager | None | - +------------------+--------------------------------------+-------------+-----------------------+-------------+ + $ sahara node-group-template-list + +------------------------+--------------------------------------+-------------+---------------------------------------+-------------+ + | name | id | plugin_name | node_processes | description | + +------------------------+--------------------------------------+-------------+---------------------------------------+-------------+ + | vanilla-default-master | 9d3b5b2c-d5d5-4d16-8a93-a568d29c6569 | vanilla | namenode, resourcemanager, hiveserver | None | + | vanilla-default-worker | 1aa4a397-cb1e-4f38-be18-7f65fa0cc2eb | vanilla | nodemanager, datanode | None | + +------------------------+--------------------------------------+-------------+---------------------------------------+-------------+ Save the id for the master and worker node group templates as they will be -used during cluster template creation. For example: - -* Master node group template id: ``b38227dc-64fe-42bf-8792-d1456b453ef3`` -* Worker node group template id: ``634827b9-6a18-4837-ae15-5371d6ecf02c`` +used during cluster template creation. +For example: +* Master node group template id: ``9d3b5b2c-d5d5-4d16-8a93-a568d29c6569`` +* Worker node group template id: ``1aa4a397-cb1e-4f38-be18-7f65fa0cc2eb`` 6. Create a cluster template ---------------------------- @@ -217,147 +243,329 @@ used during cluster template creation. For example: The last step before provisioning the cluster is to create a template that describes the node groups of the cluster. -Create a file named ``cluster_template_create.json`` with the following +Create a file named ``my_cluster_template_create.json`` with the following content: .. sourcecode:: json { - "name": "demo-cluster-template", "plugin_name": "vanilla", - "hadoop_version": "1.2.1", + "hadoop_version": "", "node_groups": [ { - "name": "master", - "node_group_template_id": "b38227dc-64fe-42bf-8792-d1456b453ef3", - "count": 1 + "name": "worker", + "count": 2, + "node_group_template_id": "1aa4a397-cb1e-4f38-be18-7f65fa0cc2eb" }, { - "name": "workers", - "node_group_template_id": "634827b9-6a18-4837-ae15-5371d6ecf02c", - "count": 2 + "name": "master", + "count": 1, + "node_group_template_id": "9d3b5b2c-d5d5-4d16-8a93-a568d29c6569" } - ] + ], + "name": "vanilla-default-cluster", + "cluster_configs": {} } Upload the Cluster template using the ``sahara`` command line tool: .. sourcecode:: console - $ sahara cluster-template-create --json cluster_template_create.json + $ sahara cluster-template-create --json my_cluster_template_create.json -Save the template id for use in the cluster provisioning command. The -cluster id can be found in the output of the creation command or by -listing the cluster templates as follows: +Save the cluster template id for use in the cluster provisioning command. The +cluster id can be found in the output of the creation command or by listing +the cluster templates as follows: .. sourcecode:: console $ sahara cluster-template-list - +-----------------------+--------------------------------------+-------------+-----------------------+-------------+ - | name | id | plugin_name | node_groups | description | - +-----------------------+--------------------------------------+-------------+-----------------------+-------------+ - | demo-cluster-template | c0609da7-faac-4dcf-9cbc-858a3aa130cd | vanilla | master: 1, workers: 2 | None | - +-----------------------+--------------------------------------+-------------+-----------------------+-------------+ - + +-------------------------+--------------------------------------+-------------+----------------------+-------------+ + | name | id | plugin_name | node_groups | description | + +-------------------------+--------------------------------------+-------------+----------------------+-------------+ + | vanilla-default-cluster | 74add4df-07c2-4053-931f-d5844712727f | vanilla | master: 1, worker: 2 | None | + +-------------------------+--------------------------------------+-------------+----------------------+-------------+ 7. Create cluster ----------------- -Now you will provision the cluster. This step requires a few pieces of -information that will be found by querying various OpenStack services. +Now you are ready to provision the cluster. This step requires a few pieces of +information that can be found by querying various OpenStack services. -Create a file named ``cluster_create.json`` with the following content: +Create a file named ``my_cluster_create.json`` with the following content: .. sourcecode:: json { - "name": "cluster-1", + "name": "my-cluster-1", "plugin_name": "vanilla", - "hadoop_version": "1.2.1", - "cluster_template_id" : "c0609da7-faac-4dcf-9cbc-858a3aa130cd", - "user_keypair_id": "stack", - "default_image_id": "3f9fc974-b484-4756-82a4-bff9e116919b" + "hadoop_version": "", + "cluster_template_id" : "74add4df-07c2-4053-931f-d5844712727f", + "user_keypair_id": "my_stack", + "default_image_id": "c119f99c-67f2-4404-9cff-f30e4b185036", "neutron_management_network": "8cccf998-85e4-4c5f-8850-63d33c1c6916" } -The parameter ``user_keypair_id`` with the value ``stack`` is generated by +The parameter ``user_keypair_id`` with the value ``my_stack`` is generated by creating a keypair. You can create your own keypair in the OpenStack Dashboard, or through the ``nova`` command line client as follows: .. sourcecode:: console - $ nova keypair-add stack --pub-key $PATH_TO_PUBLIC_KEY + $ nova keypair-add my_stack --pub-key $PATH_TO_PUBLIC_KEY If sahara is configured to use neutron for networking, you will also need to include the ``neutron_management_network`` parameter in -``cluster_create.json``. Cluster instances will get fixed IP addresses in -this network. You can determine the neutron network id with the following -command: +``my_cluster_create.json``. If your environment does not use neutron, you can +omit ``neutron_management_network`` above. You can determine the neutron +network id with the following command: .. sourcecode:: console - $ neutron net-list + $ neutron net-list Create and start the cluster: .. sourcecode:: console - $ sahara cluster-create --json cluster_create.json + $ sahara cluster-create --json my_cluster_create.json +----------------------------+-------------------------------------------------+ | Property | Value | +----------------------------+-------------------------------------------------+ - | status | Validating | - | neutron_management_network | 8cccf998-85e4-4c5f-8850-63d33c1c6916 | + | status | Active | + | neutron_management_network | None | | is_transient | False | | description | None | - | user_keypair_id | stack | - | updated_at | 2013-07-07T19:01:51 | + | user_keypair_id | my_stack | + | updated_at | 2015-09-02T10:58:02 | | plugin_name | vanilla | + | provision_progress | [{u'successful': True, u'tenant_id': | + | | u'c82e4bce56ce4cf9b90bd15dfdef699d', | + | | u'created_at': u'2015-09-02T10:41:07', | + | | u'step_type': u'Engine: create cluster', | + | | u'updated_at': u'2015-09-02T10:41:12', | + | | u'cluster_id': u'9b094131-a858-4ddb- | + | | 81a8-b71597417cad', u'step_name': u'Wait for | + | | instances to become active', u'total': 3, | + | | u'id': u'34b4b23e- | + | | dc94-4253-bb36-d343a4ec1e57'}, {u'successful': | + | | True, u'tenant_id': | + | | u'c82e4bce56ce4cf9b90bd15dfdef699d', | + | | u'created_at': u'2015-09-02T10:41:05', | + | | u'step_type': u'Engine: create cluster', | + | | u'updated_at': u'2015-09-02T10:41:07', | + | | u'cluster_id': u'9b094131-a858-4ddb- | + | | 81a8-b71597417cad', u'step_name': u'Run | + | | instances', u'total': 3, u'id': u'401f6812 | + | | -d92c-44f0-acfe-f22f4dc1c3fe'}, {u'successful': | + | | True, u'tenant_id': | + | | u'c82e4bce56ce4cf9b90bd15dfdef699d', | + | | u'created_at': u'2015-09-02T10:52:12', | + | | u'step_type': u'Plugin: start cluster', | + | | u'updated_at': u'2015-09-02T10:55:02', | + | | u'cluster_id': u'9b094131-a858-4ddb- | + | | 81a8-b71597417cad', u'step_name': u'Await | + | | DataNodes start up', u'total': 1, u'id': u | + | | '407379af-94a4-4821-9952-14a21be06ebc'}, | + | | {u'successful': True, u'tenant_id': | + | | u'c82e4bce56ce4cf9b90bd15dfdef699d', | + | | u'created_at': u'2015-09-02T10:41:13', | + | | u'step_type': u'Engine: create cluster', | + | | u'updated_at': u'2015-09-02T10:48:21', | + | | u'cluster_id': u'9b094131-a858-4ddb- | + | | 81a8-b71597417cad', u'step_name': u'Wait for | + | | instance accessibility', u'total': 3, u'id': | + | | u'534a3a7b-2678-44f4-9562-f859fef00b1f'}, | + | | {u'successful': True, u'tenant_id': | + | | u'c82e4bce56ce4cf9b90bd15dfdef699d', | + | | u'created_at': u'2015-09-02T10:51:43', | + | | u'step_type': u'Plugin: start cluster', | + | | u'updated_at': u'2015-09-02T10:52:12', | + | | u'cluster_id': u'9b094131-a858-4ddb- | + | | 81a8-b71597417cad', u'step_name': u'Start the | + | | following process(es): DataNodes, | + | | NodeManagers', u'total': 2, u'id': u'628a995c- | + | | 316c-4eed-acbf-17076ffa34db'}, {u'successful': | + | | True, u'tenant_id': | + | | u'c82e4bce56ce4cf9b90bd15dfdef699d', | + | | u'created_at': u'2015-09-02T10:48:21', | + | | u'step_type': u'Engine: create cluster', | + | | u'updated_at': u'2015-09-02T10:48:33', | + | | u'cluster_id': u'9b094131-a858-4ddb- | + | | 81a8-b71597417cad', u'step_name': u'Configure | + | | instances', u'total': 3, u'id': u'7fa3987a- | + | | 636f-48a5-a34c-7a6ecd6b5a44'}, {u'successful': | + | | True, u'tenant_id': | + | | u'c82e4bce56ce4cf9b90bd15dfdef699d', | + | | u'created_at': u'2015-09-02T10:50:26', | + | | u'step_type': u'Plugin: start cluster', | + | | u'updated_at': u'2015-09-02T10:51:30', | + | | u'cluster_id': u'9b094131-a858-4ddb- | + | | 81a8-b71597417cad', u'step_name': u'Start the | + | | following process(es): NameNode', u'total': 1, | + | | u'id': u'8988c41f-9bef-484a- | + | | bd93-58700f55f82b'}, {u'successful': True, | + | | u'tenant_id': | + | | u'c82e4bce56ce4cf9b90bd15dfdef699d', | + | | u'created_at': u'2015-09-02T10:50:14', | + | | u'step_type': u'Plugin: configure cluster', | + | | u'updated_at': u'2015-09-02T10:50:25', | + | | u'cluster_id': u'9b094131-a858-4ddb- | + | | 81a8-b71597417cad', u'step_name': u'Configure | + | | topology data', u'total': 1, u'id': | + | | u'bc20afb9-c44a-4825-9ac2-8bd69bf7efcc'}, | + | | {u'successful': True, u'tenant_id': | + | | u'c82e4bce56ce4cf9b90bd15dfdef699d', | + | | u'created_at': u'2015-09-02T10:48:33', | + | | u'step_type': u'Plugin: configure cluster', | + | | u'updated_at': u'2015-09-02T10:50:14', | + | | u'cluster_id': u'9b094131-a858-4ddb- | + | | 81a8-b71597417cad', u'step_name': u'Configure | + | | instances', u'total': 3, u'id': u'c0a3f2ac- | + | | 508f-4ef4-ac87-db82a4999795'}, {u'successful': | + | | True, u'tenant_id': | + | | u'c82e4bce56ce4cf9b90bd15dfdef699d', | + | | u'created_at': u'2015-09-02T10:55:02', | + | | u'step_type': u'Plugin: start cluster', | + | | u'updated_at': u'2015-09-02T10:58:01', | + | | u'cluster_id': u'9b094131-a858-4ddb- | + | | 81a8-b71597417cad', u'step_name': u'Start the | + | | following process(es): HiveServer', u'total': | + | | 1, u'id': u'd5ab5d4c-b8e7-4fe0-b36f- | + | | 116861bdfcb3'}, {u'successful': True, | + | | u'tenant_id': | + | | u'c82e4bce56ce4cf9b90bd15dfdef699d', | + | | u'created_at': u'2015-09-02T10:41:13', | + | | u'step_type': u'Engine: create cluster', | + | | u'updated_at': u'2015-09-02T10:41:13', | + | | u'cluster_id': u'9b094131-a858-4ddb- | + | | 81a8-b71597417cad', u'step_name': u'Assign | + | | IPs', u'total': 3, u'id': | + | | u'd6848957-6206-4116-a310-ec458e651c12'}, | + | | {u'successful': True, u'tenant_id': | + | | u'c82e4bce56ce4cf9b90bd15dfdef699d', | + | | u'created_at': u'2015-09-02T10:51:30', | + | | u'step_type': u'Plugin: start cluster', | + | | u'updated_at': u'2015-09-02T10:51:43', | + | | u'cluster_id': u'9b094131-a858-4ddb- | + | | 81a8-b71597417cad', u'step_name': u'Start the | + | | following process(es): ResourceManager', | + | | u'total': 1, u'id': u'dcd433e3-017a- | + | | 430a-8217-94cae4b813c2'}] | + | use_autoconfig | True | | anti_affinity | [] | - | node_groups | [{u'count': 1, u'name': u'master', | - | | u'instances': [], u'volume_mount_prefix': | - | | u'/volumes/disk', u'created_at': u'2015-03-17 | - | | 18:33:42', u'updated_at': None, | - | | u'floating_ip_pool': u'70b8c139-096b-4b3b-b29f- | - | | f42b16316758', u'image_id': None, | - | | u'volumes_size': 0, u'node_configs': {}, | - | | u'node_group_template_id': u'09946a01-7973-4f63 | - | | -9aca-7fc6d498d8a6', u'volumes_per_node': 0, | - | | u'node_processes': [u'jobtracker', | - | | u'namenode'], u'auto_security_group': True, | - | | u'security_groups': None, u'flavor_id': u'2'}, | - | | {u'count': 2, u'name': u'workers', | - | | u'instances': [], u'volume_mount_prefix': | - | | u'/volumes/disk', u'created_at': u'2015-03-17 | - | | 18:33:42', u'updated_at': None, | - | | u'floating_ip_pool': u'70b8c139-096b-4b3b-b29f- | - | | f42b16316758', u'image_id': None, | - | | u'volumes_size': 0, u'node_configs': {}, | - | | u'node_group_template_id': u'ceb017bd-0568-42e9 | - | | -890b-03eb298dc99f', u'volumes_per_node': 0, | - | | u'node_processes': [u'tasktracker', | - | | u'datanode'], u'auto_security_group': True, | - | | u'security_groups': None, u'flavor_id': u'2'}] | - | management_public_key | ssh-rsa BBBBB3NzaB1yc2EAAAADAQABAAABAQCziEF+3oJ | - | | ki6Fd1rvuiducJ470DN9ZFagiFbLfcwqu7TNKee10uice5P | - | | KmvpusXMaL5LiZFTHafbFJfNUlah90yGpfsYqbcx2dMNqoU | - | | EF4ZvEVO7RVU8jCe7DXBEkBFGQ1x/v17vyaxIJ8AqnFVSuu | - | | FgfcHuihLAC250ZlfNWMcoFhUy6MsBocoxCF6MVal5Xt8nw | - | | Y8o8xTQwd/f4wbAeAE3P0TaOCpXpMxxLL/hMDALekdxs1Gh | - | | Mk0k5rbj4oD9AKx8+/jucIxS6mmwqWwwqo7jmy2jIsukOGZ | - | | 1LdeNe0ctOX56k1LoZybzMzT6NbgUwfuIRbOwuryy2QbWwV | - | | gX6t Generated by Sahara | + | node_groups | [{u'volume_local_to_instance': False, | + | | u'availability_zone': None, u'updated_at': | + | | u'2015-09-02T10:41:06', u'instances': | + | | [{u'instance_id': u'949da8aa-7c9e-48b3-882e- | + | | 0c7a0049100e', u'created_at': | + | | u'2015-09-02T10:41:06', u'updated_at': | + | | u'2015-09-02T10:41:13', u'instance_name': | + | | u'cluster-3-master-001', u'management_ip': | + | | u'192.168.1.134', u'internal_ip': | + | | u'172.24.17.2', u'id': u'e27503e8-a118-4c3e- | + | | a7d7-ee64fcd4568a'}], | + | | u'node_group_template_id': u'9d3b5b2c- | + | | d5d5-4d16-8a93-a568d29c6569', | + | | u'volumes_per_node': 0, u'id': u'6a53f95a-c2aa- | + | | 48d7-b43a-62d149c656af', u'security_groups': | + | | [6], u'shares': None, u'node_configs': | + | | {u'MapReduce': {u'mapreduce.map.memory.mb': | + | | 256, u'mapreduce.reduce.memory.mb': 512, | + | | u'yarn.app.mapreduce.am.command-opts': | + | | u'-Xmx204m', u'mapreduce.reduce.java.opts': | + | | u'-Xmx409m', | + | | u'yarn.app.mapreduce.am.resource.mb': 256, | + | | u'mapreduce.map.java.opts': u'-Xmx204m', | + | | u'mapreduce.task.io.sort.mb': 102}, u'YARN': | + | | {u'yarn.scheduler.minimum-allocation-mb': 256, | + | | u'yarn.scheduler.maximum-allocation-mb': 2048, | + | | u'yarn.nodemanager.vmem-check-enabled': | + | | u'false', u'yarn.nodemanager.resource.memory- | + | | mb': 2048}}, u'auto_security_group': True, | + | | u'volumes_availability_zone': None, | + | | u'volume_mount_prefix': u'/volumes/disk', | + | | u'floating_ip_pool': u'public', u'image_id': | + | | None, u'volumes_size': 0, u'is_proxy_gateway': | + | | False, u'count': 1, u'name': u'master', | + | | u'created_at': u'2015-09-02T10:41:02', | + | | u'volume_type': None, u'node_processes': | + | | [u'namenode', u'resourcemanager', | + | | u'hiveserver'], u'flavor_id': u'2', | + | | u'use_autoconfig': True}, | + | | {u'volume_local_to_instance': False, | + | | u'availability_zone': None, u'updated_at': | + | | u'2015-09-02T10:41:07', u'instances': | + | | [{u'instance_id': u'47f97841-4a17-4e18-a8eb- | + | | b4ff7dd4c3d8', u'created_at': | + | | u'2015-09-02T10:41:06', u'updated_at': | + | | u'2015-09-02T10:41:13', u'instance_name': | + | | u'cluster-3-worker-001', u'management_ip': | + | | u'192.168.1.135', u'internal_ip': | + | | u'172.24.17.3', u'id': u'c4a02678-113b-432e- | + | | 8f91-927b8e7cfe83'}, {u'instance_id': | + | | u'a02aea39-cc1f-4a1f-8232-2470ab6e8478', | + | | u'created_at': u'2015-09-02T10:41:07', | + | | u'updated_at': u'2015-09-02T10:41:13', | + | | u'instance_name': u'cluster-3-worker-002', | + | | u'management_ip': u'192.168.1.130', | + | | u'internal_ip': u'172.24.17.4', u'id': u | + | | 'b7b2d6db-cd50-484b-8036-09820d2623f2'}], | + | | u'node_group_template_id': u'1aa4a397-cb1e- | + | | 4f38-be18-7f65fa0cc2eb', u'volumes_per_node': | + | | 0, u'id': u'b666103f-a44b-4cf8-b3ae- | + | | 7d2623c6cd18', u'security_groups': [7], | + | | u'shares': None, u'node_configs': | + | | {u'MapReduce': {u'mapreduce.map.memory.mb': | + | | 256, u'mapreduce.reduce.memory.mb': 512, | + | | u'yarn.app.mapreduce.am.command-opts': | + | | u'-Xmx204m', u'mapreduce.reduce.java.opts': | + | | u'-Xmx409m', | + | | u'yarn.app.mapreduce.am.resource.mb': 256, | + | | u'mapreduce.map.java.opts': u'-Xmx204m', | + | | u'mapreduce.task.io.sort.mb': 102}, u'YARN': | + | | {u'yarn.scheduler.minimum-allocation-mb': 256, | + | | u'yarn.scheduler.maximum-allocation-mb': 2048, | + | | u'yarn.nodemanager.vmem-check-enabled': | + | | u'false', u'yarn.nodemanager.resource.memory- | + | | mb': 2048}}, u'auto_security_group': True, | + | | u'volumes_availability_zone': None, | + | | u'volume_mount_prefix': u'/volumes/disk', | + | | u'floating_ip_pool': u'public', u'image_id': | + | | None, u'volumes_size': 0, u'is_proxy_gateway': | + | | False, u'count': 2, u'name': u'worker', | + | | u'created_at': u'2015-09-02T10:41:02', | + | | u'volume_type': None, u'node_processes': | + | | [u'nodemanager', u'datanode'], u'flavor_id': | + | | u'2', u'use_autoconfig': True}] | + | is_public | False | + | management_public_key | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDiFXlWNVD | + | | 6gJT74wherHWtgchqpvgi2aJ4fPWXP+WgB4GEKpfD7a/dWu | + | | Qg9eDBQIrWvVsKgG1i9YgRTHOQ7DdwoSKUAcpEewgw927ER | + | | wdJ3IV7EDu0xENUgrUgp+CwPdk94SXPg1G4oHOCbOvJYcW6 | + | | /b8Ci86vH9A7Uyu2T7tbVS4ciMKfwI0Z47lzcp2qDV6W8M7 | + | | neghC1mNT4k29ghgcYOzY4SxQjxp1a5Iu6RtnJ2fvHbLeMS | + | | 0hgeobSZ8heQzLImrp2dbyZy74goOcwKtk9dDPV853aZrjL | + | | yOsc78EgW6n2Gugu7Ks12v9QEDr4H3yTt3DNTrB5Y8tt468 | + | | k2n1 Generated-by-Sahara | | status_description | | - | hadoop_version | 1.2.1 | - | id | c5e755a2-b3f9-417b-948b-e99ed7fbf1e3 | + | hadoop_version | | + | id | 9b094131-a858-4ddb-81a8-b71597417cad | | trust_id | None | - | info | {} | - | cluster_template_id | c0609da7-faac-4dcf-9cbc-858a3aa130cd | - | name | cluster-1 | - | cluster_configs | {} | - | created_at | 2013-07-07T19:01:51 | - | default_image_id | 3f9fc974-b484-4756-82a4-bff9e116919b | - | tenant_id | 3fd7266fb3b547b1a45307b481bcadfd | + | info | {u'HDFS': {u'NameNode': | + | | u'hdfs://cluster-3-master-001:9000', u'Web UI': | + | | u'http://192.168.1.134:50070'}, u'YARN': {u'Web | + | | UI': u'http://192.168.1.134:8088', | + | | u'ResourceManager': | + | | u'http://192.168.1.134:8032'}} | + | cluster_template_id | 74add4df-07c2-4053-931f-d5844712727f | + | name | my-cluster-1 | + | cluster_configs | {u'HDFS': {u'dfs.replication': 2}} | + | created_at | 2015-09-02T10:41:02 | + | default_image_id | c119f99c-67f2-4404-9cff-f30e4b185036 | + | shares | None | + | is_protected | False | + | tenant_id | c82e4bce56ce4cf9b90bd15dfdef699d | +----------------------------+-------------------------------------------------+ Verify the cluster launched successfully by using the ``sahara`` command @@ -365,41 +573,41 @@ line tool as follows: .. sourcecode:: console - $ sahara cluster-list - +-----------+--------------------------------------+--------+------------+ - | name | id | status | node_count | - +-----------+--------------------------------------+--------+------------+ - | cluster-1 | c5e755a2-b3f9-417b-948b-e99ed7fbf1e3 | Active | 3 | - +-----------+--------------------------------------+--------+------------+ + $ sahara cluster-list + +--------------+--------------------------------------+--------+------------+ + | name | id | status | node_count | + +--------------+--------------------------------------+--------+------------+ + | my-cluster-1 | 9b094131-a858-4ddb-81a8-b71597417cad | Active | 3 | + +--------------+--------------------------------------+--------+------------+ The cluster creation operation may take several minutes to complete. During this time the "status" returned from the previous command may show states other than "Active". -8. Run a MapReduce job ----------------------- +8. Run a MapReduce job to check Hadoop installation +--------------------------------------------------- Check that your Hadoop installation is working properly by running an example job on the cluster manually. -* Login to the NameNode via ssh: +* Login to NameNode (usually master node) via ssh with ssh-key used above: .. sourcecode:: console - $ ssh ubuntu@ + $ ssh -i my_stack.pem ubuntu@ * Switch to the hadoop user: .. sourcecode:: console - $ sudo su hadoop + $ sudo su hadoop * Go to the shared hadoop directory and run the simplest MapReduce example: .. sourcecode:: console - $ cd /usr/share/hadoop - $ hadoop jar hadoop-examples-1.2.1.jar pi 10 100 + $ cd /opt/hadoop-/share/hadoop/mapreduce + $ /opt/hadoop-/bin/hadoop jar hadoop-mapreduce-examples-.jar pi 10 100 -Congratulations! Now you have the Hadoop cluster ready on the OpenStack -cloud. +Congratulations! Your Hadoop cluster is ready to use, running on your +OpenStack cloud.