openstack/sahara
Code Issues Proposed changes

Fix doc8 check failures

Browse Source 
Resolves D000 and D001 checks in sahara docs.

Change-Id: If6d577859170ae55d578d70885a9ed5a6b0db651
Closes-Bug: #1514887
This commit is contained in:
Alok Jani 2015-11-13 18:36:08 +05:30
parent f0d11daf6d
commit 7ba05c5e30
30 changed files with 577 additions and 449 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

46
doc/source/devref/adding_database_migrations.rst
View File

@ -1,16 +1,18 @@
Adding Database Migrations Adding Database Migrations
========================== ==========================
The migrations in ``sahara/db/migration/alembic_migrations/versions`` contain the changes needed to migrate The migrations in ``sahara/db/migration/alembic_migrations/versions`` contain
between Sahara database revisions. A migration occurs by executing a script that details the changes needed to the changes needed to migrate between Sahara database revisions. A migration
upgrade or downgrade the database. The migration scripts are ordered so that multiple scripts occurs by executing a script that details the changes needed to upgrade or
can run sequentially. The scripts are executed by Sahara's migration wrapper downgrade the database. The migration scripts are ordered so that multiple
which uses the Alembic library to manage the migration. Sahara supports scripts can run sequentially. The scripts are executed by Sahara's migration
wrapper which uses the Alembic library to manage the migration. Sahara supports
migration from Icehouse or later. migration from Icehouse or later.
Any code modifications that change the structure of the database require a migration script so that Any code modifications that change the structure of the database require a
previously existing databases will continue to function when the new code is released. This page gives a migration script so that previously existing databases will continue to
brief overview of how to add the migration. function when the new code is released. This page gives a brief overview of how
to add the migration.
Generate a New Migration Script Generate a New Migration Script
+++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++
@ -21,22 +23,26 @@ To generate a migration stub to be filled in by the developer::
$ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision" $ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision"
To autogenerate a migration script that reflects the current structure of the database:: To autogenerate a migration script that reflects the current structure of the
database::
$ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision" --autogenerate $ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision" --autogenerate
Each of these commands will create a file of the form ``revision_description`` where ``revision`` is a string Each of these commands will create a file of the form ``revision_description``
generated by Alembic and ``description`` is based on the text passed with the ``-m`` option. where ``revision`` is a string generated by Alembic and ``description`` is
based on the text passed with the ``-m`` option.
Follow the Sahara Naming Convention Follow the Sahara Naming Convention
+++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++
By convention Sahara uses 3-digit revision numbers, and this scheme differs from the strings generated by Alembic. By convention Sahara uses 3-digit revision numbers, and this scheme differs
Consequently, it's necessary to rename the generated script and modify the revision identifiers in the script. from the strings generated by Alembic. Consequently, it's necessary to rename
the generated script and modify the revision identifiers in the script.
Open the new script and look for the variable ``down_revision``. The value should be a 3-digit numeric string, and it Open the new script and look for the variable ``down_revision``. The value
identifies the current revision number of the database. Set the ``revision`` value to the ``down_revision`` value + 1. should be a 3-digit numeric string, and it identifies the current revision
For example, the lines:: number of the database. Set the ``revision`` value to the ``down_revision``
value + 1. For example, the lines::
# revision identifiers, used by Alembic. # revision identifiers, used by Alembic.
revision = '507eb70202af' revision = '507eb70202af'
@ -48,7 +54,8 @@ will become::
revision = '007' revision = '007'
down_revision = '006' down_revision = '006'
Modify any comments in the file to match the changes and rename the file to match the new revision number:: Modify any comments in the file to match the changes and rename the file to
match the new revision number::
$ mv 507eb70202af_my_new_revision.py 007_my_new_revision.py $ mv 507eb70202af_my_new_revision.py 007_my_new_revision.py
@ -56,7 +63,10 @@ $ mv 507eb70202af_my_new_revision.py 007_my_new_revision.py
Add Alembic Operations to the Script Add Alembic Operations to the Script
++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++++++++++++++++++
The migration script contains method ``upgrade()``. Since Kilo release Sahara doesn't support downgrades. Fill in this method with the appropriate Alembic operations to perform upgrades. In the above example, an upgrade will move from revision '006' to revision '007'. The migration script contains method ``upgrade()``. Since Kilo release Sahara
doesn't support downgrades. Fill in this method with the appropriate Alembic
operations to perform upgrades. In the above example, an upgrade will move from
revision '006' to revision '007'.
Command Summary for sahara-db-manage Command Summary for sahara-db-manage
++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++++++++++++++++++

3
doc/source/devref/development.environment.rst
View File

@ -119,7 +119,8 @@ Just add the following lines to .git/hooks/pre-commit and do chmod +x for it.
# Run fast checks (PEP8 style check and PyFlakes fast static analysis) # Run fast checks (PEP8 style check and PyFlakes fast static analysis)
tools/run_fast_checks tools/run_fast_checks
You can added the same check for pre-push, for example, run_tests and run_pylint. You can added the same check for pre-push, for example, run_tests and
run_pylint.
3. Running static analysis (PyLint) 3. Running static analysis (PyLint)

28
doc/source/devref/development.guidelines.rst
View File

@ -20,9 +20,10 @@ To check your code against PEP 8 and bashate run:
Static analysis Static analysis
--------------- ---------------
The static analysis checks are optional in Sahara. but they are still very useful. The static analysis checks are optional in Sahara. but they are still very
The gate job will inform you if the number of static analysis warnings useful. The gate job will inform you if the number of static analysis warnings
has increased after your change. We recommend to always check the static warnings. has increased after your change. We recommend to always check the static
warnings.
To run the check commit yor change first and execute the following command: To run the check commit yor change first and execute the following command:
@ -33,15 +34,16 @@ To run the check commit yor change first and execute the following command:
Modification of Upstream Files Modification of Upstream Files
------------------------------ ------------------------------
We never modify upstream files in Sahara. Any changes in upstream files should be made We never modify upstream files in Sahara. Any changes in upstream files should
in the upstream project and then merged back in to Sahara. This includes whitespace be made in the upstream project and then merged back in to Sahara. This
changes, comments, and typos. Any change requests containing upstream file modifications includes whitespace changes, comments, and typos. Any change requests
are almost certain to receive lots of negative reviews. Be warned. containing upstream file modifications are almost certain to receive lots of
negative reviews. Be warned.
Examples of upstream files are default xml configuration files used to configure Hadoop, or Examples of upstream files are default xml configuration files used to
code imported from the OpenStack Oslo project. The xml files will usually be found in configure Hadoop, or code imported from the OpenStack Oslo project. The xml
``resource`` directories with an accompanying ``README`` file that identifies where the files will usually be found in ``resource`` directories with an accompanying
files came from. For example: ``README`` file that identifies where the files came from. For example:
.. sourcecode:: console .. sourcecode:: console
@ -101,8 +103,8 @@ Run the following command to build docs locally.
$ tox -e docs $ tox -e docs
After it you can access generated docs in ``doc/build/`` directory, for example, After it you can access generated docs in ``doc/build/`` directory, for
main page - ``doc/build/html/index.html``. example, main page - ``doc/build/html/index.html``.
To make docs generation process faster you can use: To make docs generation process faster you can use:

12
doc/source/devref/devstack.rst
View File

@ -119,7 +119,8 @@ appended after the git repo URL as follows:
5. Once previous step is finished Devstack will print Horizon URL. Navigate to 5. Once previous step is finished Devstack will print Horizon URL. Navigate to
this URL and login with login "admin" and password from ``local.conf``. this URL and login with login "admin" and password from ``local.conf``.
6. Congratulations! You have OpenStack running in your VM and ready to launch VMs inside that VM :) 6. Congratulations! You have OpenStack running in your VM and ready to launch
VMs inside that VM :)
Managing sahara in DevStack Managing sahara in DevStack
@ -137,6 +138,7 @@ Setting fixed IP address for VMware Fusion VM
--------------------------------------------- ---------------------------------------------
1. Open file ``/Library/Preferences/VMware Fusion/vmnet8/dhcpd.conf`` 1. Open file ``/Library/Preferences/VMware Fusion/vmnet8/dhcpd.conf``
2. There is a block named "subnet". It might look like this: 2. There is a block named "subnet". It might look like this:
.. sourcecode:: text .. sourcecode:: text
@ -144,9 +146,11 @@ Setting fixed IP address for VMware Fusion VM
subnet 192.168.55.0 netmask 255.255.255.0 { subnet 192.168.55.0 netmask 255.255.255.0 {
range 192.168.55.128 192.168.55.254; range 192.168.55.128 192.168.55.254;
3. You need to pick an IP address outside of that range. For example - ``192.168.55.20`` 3. You need to pick an IP address outside of that range. For example -
4. Copy VM MAC address from VM settings->Network->Advanced ``192.168.55.20`` 4. Copy VM MAC address from VM settings->Network->Advanced
5. Append the following block to file ``dhcpd.conf`` (don't forget to replace ``VM_HOSTNAME`` and ``VM_MAC_ADDRESS`` with actual values):
5. Append the following block to file ``dhcpd.conf`` (don't forget to replace
``VM_HOSTNAME`` and ``VM_MAC_ADDRESS`` with actual values):
.. sourcecode:: text .. sourcecode:: text

39
doc/source/devref/edp.spi.rst
View File

@ -1,13 +1,13 @@
Elastic Data Processing (EDP) SPI Elastic Data Processing (EDP) SPI
================================= =================================
EDP job engine objects provide methods for creating, monitoring, and terminating EDP job engine objects provide methods for creating, monitoring, and
jobs on Sahara clusters. Provisioning plugins that support EDP must return an terminating jobs on Sahara clusters. Provisioning plugins that support EDP must
EDP job engine object from the :ref:`get_edp_engine` method described in return an EDP job engine object from the :ref:`get_edp_engine` method described
:doc:`plugin.spi`. in :doc:`plugin.spi`.
Sahara provides subclasses of the base job engine interface that support EDP on Sahara provides subclasses of the base job engine interface that support EDP
clusters running Oozie or on Spark standalone clusters. These are described on clusters running Oozie or on Spark standalone clusters. These are described
below. below.
.. _edp_spi_job_types: .. _edp_spi_job_types:
@ -60,8 +60,8 @@ cancel_job(job_execution)
Stops the running job whose id is stored in the job_execution object. Stops the running job whose id is stored in the job_execution object.
*Returns*: None if the operation was unsuccessful or an updated job status value *Returns*: None if the operation was unsuccessful or an updated job status
value
get_job_status(job_execution) get_job_status(job_execution)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -133,10 +133,10 @@ JobEngine. It provides implementations for all of the methods in the base
interface but adds a few more abstract methods. interface but adds a few more abstract methods.
Note, the *validate_job_execution(cluster, job, data)* method does basic checks Note, the *validate_job_execution(cluster, job, data)* method does basic checks
on the job configuration but probably should be overloaded to include additional on the job configuration but probably should be overloaded to include
checks on the cluster configuration. For example, the job engines for plugins additional checks on the cluster configuration. For example, the job engines
that support Oozie add checks to make sure that the Oozie service is up and for plugins that support Oozie add checks to make sure that the Oozie service
running. is up and running.
get_hdfs_user() get_hdfs_user()
@ -160,8 +160,8 @@ creates the HDFS directory *dir_name* under the user specified by
if the HDFS user is 'hadoop' and the dir_name is 'test' this method would if the HDFS user is 'hadoop' and the dir_name is 'test' this method would
create '/user/hadoop/test'. create '/user/hadoop/test'.
The reason that this method is broken out in the interface as an abstract method The reason that this method is broken out in the interface as an abstract
is that different versions of Hadoop treat path creation differently. method is that different versions of Hadoop treat path creation differently.
*Returns*: None *Returns*: None
@ -207,14 +207,15 @@ The sahara.service.edp.spark.engine.SparkJobEngine class provides a full EDP
implementation for Spark standalone clusters. implementation for Spark standalone clusters.
Note, the *validate_job_execution(cluster, job, data)* method does basic checks Note, the *validate_job_execution(cluster, job, data)* method does basic checks
on the job configuration but probably should be overloaded to include additional on the job configuration but probably should be overloaded to include
checks on the cluster configuration. For example, the job engine returned by the additional checks on the cluster configuration. For example, the job engine
Spark plugin checks that the Spark version is >= 1.0.0 to ensure that returned by the Spark plugin checks that the Spark version is >= 1.0.0 to
*spark-submit* is available. ensure that *spark-submit* is available.
get_driver_classpath(self) get_driver_classpath(self)
~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~
Returns driver class path. Returns driver class path.
*Returns*: a string of the following format ' --driver-class-path *class_path_value*' *Returns*: a string of the following format ' --driver-class-path
*class_path_value*'

14
doc/source/devref/how_to_build_oozie.rst
View File

@ -40,10 +40,14 @@ JDK Versions
By default, the build configuration enforces that JDK 1.6.* is being used. By default, the build configuration enforces that JDK 1.6.* is being used.
There are 2 build properties that can be used to change the JDK version requirements: There are 2 build properties that can be used to change the JDK version
requirements:
* ``javaVersion`` specifies the version of the JDK used to compile (default 1.6) * ``javaVersion`` specifies the version of the JDK used to compile (default
* ``targetJavaVersion`` specifies the version of the generated bytecode (default 1.6) 1.6)
* ``targetJavaVersion`` specifies the version of the generated bytecode
(default 1.6)
For example, to specify 1.7 JDK version, build command should contain For example, to specify 1.7 JDK version, build command should contain
``-D javaVersion=1.7 -D tagetJavaVersion=1.7`` flags. ``-D javaVersion=1.7 -D tagetJavaVersion=1.7`` flags.
@ -63,4 +67,6 @@ used:
Also, pig version can be passed as maven property with ``-D pig.version=x.x.x`` Also, pig version can be passed as maven property with ``-D pig.version=x.x.x``
flag. flag.
Similar instruction to build oozie.tar.gz you may find there: http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Building_Oozie Similar instruction to build oozie.tar.gz you may find there:
http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Building_Oozie

3
doc/source/devref/how_to_participate.rst
View File

@ -43,7 +43,8 @@ How to stay in touch with the community?
* Weekly on Thursdays at 1400 UTC and 1800 UTC (on alternate weeks) * Weekly on Thursdays at 1400 UTC and 1800 UTC (on alternate weeks)
* IRC channel: ``#openstack-meeting-alt`` (1800UTC) and ``#openstack-meeting-3`` (1400UTC) * IRC channel: ``#openstack-meeting-alt`` (1800UTC) and
``#openstack-meeting-3`` (1400UTC)
How to send your first patch on review? How to send your first patch on review?

44
doc/source/devref/jenkins.rst
View File

@ -1,23 +1,41 @@
Continuous Integration with Jenkins Continuous Integration with Jenkins
=================================== ===================================
Each change made to Sahara core code is tested with unit and integration tests and style checks flake8. Each change made to Sahara core code is tested with unit and integration tests
and style checks flake8.
Unit tests and style checks are performed on public `OpenStack Jenkins
<https://jenkins.openstack.org/>`_ managed by `Zuul
<http://status.openstack.org/zuul/>`_.
Unit tests and style checks are performed on public `OpenStack Jenkins <https://jenkins.openstack.org/>`_ managed by `Zuul <http://status.openstack.org/zuul/>`_.
Unit tests are checked using python 2.7. Unit tests are checked using python 2.7.
The result of those checks and Unit tests are +1 or -1 to *Verify* column in a code review from *Jenkins* user. The result of those checks and Unit tests are +1 or -1 to *Verify* column in a
code review from *Jenkins* user.
Integration tests check CRUD operations for Image Registry, Templates and Clusters. Integration tests check CRUD operations for Image Registry, Templates and
Also a test job is launched on a created Cluster to verify Hadoop work. Clusters. Also a test job is launched on a created Cluster to verify Hadoop
work.
All integration tests are launched by `Jenkins <https://sahara.mirantis.com/jenkins/>`_ on the internal Mirantis OpenStack Lab. All integration tests are launched by `Jenkins
Jenkins keeps a pool of VMs to run tests in parallel. Even with the pool of VMs integration testing may take a while. <https://sahara.mirantis.com/jenkins/>`_ on the internal Mirantis OpenStack
Jenkins is controlled for the most part by Zuul which determines what jobs are run when. Lab.
Zuul status is available by address: `Zuul Status <https://sahara.mirantis.com/zuul>`_.
For more information see: `Sahara Hadoop Cluster CI <https://wiki.openstack.org/wiki/Sahara/SaharaCI>`_.
The integration tests result is +1 or -1 to *Verify* column in a code review from *Sahara Hadoop Cluster CI* user. Jenkins keeps a pool of VMs to run tests in parallel. Even with the pool of VMs
integration testing may take a while.
You can put *sahara-ci-recheck* in comment, if you want to recheck sahara-ci jobs. Jenkins is controlled for the most part by Zuul which determines what jobs are
Also, you can put *recheck* in comment, if you want to recheck both jenkins and sahara-ci jobs. run when.
Zuul status is available by address: `Zuul Status
<https://sahara.mirantis.com/zuul>`_.
For more information see: `Sahara Hadoop Cluster CI
<https://wiki.openstack.org/wiki/Sahara/SaharaCI>`_.
The integration tests result is +1 or -1 to *Verify* column in a code review
from *Sahara Hadoop Cluster CI* user.
You can put *sahara-ci-recheck* in comment, if you want to recheck sahara-ci
jobs. Also, you can put *recheck* in comment, if you want to recheck both
jenkins and sahara-ci jobs.

10
doc/source/devref/launchpad.rst
View File

@ -1,8 +1,8 @@
Project hosting Project hosting
=============== ===============
`Launchpad`_ hosts the Sahara project. The Sahara project homepage on Launchpad is `Launchpad`_ hosts the Sahara project. The Sahara project homepage on Launchpad
http://launchpad.net/sahara. is http://launchpad.net/sahara.
Launchpad credentials Launchpad credentials
--------------------- ---------------------
@ -18,9 +18,9 @@ OpenStack-related sites. These sites include:
Mailing list Mailing list
------------ ------------
The mailing list email is ``openstack-dev@lists.openstack.org`` with subject prefix The mailing list email is ``openstack-dev@lists.openstack.org`` with subject
``[sahara]``. To participate in the mailing list subscribe to the list at prefix ``[sahara]``. To participate in the mailing list subscribe to the list
http://lists.openstack.org/cgi-bin/mailman/listinfo at http://lists.openstack.org/cgi-bin/mailman/listinfo
Bug tracking Bug tracking
------------ ------------

8
doc/source/devref/log.guidelines.rst
View File

@ -5,8 +5,8 @@ Log Guidelines
Levels Guidelines Levels Guidelines
----------------- -----------------
During the Kilo release cycle the sahara community defined the following log levels: During the Kilo release cycle the sahara community defined the following
log levels:
* Debug: Shows everything and is likely not suitable for normal production * Debug: Shows everything and is likely not suitable for normal production
operation due to the sheer size of logs generated (e.g. scripts executions, operation due to the sheer size of logs generated (e.g. scripts executions,
@ -41,8 +41,8 @@ Now sahara uses string formatting defined in `PEP 3101`_ for logs.
Translation Guidelines Translation Guidelines
---------------------- ----------------------
All log levels except Debug requires translation. None of the separate cli tools packaged All log levels except Debug requires translation. None of the separate
with sahara contain log translations. cli tools packaged with sahara contain log translations.
* Debug: no translation * Debug: no translation
* Info: _LI * Info: _LI

322
doc/source/devref/plugin.spi.rst
View File

@ -7,9 +7,10 @@ Plugin interface
get_versions() get_versions()
~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
Returns all versions of Hadoop that could be used with the plugin. Returns all versions of Hadoop that could be used with the plugin. It is
It is responsibility of the plugin to make sure that all required images for each hadoop version are available, responsibility of the plugin to make sure that all required images for each
as well as configs and whatever else that plugin needs to create the Hadoop cluster. hadoop version are available, as well as configs and whatever else that plugin
needs to create the Hadoop cluster.
*Returns*: list of strings - Hadoop versions *Returns*: list of strings - Hadoop versions
@ -18,27 +19,31 @@ as well as configs and whatever else that plugin needs to create the Hadoop clus
get_configs( hadoop_version) get_configs( hadoop_version)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lists all configs supported by plugin with descriptions, defaults and targets for which this config is applicable. Lists all configs supported by plugin with descriptions, defaults and targets
for which this config is applicable.
*Returns*: list of configs *Returns*: list of configs
*Example return value*: ((“JobTracker heap size”, "JobTracker heap size, in MB", "int", “512”, `“mapreduce”`, "node", True, 1)) *Example return value*: ((“JobTracker heap size”, "JobTracker heap size, in
MB", "int", “512”, `“mapreduce”`, "node", True, 1))
get_node_processes( hadoop_version) get_node_processes( hadoop_version)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Returns all supported services and node processes for a given Hadoop version. Returns all supported services and node processes for a given Hadoop version.
Each node process belongs to a single service and that relationship is reflected in the returned dict object. Each node process belongs to a single service and that relationship is
See example for details. reflected in the returned dict object. See example for details.
*Returns*: dictionary having entries (service -> list of processes) *Returns*: dictionary having entries (service -> list of processes)
*Example return value*: {"mapreduce": ["tasktracker", "jobtracker"], "hdfs": ["datanode", "namenode"]} *Example return value*: {"mapreduce": ["tasktracker", "jobtracker"], "hdfs":
["datanode", "namenode"]}
get_required_image_tags( hadoop_version) get_required_image_tags( hadoop_version)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lists tags, that should be added to OpenStack Image via Image Registry. Tags are used to filter Images by plugin and hadoop version. Lists tags, that should be added to OpenStack Image via Image Registry. Tags
are used to filter Images by plugin and hadoop version.
*Returns*: list of tags *Returns*: list of tags
@ -47,11 +52,14 @@ Lists tags, that should be added to OpenStack Image via Image Registry. Tags are
validate(cluster) validate(cluster)
~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~
Validates a given cluster object. Raises *SaharaException* with meaningful message. Validates a given cluster object. Raises *SaharaException* with meaningful
message.
*Returns*: None *Returns*: None
*Example exception*: <NotSingleNameNodeException {code='NOT_SINGLE_NAME_NODE', message='Hadoop cluster should contain only 1 NameNode instance. Actual NN count is 2' }> *Example exception*: <NotSingleNameNodeException {code='NOT_SINGLE_NAME_NODE',
message='Hadoop cluster should contain only 1 NameNode instance. Actual NN
count is 2' }>
validate_scaling(cluster, existing, additional) validate_scaling(cluster, existing, additional)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -75,24 +83,26 @@ For instance, plugin can ask for additional VMs for the management tool.
configure_cluster(cluster) configure_cluster(cluster)
~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~
Configures cluster on provisioned by sahara VMs. Configures cluster on provisioned by sahara VMs. In this function plugin
In this function plugin should perform all actions like adjusting OS, installing required packages (including Hadoop, if needed), configuring Hadoop, etc. should perform all actions like adjusting OS, installing required packages
(including Hadoop, if needed), configuring Hadoop, etc.
*Returns*: None *Returns*: None
start_cluster(cluster) start_cluster(cluster)
~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~
Start already configured cluster. This method is guaranteed to be called only on cluster which was already prepared with configure_cluster(...) call. Start already configured cluster. This method is guaranteed to be called only
on cluster which was already prepared with configure_cluster(...) call.
*Returns*: None *Returns*: None
scale_cluster(cluster, instances) scale_cluster(cluster, instances)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scale an existing cluster with additional instances. Instances argument is a list Scale an existing cluster with additional instances. Instances argument is a
of ready-to-configure instances. Plugin should do all configuration operations in this list of ready-to-configure instances. Plugin should do all configuration
method and start all services on those instances. operations in this method and start all services on those instances.
*Returns*: None *Returns*: None
@ -103,17 +113,19 @@ get_edp_engine(cluster, job_type)
Returns an EDP job engine object that supports the specified job_type on the Returns an EDP job engine object that supports the specified job_type on the
given cluster, or None if there is no support. The EDP job engine object given cluster, or None if there is no support. The EDP job engine object
returned must implement the interface described in :doc:`edp.spi`. The job_type returned must implement the interface described in :doc:`edp.spi`. The
is a String matching one of the job types listed in :ref:`edp_spi_job_types`. job_type is a String matching one of the job types listed in
:ref:`edp_spi_job_types`.
*Returns*: an EDP job engine object or None *Returns*: an EDP job engine object or None
decommission_nodes(cluster, instances) decommission_nodes(cluster, instances)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scale cluster down by removing a list of instances. Plugin should stop services on a provided list Scale cluster down by removing a list of instances. Plugin should stop services
of instances. Plugin also may want to update some configurations on other instances, so this on a provided list of instances. Plugin also may want to update some
method is the right place to do that. configurations on other instances, so this method is the right place to do
that.
*Returns*: None *Returns*: None
@ -129,7 +141,9 @@ See “Cluster Lifecycle for Config File Mode” section below for clarification
on_terminate_cluster(cluster) on_terminate_cluster(cluster)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When user terminates cluster, sahara simply shuts down all the cluster VMs. This method is guaranteed to be invoked before that, allowing plugin to do some clean-up. When user terminates cluster, sahara simply shuts down all the cluster VMs.
This method is guaranteed to be invoked before that, allowing plugin to do some
clean-up.
*Returns*: None *Returns*: None
@ -140,19 +154,22 @@ When user requests sahara to automatically create security group for the node
group (``auto_security_group`` property set to True), sahara will call this group (``auto_security_group`` property set to True), sahara will call this
plugin method to get list of ports that need to be opened. plugin method to get list of ports that need to be opened.
*Returns*: list of ports to be open in auto security group for the given node group *Returns*: list of ports to be open in auto security group for the given node
group
def get_edp_job_types(versions) def get_edp_job_types(versions)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Optional method, which provides ability to see all supported job types for specified plugin versions Optional method, which provides ability to see all supported job types for
specified plugin versions
*Returns*: dict with supported job types for specified versions of plugin *Returns*: dict with supported job types for specified versions of plugin
def recommend_configs(self, cluster, scaling=False) def recommend_configs(self, cluster, scaling=False)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Optional method, which provides recommendations for cluster configuration before creating/scaling operation. Optional method, which provides recommendations for cluster configuration
before creating/scaling operation.
*Returns*: None *Returns*: None
@ -163,8 +180,9 @@ Here is a description of all the objects involved in the API.
Notes: Notes:
- cluster and node_group have ‘extra’ field allowing plugin to persist any complementary info about the cluster. - cluster and node_group have ‘extra’ field allowing plugin to persist any
- node_process is just a process that runs at some node in cluster. complementary info about the cluster.
- node_process is just a process that runs at some node in cluster.
Example list of node processes: Example list of node processes:
@ -180,30 +198,36 @@ Config
An object, describing one configuration entry An object, describing one configuration entry
+-------------------+--------+--------------------------------------------------------------------------------+ +-------------------+--------+------------------------------------------------+
| Property | Type | Description | | Property | Type | Description |
+===================+========+================================================================================+ +===================+========+================================================+
| name | string | Config name. | | name | string | Config name. |
+-------------------+--------+--------------------------------------------------------------------------------+ +-------------------+--------+------------------------------------------------+
| description | string | A hint for user, what this config is used for. | | description | string | A hint for user, what this config is used for. |
+-------------------+--------+--------------------------------------------------------------------------------+ +-------------------+--------+------------------------------------------------+
| config_type | enum | possible values are: 'string', 'integer', 'boolean', 'enum'. | | config_type | enum | possible values are: 'string', 'integer', |
+-------------------+--------+--------------------------------------------------------------------------------+ | | | 'boolean', 'enum'. |
| config_values | list | List of possible values, if config_type is enum. | +-------------------+--------+------------------------------------------------+
+-------------------+--------+--------------------------------------------------------------------------------+ | config_values | list | List of possible values, if config_type is |
| default_value | string | Default value for config. | | | | enum. |
+-------------------+--------+--------------------------------------------------------------------------------+ +-------------------+--------+------------------------------------------------+
| applicable_target | string | The target could be either a service returned by get_node_processes(...) call | | default_value | string | Default value for config. |
| | | in form of 'service:<service name>', or 'general'. | +-------------------+--------+------------------------------------------------+
+-------------------+--------+--------------------------------------------------------------------------------+ | applicable_target | string | The target could be either a service returned |
| scope | enum | Could be either 'node' or 'cluster'. | | | | by get_node_processes(...) call |
+-------------------+--------+--------------------------------------------------------------------------------+ | | | in form of 'service:<service name>', or |
| is_optional | bool | If is_optional is False and no default_value is specified, user should | | | | 'general'. |
| | | provide a value. | +-------------------+--------+------------------------------------------------+
+-------------------+--------+--------------------------------------------------------------------------------+ | scope | enum | Could be either 'node' or 'cluster'. |
| priority | int | 1 or 2. A Hint for UI. Configs with priority *1* are always displayed. | +-------------------+--------+------------------------------------------------+
| | | Priority *2* means user should click a button to see the config. | | is_optional | bool | If is_optional is False and no default_value |
+-------------------+--------+--------------------------------------------------------------------------------+ | | | is specified, user should provide a value |
+-------------------+--------+------------------------------------------------+
| priority | int | 1 or 2. A Hint for UI. Configs with priority |
| | | *1* are always displayed. |
| | | Priority *2* means user should click a button |
| | | to see the config. |
+-------------------+--------+------------------------------------------------+
User Input User Input
@ -225,31 +249,33 @@ Instance
An instance created for cluster. An instance created for cluster.
+---------------+---------+---------------------------------------------------------+ +---------------+---------+---------------------------------------------------+
| Property | Type | Description | | Property | Type | Description |
+===============+=========+=========================================================+ +===============+=========+===================================================+
| instance_id | string | Unique instance identifier. | | instance_id | string | Unique instance identifier. |
+---------------+---------+---------------------------------------------------------+ +---------------+---------+---------------------------------------------------+
| instance_name | string | OpenStack Instance name. | | instance_name | string | OpenStack Instance name. |
+---------------+---------+---------------------------------------------------------+ +---------------+---------+---------------------------------------------------+
| internal_ip | string | IP to communicate with other instances. | | internal_ip | string | IP to communicate with other instances. |
+---------------+---------+---------------------------------------------------------+ +---------------+---------+---------------------------------------------------+
| management_ip | string | IP of instance, accessible outside of internal network. | | management_ip | string | IP of instance, accessible outside of internal |
+---------------+---------+---------------------------------------------------------+ | | | network. |
| volumes | list | List of volumes attached to instance. Empty if | +---------------+---------+---------------------------------------------------+
| | | ephemeral drive is used. | | volumes | list | List of volumes attached to instance. Empty if |
+---------------+---------+---------------------------------------------------------+ | | | ephemeral drive is used. |
| nova_info | object | Nova Instance object. | +---------------+---------+---------------------------------------------------+
+---------------+---------+---------------------------------------------------------+ | nova_info | object | Nova Instance object. |
| username | string | Username, that sahara uses for establishing remote | +---------------+---------+---------------------------------------------------+
| | | connections to instance. | | username | string | Username, that sahara uses for establishing |
+---------------+---------+---------------------------------------------------------+ | | | remote connections to instance. |
| hostname | string | Same as instance_name. | +---------------+---------+---------------------------------------------------+
+---------------+---------+---------------------------------------------------------+ | hostname | string | Same as instance_name. |
| fqdn | string | Fully qualified domain name for this instance. | +---------------+---------+---------------------------------------------------+
+---------------+---------+---------------------------------------------------------+ | fqdn | string | Fully qualified domain name for this instance. |
| remote | helpers | Object with helpers for performing remote operations | +---------------+---------+---------------------------------------------------+
+---------------+---------+---------------------------------------------------------+ | remote | helpers | Object with helpers for performing remote |
| | | operations |
+---------------+---------+---------------------------------------------------+
Node Group Node Group
@ -257,78 +283,86 @@ Node Group
Group of instances. Group of instances.
+----------------------+--------+--------------------------------------------------------+ +----------------------+--------+---------------------------------------------+
| Property | Type | Description | | Property | Type | Description |
+======================+========+========================================================+ +======================+========+=============================================+
| name | string | Name of this Node Group in Cluster. | | name | string | Name of this Node Group in Cluster. |
+----------------------+--------+--------------------------------------------------------+ +----------------------+--------+---------------------------------------------+
| flavor_id | string | OpenStack Flavor used to boot instances. | | flavor_id | string | OpenStack Flavor used to boot instances. |
+----------------------+--------+--------------------------------------------------------+ +----------------------+--------+---------------------------------------------+
| image_id | string | Image id used to boot instances. | | image_id | string | Image id used to boot instances. |
+----------------------+--------+--------------------------------------------------------+ +----------------------+--------+---------------------------------------------+
| node_processes | list | List of processes running on each instance. | | node_processes | list | List of processes running on each instance. |
+----------------------+--------+--------------------------------------------------------+ +----------------------+--------+---------------------------------------------+
| node_configs | dict | Configs dictionary, applied to instances. | | node_configs | dict | Configs dictionary, applied to instances. |
+----------------------+--------+--------------------------------------------------------+ +----------------------+--------+---------------------------------------------+
| volumes_per_node | int | Number of volumes mounted to each instance. 0 means | | volumes_per_node | int | Number of volumes mounted to each instance. |
| | | use ephemeral drive. | | | | 0 means use ephemeral drive. |
+----------------------+--------+--------------------------------------------------------+ +----------------------+--------+---------------------------------------------+
| volumes_size | int | Size of each volume (GB). | | volumes_size | int | Size of each volume (GB). |
+----------------------+--------+--------------------------------------------------------+ +----------------------+--------+---------------------------------------------+
| volumes_mount_prefix | string | Prefix added to mount path of each volume. | | volumes_mount_prefix | string | Prefix added to mount path of each volume. |
+----------------------+--------+--------------------------------------------------------+ +----------------------+--------+---------------------------------------------+
| floating_ip_pool | string | Floating IP Pool name. All instances in the Node Group | | floating_ip_pool | string | Floating IP Pool name. All instances in the |
| | | will have Floating IPs assigned from this pool. | | | | Node Group will have Floating IPs assigned |
+----------------------+--------+--------------------------------------------------------+ | | | from this pool. |
| count | int | Number of instances in this Node Group. | +----------------------+--------+---------------------------------------------+
+----------------------+--------+--------------------------------------------------------+ | count | int | Number of instances in this Node Group. |
| username | string | Username used by sahara to establish remote | +----------------------+--------+---------------------------------------------+
| | | connections to instances. | | username | string | Username used by sahara to establish remote |
+----------------------+--------+--------------------------------------------------------+ | | | connections to instances. |
| configuration | dict | Merged dictionary of node configurations and cluster | +----------------------+--------+---------------------------------------------+
| | | configurations. | | configuration | dict | Merged dictionary of node configurations |
+----------------------+--------+--------------------------------------------------------+ | | | and cluster configurations. |
| storage_paths | list | List of directories where storage should be placed. | +----------------------+--------+---------------------------------------------+
+----------------------+--------+--------------------------------------------------------+ | storage_paths | list | List of directories where storage should be |
| | | placed. |
+----------------------+--------+---------------------------------------------+
Cluster Cluster
------- -------
Contains all relevant info about cluster. Contains all relevant info about cluster. This object is is provided to the
This object is is provided to the plugin for both cluster creation and scaling. plugin for both cluster creation and scaling. The “Cluster Lifecycle” section
The “Cluster Lifecycle” section below further specifies which fields are filled at which moment. below further specifies which fields are filled at which moment.
+----------------------------+--------+-----------------------------------------------------------+ +----------------------------+--------+---------------------------------------+
| Property | Type | Description | | Property | Type | Description |
+============================+========+===========================================================+ +============================+========+=======================================+
| name | string | Cluster name. | | name | string | Cluster name. |
+----------------------------+--------+-----------------------------------------------------------+ +----------------------------+--------+---------------------------------------+
| tenant_id | string | OpenStack Tenant id where this Cluster is available. | | tenant_id | string | OpenStack Tenant id where this |
+----------------------------+--------+-----------------------------------------------------------+ | | | Cluster is available. |
| plugin_name | string | Plugin name. | +----------------------------+--------+---------------------------------------+
+----------------------------+--------+-----------------------------------------------------------+ | plugin_name | string | Plugin name. |
| hadoop_version | string | Hadoop version running on instances. | +----------------------------+--------+---------------------------------------+
+----------------------------+--------+-----------------------------------------------------------+ | hadoop_version | string | Hadoop version running on instances. |
| default_image_id | string | OpenStack image used to boot instances. | +----------------------------+--------+---------------------------------------+
+----------------------------+--------+-----------------------------------------------------------+ | default_image_id | string | OpenStack image used to boot |
| node_groups | list | List of Node Groups. | | | | instances. |
+----------------------------+--------+-----------------------------------------------------------+ +----------------------------+--------+---------------------------------------+
| cluster_configs | dict | Dictionary of Cluster scoped configurations. | | node_groups | list | List of Node Groups. |
+----------------------------+--------+-----------------------------------------------------------+ +----------------------------+--------+---------------------------------------+
| cluster_template_id | string | Cluster Template used for Node Groups and Configurations. | | cluster_configs | dict | Dictionary of Cluster scoped |
+----------------------------+--------+-----------------------------------------------------------+ | | | configurations. |
| user_keypair_id | string | OpenStack keypair added to instances to make them | +----------------------------+--------+---------------------------------------+
| | | accessible for user. | | cluster_template_id | string | Cluster Template used for Node Groups |
+----------------------------+--------+-----------------------------------------------------------+ | | | and Configurations. |
| neutron_management_network | string | Neutron network ID. Instances will get fixed IPs in this | +----------------------------+--------+---------------------------------------+
| | | network if 'use_neutron' config is set to True. | | user_keypair_id | string | OpenStack keypair added to instances |
+----------------------------+--------+-----------------------------------------------------------+ | | | to make them accessible for user. |
| anti_affinity | list | List of processes that will be run on different hosts. | +----------------------------+--------+---------------------------------------+
+----------------------------+--------+-----------------------------------------------------------+ | neutron_management_network | string | Neutron network ID. Instances will |
| description | string | Cluster Description. | | | | get fixed IPs in this network if |
+----------------------------+--------+-----------------------------------------------------------+ | | | 'use_neutron' config is set to True. |
| info | dict | Dictionary for additional information. | +----------------------------+--------+---------------------------------------+
+----------------------------+--------+-----------------------------------------------------------+ | anti_affinity | list | List of processes that will be run on |
| | | different hosts. |
+----------------------------+--------+---------------------------------------+
| description | string | Cluster Description. |
+----------------------------+--------+---------------------------------------+
| info | dict | Dictionary for additional information.|
+----------------------------+--------+---------------------------------------+
Validation Error Validation Error

39
doc/source/horizon/dashboard.user.guide.rst
View File

@ -2,8 +2,8 @@ Sahara (Data Processing) UI User Guide
====================================== ======================================
This guide assumes that you already have the Sahara service and Horizon This guide assumes that you already have the Sahara service and Horizon
dashboard up and running. Don't forget to make sure that Sahara is registered in dashboard up and running. Don't forget to make sure that Sahara is
Keystone. If you require assistance with that, please see the registered in Keystone. If you require assistance with that, please see the
`installation guide <../installation.guide.html>`_. `installation guide <../installation.guide.html>`_.
The sections below give a panel by panel overview of setting up clusters The sections below give a panel by panel overview of setting up clusters
@ -25,8 +25,8 @@ Registering an Image
4) Enter the username of the cloud-init user on the image 4) Enter the username of the cloud-init user on the image
5) Click on the tags that you want to add to the image. (A version ie: 1.2.1 and 5) Click on the tags that you want to add to the image. (A version ie: 1.2.1
a type ie: vanilla are required for cluster functionality) and a type ie: vanilla are required for cluster functionality)
6) Click the "Done" button to finish the registration 6) Click the "Done" button to finish the registration
@ -46,8 +46,8 @@ Create Node Group Templates
5) Choose a flavor for this template (based on your CPU/memory/disk needs) 5) Choose a flavor for this template (based on your CPU/memory/disk needs)
6) Choose the storage location for your instance, this can be either "Ephemeral 6) Choose the storage location for your instance, this can be either "Ephemeral
Drive" or "Cinder Volume". If you choose "Cinder Volume", you will need to add Drive" or "Cinder Volume". If you choose "Cinder Volume", you will need to
additional configuration add additional configuration
7) Choose which processes should be run for any instances that are spawned from 7) Choose which processes should be run for any instances that are spawned from
this Node Group Template this Node Group Template
@ -131,8 +131,8 @@ Data Sources
------------ ------------
Data Sources are where the input and output from your jobs are housed. Data Sources are where the input and output from your jobs are housed.
1) From the Data Processing/Data Sources page, click on the "Create Data Source" 1) From the Data Processing/Data Sources page, click on the "Create Data
button at the top right Source" button at the top right
2) Give your Data Source a name 2) Give your Data Source a name
@ -223,8 +223,8 @@ status of your job to see when it has completed its run
- Additional configuration properties can be defined by clicking on the "Add" - Additional configuration properties can be defined by clicking on the "Add"
button button
- An example configuration entry might be mapred.mapper.class for the Name and - An example configuration entry might be mapred.mapper.class for the Name
org.apache.oozie.example.SampleMapper for the Value and org.apache.oozie.example.SampleMapper for the Value
5) Click on "Launch". To monitor the status of your job, you can navigate to 5) Click on "Launch". To monitor the status of your job, you can navigate to
the Data Processing/Jobs panel the Data Processing/Jobs panel
@ -241,8 +241,9 @@ status of your job to see when it has completed its run
Example Jobs Example Jobs
------------ ------------
There are sample jobs located in the Sahara repository. In this section, we There are sample jobs located in the Sahara repository. In this section, we
will give a walkthrough on how to run those jobs via the Horizon UI. These steps will give a walkthrough on how to run those jobs via the Horizon UI. These
assume that you already have a cluster up and running (in the "Active" state). steps assume that you already have a cluster up and running (in the "Active"
state).
1) Sample Pig job - 1) Sample Pig job -
https://github.com/openstack/sahara/tree/master/etc/edp-examples/edp-pig/trim-spaces https://github.com/openstack/sahara/tree/master/etc/edp-examples/edp-pig/trim-spaces
@ -287,8 +288,8 @@ assume that you already have a cluster up and running (in the "Active" state).
- Name = pigsample, Job Type = Pig, Choose "example.pig" as the main binary - Name = pigsample, Job Type = Pig, Choose "example.pig" as the main binary
- Click on the "Libs" tab and choose "udf.jar", then hit the "Choose" button - Click on the "Libs" tab and choose "udf.jar", then hit the "Choose"
beneath the dropdown, then click on "Create" button beneath the dropdown, then click on "Create"
- Launch your job - Launch your job
@ -364,11 +365,11 @@ Additional Notes
existing Job. In order to be able to delete that job, you would existing Job. In order to be able to delete that job, you would
first need to delete any Job Templates that relate to that job. first need to delete any Job Templates that relate to that job.
2) In the examples above, we mention adding your username/password for the Swift 2) In the examples above, we mention adding your username/password for the
Data Sources. It should be noted that it is possible to configure Sahara such Swift Data Sources. It should be noted that it is possible to configure
that the username/password credentials are *not* required. For more Sahara such that the username/password credentials are *not* required. For
information on that, please refer to: more information on that, please refer to: :doc:`Sahara Advanced
:doc:`Sahara Advanced Configuration Guide <../userdoc/advanced.configuration.guide>` Configuration Guide <../userdoc/advanced.configuration.guide>`
Launching a cluster via the Cluster Creation Guide Launching a cluster via the Cluster Creation Guide
-------------------------------------------------- --------------------------------------------------

37
doc/source/horizon/dev.environment.guide.rst
View File

@ -2,8 +2,8 @@ Sahara UI Dev Environment Setup
=============================== ===============================
This page describes how to setup Horizon for developing Sahara by either This page describes how to setup Horizon for developing Sahara by either
installing it as part of DevStack with Sahara or installing it in an isolated environment installing it as part of DevStack with Sahara or installing it in an
and running from the command line. isolated environment and running from the command line.
Install as a part of DevStack Install as a part of DevStack
----------------------------- -----------------------------
@ -11,8 +11,8 @@ Install as a part of DevStack
See the `DevStack guide <../devref/devstack.html>`_ for more information See the `DevStack guide <../devref/devstack.html>`_ for more information
on installing and configuring DevStack with Sahara. on installing and configuring DevStack with Sahara.
After Horizon installation, it will contain a Data Processing tab under Projects tab. After Horizon installation, it will contain a Data Processing tab under
Sahara UI source code will be located at Projects tab. Sahara UI source code will be located at
``$DEST/horizon/openstack_dashboard/contrib/sahara/content/data_processing`` ``$DEST/horizon/openstack_dashboard/contrib/sahara/content/data_processing``
where ``$DEST/`` is usually ``/opt/stack/``. where ``$DEST/`` is usually ``/opt/stack/``.
@ -42,7 +42,8 @@ You can list the registered services with this command:
.. sourcecode:: console .. sourcecode:: console
$ sudo apt-get update $ sudo apt-get update
$ sudo apt-get install git-core python-dev gcc python-setuptools python-virtualenv node-less libssl-dev libffi-dev libxslt-dev $ sudo apt-get install git-core python-dev gcc python-setuptools \
python-virtualenv node-less libssl-dev libffi-dev libxslt-dev
.. ..
On Ubuntu 12.10 and higher you have to install the following lib as well: On Ubuntu 12.10 and higher you have to install the following lib as well:
@ -72,7 +73,8 @@ You can list the registered services with this command:
.. sourcecode:: console .. sourcecode:: console
$ cp openstack_dashboard/local/local_settings.py.example openstack_dashboard/local/local_settings.py $ cp openstack_dashboard/local/local_settings.py.example
openstack_dashboard/local/local_settings.py
.. ..
4. Modify ``openstack_dashboard/local/local_settings.py`` 4. Modify ``openstack_dashboard/local/local_settings.py``
@ -84,16 +86,18 @@ You can list the registered services with this command:
OPENSTACK_HOST = "ip of your controller" OPENSTACK_HOST = "ip of your controller"
.. ..
If you are using Nova-Network with ``auto_assign_floating_ip=True`` add the following parameter: If you are using Nova-Network with ``auto_assign_floating_ip=True`` add the
following parameter:
.. sourcecode:: python .. sourcecode:: python
SAHARA_AUTO_IP_ALLOCATION_ENABLED = True SAHARA_AUTO_IP_ALLOCATION_ENABLED = True
.. ..
5. If Sahara is not registered with the keystone service catalog, it may be added 5. If Sahara is not registered with the keystone service catalog, it may be
with the following commands. To use Sahara from Horizon without keystone added with the following commands. To use Sahara from Horizon without keystone
registration, see `Using the Data Processing Dashboard without Keystone Registration`_. registration, see `Using the Data Processing Dashboard without Keystone
Registration`_.
.. sourcecode:: console .. sourcecode:: console
@ -109,8 +113,9 @@ You can list the registered services with this command:
$ tools/with_venv.sh python manage.py runserver 0.0.0.0:8080 $ tools/with_venv.sh python manage.py runserver 0.0.0.0:8080
.. ..
This will start Horizon in debug mode. That means the logs will be written to console This will start Horizon in debug mode. That means the logs will be written to
and if any exceptions happen, you will see the stack-trace rendered as a web-page. console and if any exceptions happen, you will see the stack-trace rendered
as a web-page.
Debug mode can be disabled by changing ``DEBUG=True`` to ``False`` in Debug mode can be disabled by changing ``DEBUG=True`` to ``False`` in
``local_settings.py``. In that case Horizon should be started slightly ``local_settings.py``. In that case Horizon should be started slightly
@ -126,10 +131,10 @@ You can list the registered services with this command:
7. Applying changes 7. Applying changes
If you have changed any ``*.py`` files in If you have changed any ``*.py`` files in
``horizon/openstack_dashboard/contrib/sahara/content/data_processing`` directory, ``horizon/openstack_dashboard/contrib/sahara/content/data_processing``
Horizon will notice that and reload automatically. However changes made to directory, Horizon will notice that and reload automatically. However
non-python files may not be noticed, so you have to restart Horizon again changes made to non-python files may not be noticed, so you have to restart
manually, as described in step 6. Horizon again manually, as described in step 6.
Using the Data Processing Dashboard without Keystone Registration Using the Data Processing Dashboard without Keystone Registration
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

6
doc/source/restapi.rst
View File

@ -81,6 +81,8 @@ error description.
2 API 2 API
===== =====
- `Sahara REST API Reference (OpenStack API Complete Reference - DataProcessing)`_ - `Sahara REST API Reference (OpenStack API Complete Reference -
DataProcessing)`_
.. _`Sahara REST API Reference (OpenStack API Complete Reference - DataProcessing)`: http://api.openstack.org/api-ref-data-processing-v1.1.html
.. _`Sahara REST API Reference (OpenStack API Complete Reference - DataProcessing)`: http://api.openstack.org/api-ref-data-processing-v1.1.html

25
doc/source/userdoc/advanced.configuration.guide.rst
View File

@ -446,18 +446,19 @@ NTP package is included in the image (the sahara disk image builder will
include NTP in all images it generates). The default NTP server will be include NTP in all images it generates). The default NTP server will be
``pool.ntp.org``; this can be overridden using the ``default_ntp_server`` ``pool.ntp.org``; this can be overridden using the ``default_ntp_server``
setting in the ``DEFAULT`` section of the sahara configuration file. setting in the ``DEFAULT`` section of the sahara configuration file.
If you are creating cluster templates using the sahara UI and would like to specify
a different NTP server for a particular cluster template, use the
``URL of NTP server`` setting in the ``General Parameters``
section when you create the template. If you would like to disable NTP for a
particular cluster template, deselect the ``Enable NTP service`` checkbox in
the ``General Parameters`` section when you create the template.
If you are creating clusters using the sahara CLI, you can specify another NTP server If you are creating cluster templates using the sahara UI and would like to
or disable NTP service using the examples below. specify a different NTP server for a particular cluster template, use the ``URL
of NTP server`` setting in the ``General Parameters`` section when you create
the template. If you would like to disable NTP for a particular cluster
template, deselect the ``Enable NTP service`` checkbox in the ``General
Parameters`` section when you create the template.
If you want to disable configuring the NTP service, you should specify the following If you are creating clusters using the sahara CLI, you can specify another NTP
configs for the cluster: server or disable NTP service using the examples below.
If you want to disable configuring the NTP service, you should specify the
following configs for the cluster:
.. sourcecode:: json .. sourcecode:: json
@ -467,8 +468,8 @@ configs for the cluster:
} }
} }
If you want to disable configuring NTP service, you should specify following configs If you want to disable configuring NTP service, you should specify following
for the cluster: configs for the cluster:
.. sourcecode:: json .. sourcecode:: json

30
doc/source/userdoc/cdh_imagebuilder.rst
View File

@ -30,18 +30,18 @@ To create cloudera images follow these steps:
2. Use tox to build images. 2. Use tox to build images.
You can run "tox -e venv -- sahara-image-create" command in sahara-image-elements You can run "tox -e venv -- sahara-image-create" command in
directory to build images. By default this script will attempt to create cloud sahara-image-elements directory to build images. By default this script will
images for all versions of supported plugins and all operating systems attempt to create cloud images for all versions of supported plugins and all
(subset of Ubuntu, Fedora, and CentOS depending on plugin). To only create operating systems (subset of Ubuntu, Fedora, and CentOS depending on
Cloudera images, you should use the "-p cloudera" parameter in the command plugin). To only create Cloudera images, you should use the "-p cloudera"
line. If you want to create the image only for a specific operating system, parameter in the command line. If you want to create the image only for a
you should use the "-i ubuntu|centos" parameter to assign the operating specific operating system, you should use the "-i ubuntu|centos" parameter
system (the cloudera plugin only supports Ubuntu and Centos). If you want to assign the operating system (the cloudera plugin only supports Ubuntu and
to create the image only for a specific Cloudera version, you should use the Centos). If you want to create the image only for a specific Cloudera
"-v 5.0|5.3|5.4" parameter to assign the version. Below is an example to create version, you should use the "-v 5.0|5.3|5.4" parameter to assign the
Cloudera images for both Ubuntu and CentOS with Cloudera Express 5.4.0 version. Below is an example to create Cloudera images for both Ubuntu and
version. CentOS with Cloudera Express 5.4.0 version.
.. sourcecode:: console .. sourcecode:: console
@ -57,9 +57,9 @@ To create cloudera images follow these steps:
NOTE: If you don't want to use default values, you should explicitly set the NOTE: If you don't want to use default values, you should explicitly set the
values of your required parameters. values of your required parameters.
The script will create required cloud images using image elements that install The script will create required cloud images using image elements that
all the necessary packages and configure them. You will find the created install all the necessary packages and configure them. You will find the
images in the parent directory. created images in the parent directory.
.. note:: .. note::

18
doc/source/userdoc/cdh_plugin.rst
View File

@ -13,9 +13,9 @@ to provision cluster or you could download prepared images from
http://sahara-files.mirantis.com/images/upstream/kilo/ http://sahara-files.mirantis.com/images/upstream/kilo/
They already have Cloudera Express installed (5.0.0, 5.3.0, or 5.4.0 version). They already have Cloudera Express installed (5.0.0, 5.3.0, or 5.4.0 version).
The cloudera plugin requires an image to be tagged in Sahara Image Registry with The cloudera plugin requires an image to be tagged in Sahara Image Registry
two tags: 'cdh' and '<cloudera version>' (e.g. '5', '5.3.0' or '5.4.0', here '5' with two tags: 'cdh' and '<cloudera version>' (e.g. '5', '5.3.0' or '5.4.0',
stands for '5.0.0'). here '5' stands for '5.0.0').
The default username specified for these images is different for each The default username specified for these images is different for each
distribution: distribution:
@ -32,8 +32,8 @@ Services Supported
------------------ ------------------
Currently below services are supported in both versions of Cloudera plugin: Currently below services are supported in both versions of Cloudera plugin:
HDFS, Oozie, YARN, Spark, Zookeeper, Hive, Hue, HBase. 5.3.0 version HDFS, Oozie, YARN, Spark, Zookeeper, Hive, Hue, HBase. 5.3.0 version of
of Cloudera Plugin also supported following services: Impala, Flume, Solr, Sqoop, Cloudera Plugin also supported following services: Impala, Flume, Solr, Sqoop,
and Key-value Store Indexer. 5.4.0 version added KMS service support based on and Key-value Store Indexer. 5.4.0 version added KMS service support based on
5.3.0 version. 5.3.0 version.
@ -47,8 +47,8 @@ and Key-value Store Indexer. 5.4.0 version added KMS service support based on
High Availablity Support High Availablity Support
------------------------ ------------------------
Currently HDFS NameNode High Availability is supported in Cloudera 5.4.0 version. Currently HDFS NameNode High Availability is supported in Cloudera 5.4.0
You can refer to :doc:`features` for the detail info. version. You can refer to :doc:`features` for the detail info.
YARN ResourceManager High Availability is supported in Cloudera 5.4.0 version. YARN ResourceManager High Availability is supported in Cloudera 5.4.0 version.
This feature adds redundancy in the form of an Active/Standby ResourceManager This feature adds redundancy in the form of an Active/Standby ResourceManager
@ -87,8 +87,8 @@ cloudera plugin versions:
and at least one hbase regionserver. and at least one hbase regionserver.
+ Cluster can't contain hbase regionserver without at least one hbase maser. + Cluster can't contain hbase regionserver without at least one hbase maser.
In case of 5.3.0 or 5.4.0 version of Cloudera Plugin there are few extra limitations In case of 5.3.0 or 5.4.0 version of Cloudera Plugin there are few extra
in the cluster topology: limitations in the cluster topology:
+ Cluster can't contain flume without at least one datanode. + Cluster can't contain flume without at least one datanode.
+ Cluster can contain at most one sentry server service. + Cluster can contain at most one sentry server service.

36
doc/source/userdoc/configs_recommendations.rst
View File

@ -1,31 +1,35 @@
Autoconfiguring templates Autoconfiguring templates
========================= =========================
During the Liberty development cycle sahara implemented a tool that recommends and During the Liberty development cycle sahara implemented a tool that recommends
applies configuration values for cluster templates and node group templates. and applies configuration values for cluster templates and node group
These recommendations are based on the number of specific instances and on templates. These recommendations are based on the number of specific instances
flavors of the cluster node groups. Currently the following plugins support and on flavors of the cluster node groups. Currently the following plugins
this feature: support this feature:
* Cloudera; * Cloudera;
* Spark; * Spark;
* the Vanilla Apache Hadoop plugin. * the Vanilla Apache Hadoop plugin.
By default this feature is enabled for all cluster templates and node group templates. By default this feature is enabled for all cluster templates and node group
If you want to disable this feature for a particular cluster or node group template templates. If you want to disable this feature for a particular cluster or
you should set the ``use_autoconfig`` field to ``false``. node group template you should set the ``use_autoconfig`` field to ``false``.
.. NOTE .. NOTE
Also, if you manually set configs from the list below, the recommended configs Also, if you manually set configs from the list below, the recommended
will not be applied. configs will not be applied.
The following describes the settings for which sahara can recommend autoconfiguration: The following describes the settings for which sahara can recommend
autoconfiguration:
The Cloudera, Spark and Vanilla Apache Hadoop plugin support configuring The Cloudera, Spark and Vanilla Apache Hadoop plugin support configuring
``dfs.replication`` (``dfs_replication`` for Cloudera plugin) which is calculated as ``dfs.replication`` (``dfs_replication`` for Cloudera plugin) which is
a minimun from the amount of ``datanode`` (``HDFS_DATANODE`` for Cloudera plugin) instances in calculated as a minimun from the amount of ``datanode`` (``HDFS_DATANODE`` for
the cluster and the default value for ``dfs.replication``. Cloudera plugin) instances in the cluster and the default value for
``dfs.replication``.
The Vanilla Apache Hadoop plugin and Cloudera plugin support autoconfiguration The Vanilla Apache Hadoop plugin and Cloudera plugin support autoconfiguration
of basic YARN and MapReduce configs. These autoconfigurations are based on the following of basic YARN and MapReduce configs. These autoconfigurations are based on the
documentation: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html following documentation:
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html

4
doc/source/userdoc/dashboard.guide.rst
View File

@ -12,8 +12,8 @@ The sample configuration file is available `here. <https://github.com/openstack/
1. Networking 1. Networking
------------- -------------
Depending on the Networking backend (Nova Network or Neutron) used in the cloud, Depending on the Networking backend (Nova Network or Neutron) used in the
Sahara panels will determine automatically which input fields should be cloud, Sahara panels will determine automatically which input fields should be
displayed. displayed.
While using Nova Network backend the cloud may be configured to automatically While using Nova Network backend the cloud may be configured to automatically

34
doc/source/userdoc/edp.rst
View File

@ -281,17 +281,21 @@ in the future), sahara allows the addition of an interface (or method
signature) to your job template. A sample interface for the Teragen Hadoop signature) to your job template. A sample interface for the Teragen Hadoop
example might be: example might be:
+---------------+--------------+------------------+-------------+----------+---------------------+ +---------+---------+------------+-------------+----------+-------------------+
| Name | Mapping Type | Location | Value Type | Required | Default | | Name | Mapping | Location | Value | Required | Default |
+===============+==============+==================+=============+==========+=====================+ | | Type | | Type | | |
| Example Class | args | 0 | string | false | teragen | +=========+=========+============+=============+=======+======================+
+---------------+--------------+------------------+-------------+----------+---------------------+ | Example | args | 0 | string | false | teragen |
| Rows | args | 1 | number | true | ``unset`` | | Class | | | | | |
+---------------+--------------+------------------+-------------+----------+---------------------+ +---------+---------+------------+-------------+-------+----------------------+
| Output Path | args | 2 | data_source | false | hdfs://ip:port/path | | Rows | args | 1 | number | true | ``unset`` |
+---------------+--------------+------------------+-------------+----------+---------------------+ +---------+---------+------------+-------------+-------+----------------------+
| Mapper Count | configs | mapred.map.tasks | number | false | ``unset`` | | Output | args | 2 | data_source | false | hdfs://ip:port/path |
+---------------+--------------+------------------+-------------+----------+---------------------+ | Path | | | | | |
+---------+---------+------------+-------------+-------+----------------------+
| Mapper | configs | mapred. | number | false | ``unset`` |
| Count | | map.tasks | | | |
+---------+---------+------------+-------------+-------+----------------------+
A "Description" field may also be added to each interface argument. A "Description" field may also be added to each interface argument.
@ -496,10 +500,10 @@ running jobs on a cluster generated by the Cloudera plugin with the
generated by sahara in HDFS be added to the **oozie.libpath**. This for use generated by sahara in HDFS be added to the **oozie.libpath**. This for use
when an Hbase application is driven from a Java job. Default is **False**. when an Hbase application is driven from a Java job. Default is **False**.
The **edp-wordcount** example bundled with sahara shows how to use configuration The **edp-wordcount** example bundled with sahara shows how to use
values, arguments, and swift data paths in a Java job type. Note that the configuration values, arguments, and swift data paths in a Java job type. Note
example does not use the **edp.java.adapt_for_oozie** option but includes the that the example does not use the **edp.java.adapt_for_oozie** option but
code to load the configuration values explicitly. includes the code to load the configuration values explicitly.
Additional Details for Shell jobs Additional Details for Shell jobs
+++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++

9
doc/source/userdoc/features.rst
View File

@ -190,10 +190,11 @@ Security group management
------------------------- -------------------------
Security groups are sets of IP filter rules that are applied to an instance's Security groups are sets of IP filter rules that are applied to an instance's
networking. They are project specified, and project members can edit the default networking. They are project specified, and project members can edit the
rules for their group and add new rules sets. All projects have a "default" default rules for their group and add new rules sets. All projects have a
security group, which is applied to instances that have no other security "default" security group, which is applied to instances that have no other
group defined. Unless changed, this security group denies all incoming traffic. security group defined. Unless changed, this security group denies all incoming
traffic.
Sahara allows you to control which security groups will be used for created Sahara allows you to control which security groups will be used for created
instances. This can be done by providing the ``security_groups`` parameter for instances. This can be done by providing the ``security_groups`` parameter for

27
doc/source/userdoc/guest-requirements.rst
View File

@ -1,7 +1,10 @@
Requirements for Guests Requirements for Guests
======================= =======================
Sahara manages guests of various platforms (for example Ubuntu, Fedora, RHEL, and CentOS) with various versions of the Hadoop ecosystem projects installed. There are common requirements for all guests, and additional requirements based on the plugin that is used for cluster deployment. Sahara manages guests of various platforms (for example Ubuntu, Fedora, RHEL,
and CentOS) with various versions of the Hadoop ecosystem projects installed.
There are common requirements for all guests, and additional requirements based
on the plugin that is used for cluster deployment.
Common Requirements Common Requirements
------------------- -------------------
@ -15,14 +18,16 @@ Common Requirements
Vanilla Plugin Requirements Vanilla Plugin Requirements
--------------------------- ---------------------------
If the Vanilla Plugin is used for cluster deployment the guest is required to have If the Vanilla Plugin is used for cluster deployment the guest is required to
have
* ssh-client installed * ssh-client installed
* Java (version >= 6) * Java (version >= 6)
* Apache Hadoop installed * Apache Hadoop installed
* 'hadoop' user created * 'hadoop' user created
See :doc:`hadoop-swift` for information on using Swift with your sahara cluster (for EDP support Swift integration is currently required). See :doc:`hadoop-swift` for information on using Swift with your sahara cluster
(for EDP support Swift integration is currently required).
To support EDP, the following components must also be installed on the guest: To support EDP, the following components must also be installed on the guest:
@ -30,17 +35,23 @@ To support EDP, the following components must also be installed on the guest:
* mysql * mysql
* hive * hive
See :doc:`vanilla_imagebuilder` for instructions on building images for this plugin. See :doc:`vanilla_imagebuilder` for instructions on building images for this
plugin.
Hortonworks Plugin Requirements Hortonworks Plugin Requirements
------------------------------- -------------------------------
This plugin does not have any additional requirements. Currently, only the CentOS Linux distribution is supported but other distributions will be supported in the future. This plugin does not have any additional requirements. Currently, only the
To speed up provisioning, the HDP packages can be pre-installed on the image used. The packages' versions depend on the HDP version being used. CentOS Linux distribution is supported but other distributions will be
supported in the future.
To speed up provisioning, the HDP packages can be pre-installed on the image
used. The packages' versions depend on the HDP version being used.
Cloudera Plugin Requirements Cloudera Plugin Requirements
---------------------------- ----------------------------
Cloudera Plugin does not have any additional requirements, just build a CDH image to deploy the cluster. Cloudera Plugin does not have any additional requirements, just build a CDH
image to deploy the cluster.
See :doc:`cdh_imagebuilder` for instructions on building images for this plugin. See :doc:`cdh_imagebuilder` for instructions on building images for this
plugin.

20
doc/source/userdoc/hadoop-swift.rst
View File

@ -47,13 +47,18 @@ instances using this template:
There are two types of configs here: There are two types of configs here:
1. General. The ``${name}`` in this case equals to ``fs.swift``. Here is the list of ``${config}``: 1. General. The ``${name}`` in this case equals to ``fs.swift``. Here is the
list of ``${config}``:
* ``.impl`` - Swift FileSystem implementation. The ${value} is ``org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem`` * ``.impl`` - Swift FileSystem implementation. The ${value} is
``org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem``
* ``.connect.timeout`` - timeout for all connections by default: 15000 * ``.connect.timeout`` - timeout for all connections by default: 15000
* ``.socket.timeout`` - how long the connection waits for responses from servers. by default: 60000 * ``.socket.timeout`` - how long the connection waits for responses from
* ``.connect.retry.count`` - connection retry count for all connections. by default: 3 servers. by default: 60000
* ``.connect.throttle.delay`` - delay in millis between bulk (delete, rename, copy operations). by default: 0 * ``.connect.retry.count`` - connection retry count for all connections. by
default: 3
* ``.connect.throttle.delay`` - delay in millis between bulk (delete,
rename, copy operations). by default: 0
* ``.blocksize`` - blocksize for filesystem. By default: 32Mb * ``.blocksize`` - blocksize for filesystem. By default: 32Mb
* ``.partsize`` - the partition size for uploads. By default: 4608*1024Kb * ``.partsize`` - the partition size for uploads. By default: 4608*1024Kb
* ``.requestsize`` - request size for reads in KB. By default: 64Kb * ``.requestsize`` - request size for reads in KB. By default: 64Kb
@ -76,8 +81,9 @@ There are two types of configs here:
* ``.http.port`` * ``.http.port``
* ``.https.port`` * ``.https.port``
* ``.region`` - Swift region is used when cloud has more than one Swift * ``.region`` - Swift region is used when cloud has more than one Swift
installation. If region param is not set first region from Keystone endpoint installation. If region param is not set first region from Keystone
list will be chosen. If region param not found exception will be thrown. endpoint list will be chosen. If region param not found exception will be
thrown.
* ``.location-aware`` - turn On location awareness. Is false by default * ``.location-aware`` - turn On location awareness. Is false by default
* ``.apikey`` * ``.apikey``
* ``.public`` * ``.public``

24
doc/source/userdoc/hdp_plugin.rst
View File

@ -18,11 +18,11 @@ The HDP plugin can make use of Ambari Blueprints for cluster provisioning.
Apache Ambari Blueprints Apache Ambari Blueprints
------------------------ ------------------------
Apache Ambari Blueprints is a portable document definition, which provides Apache Ambari Blueprints is a portable document definition, which provides a
a complete definition for an Apache Hadoop cluster, including cluster topology, complete definition for an Apache Hadoop cluster, including cluster topology,
components, services and their configurations. Ambari Blueprints can be components, services and their configurations. Ambari Blueprints can be
consumed by the HDP plugin to instantiate a Hadoop cluster on OpenStack. consumed by the HDP plugin to instantiate a Hadoop cluster on OpenStack. The
The benefits of this approach is that it allows for Hadoop clusters to be benefits of this approach is that it allows for Hadoop clusters to be
configured and deployed using an Ambari native format that can be used with as configured and deployed using an Ambari native format that can be used with as
well as outside of OpenStack allowing for clusters to be re-instantiated in a well as outside of OpenStack allowing for clusters to be re-instantiated in a
variety of environments. variety of environments.
@ -49,8 +49,8 @@ Images
------ ------
The sahara HDP plugin can make use of either minimal (operating system only) The sahara HDP plugin can make use of either minimal (operating system only)
images or pre-populated HDP images. The base requirement for both is that the images or pre-populated HDP images. The base requirement for both is that the
image is cloud-init enabled and contains a supported operating system image is cloud-init enabled and contains a supported operating system (see
(see http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.4/bk_hdp1-system-admin-guide/content/sysadminguides_ha_chap2_3.html). http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.4/bk_hdp1-system-admin-guide/content/sysadminguides_ha_chap2_3.html).
The advantage of a pre-populated image is that provisioning time is reduced, The advantage of a pre-populated image is that provisioning time is reduced,
as packages do not need to be downloaded and installed which make up the as packages do not need to be downloaded and installed which make up the
@ -72,10 +72,10 @@ The username specified should be 'cloud-user'.
HDFS NameNode High Availability HDFS NameNode High Availability
------------------------------- -------------------------------
HDFS NameNode High Availability (Using the Quorum Journal Manager) can be deployed HDFS NameNode High Availability (Using the Quorum Journal Manager) can be
automatically with HDP 2.0.6. Currently the only way to deploy it is through the deployed automatically with HDP 2.0.6. Currently the only way to deploy it is
command line client (python-saharaclient) or sahara REST API by simply adding the through the command line client (python-saharaclient) or sahara REST API by
following cluster_configs parameter in the cluster's JSON : simply adding the following cluster_configs parameter in the cluster's JSON :
.. sourcecode:: cfg .. sourcecode:: cfg
@ -89,8 +89,8 @@ The NameNode High Availability is deployed using 2 NameNodes, one active and
one standby. The NameNodes use a set of JOURNALNODES and ZOOKEEPER_SERVERS to one standby. The NameNodes use a set of JOURNALNODES and ZOOKEEPER_SERVERS to
ensure the necessary synchronization. ensure the necessary synchronization.
A typical Highly available HDP 2.0.6 cluster uses 2 separate NameNodes, at least 3 A typical Highly available HDP 2.0.6 cluster uses 2 separate NameNodes, at
JOURNALNODES and at least 3 ZOOKEEPER_SERVERS. least 3 JOURNALNODES and at least 3 ZOOKEEPER_SERVERS.
When HDFS NameNode High Availability is enabled, the plugin will perform the When HDFS NameNode High Availability is enabled, the plugin will perform the
following additional validations: following additional validations:

32
doc/source/userdoc/mapr_plugin.rst
View File

@ -5,22 +5,27 @@ OpenStack in an easy way and do it, quickly, conveniently and simply.
Operation Operation
--------- ---------
The MapR Plugin performs the following four primary functions during cluster creation: The MapR Plugin performs the following four primary functions during cluster
creation:
1. MapR components deployment - the plugin manages the deployment of the required software to the target VMs 1. MapR components deployment - the plugin manages the deployment of the
2. Services Installation - MapR services are installed according to provided roles list required software to the target VMs
3. Services Configuration - the plugin combines default settings with user provided settings 2. Services Installation - MapR services are installed according to provided
4. Services Start - the plugin starts appropriate services according to specified roles roles list
3. Services Configuration - the plugin combines default settings with user
provided settings
4. Services Start - the plugin starts appropriate services according to
specified roles
Images Images
------ ------
The Sahara MapR plugin can make use of either minimal (operating system only) The Sahara MapR plugin can make use of either minimal (operating system only)
images or pre-populated MapR images. The base requirement for both is that the images or pre-populated MapR images. The base requirement for both is that the
image is cloud-init enabled and contains a supported operating system image is cloud-init enabled and contains a supported operating system (see
(see http://doc.mapr.com/display/MapR/OS+Support+Matrix). http://doc.mapr.com/display/MapR/OS+Support+Matrix).
The advantage of a pre-populated image is that provisioning time is reduced, The advantage of a pre-populated image is that provisioning time is reduced, as
as packages do not need to be downloaded which make up the majority of the time packages do not need to be downloaded which make up the majority of the time
spent in the provisioning cycle. In addition, provisioning large clusters will spent in the provisioning cycle. In addition, provisioning large clusters will
put a burden on the network as packages for all nodes need to be downloaded put a burden on the network as packages for all nodes need to be downloaded
from the package repository. from the package repository.
@ -45,7 +50,8 @@ two tags: 'mapr' and '<MapR version>' (e.g. '4.0.1.mrv2').
Note that Spark should be run on plain or 4.0.1 images. Note that Spark should be run on plain or 4.0.1 images.
The default username specified for these images is different for each distribution: The default username specified for these images is different for each
distribution:
+--------------+------------+ +--------------+------------+
| OS | username | | OS | username |
@ -58,7 +64,8 @@ The default username specified for these images is different for each distributi
Hadoop Version Support Hadoop Version Support
---------------------- ----------------------
The MapR plugin currently supports Hadoop 0.20.2 (3.1.1, 4.0.1.mrv1, 4.0.2.mrv1), The MapR plugin currently supports Hadoop 0.20.2 (3.1.1, 4.0.1.mrv1,
4.0.2.mrv1),
Hadoop 2.4.1 (4.0.2.mrv2) and Hadoop 2.5.1 (4.0.2.mrv2). Hadoop 2.4.1 (4.0.2.mrv2) and Hadoop 2.5.1 (4.0.2.mrv2).
Cluster Validation Cluster Validation
@ -110,7 +117,8 @@ Hue service is considered valid if:
HttpFS service is considered valid if cluster has exactly 1 *HttpFS* process HttpFS service is considered valid if cluster has exactly 1 *HttpFS* process
Sqoop service is considered valid if cluster has exactly 1 *Sqoop2-Server* process Sqoop service is considered valid if cluster has exactly 1 *Sqoop2-Server*
process
The MapR Plugin The MapR Plugin
--------------- ---------------

9
doc/source/userdoc/registering_image.rst
View File

@ -2,9 +2,10 @@ Registering an Image
==================== ====================
Sahara deploys a cluster of machines using images stored in Glance. Sahara deploys a cluster of machines using images stored in Glance.
Each plugin has its own requirements on the image contents (see specific plugin Each plugin has its own requirements on the image contents (see specific plugin
documentation for details). Two general requirements for an image are to have the documentation for details). Two general requirements for an image are to have
cloud-init and the ssh-server packages installed. the cloud-init and the ssh-server packages installed.
Sahara requires the images to be registered in the Sahara Image Registry. Sahara requires the images to be registered in the Sahara Image Registry.
A registered image must have two properties set: A registered image must have two properties set:
@ -12,5 +13,5 @@ A registered image must have two properties set:
* username - a name of the default cloud-init user. * username - a name of the default cloud-init user.
* tags - certain tags mark image to be suitable for certain plugins. * tags - certain tags mark image to be suitable for certain plugins.
The username depends on the image that is used and the tags depend on the plugin used. The username depends on the image that is used and the tags depend on the
You can find both in the respective plugin's documentation. plugin used. You can find both in the respective plugin's documentation.

80
doc/source/userdoc/statuses.rst
View File

@ -3,8 +3,8 @@ Sahara Cluster Statuses Overview
All Sahara Cluster operations are performed in multiple steps. A Cluster object All Sahara Cluster operations are performed in multiple steps. A Cluster object
has a ``Status`` attribute which changes when Sahara finishes one step of has a ``Status`` attribute which changes when Sahara finishes one step of
operations and starts another one. Also a Cluster object has a ``Status description`` operations and starts another one. Also a Cluster object has a ``Status
attribute which changes whenever Cluster errors occur. description`` attribute which changes whenever Cluster errors occur.
Sahara supports three types of Cluster operations: Sahara supports three types of Cluster operations:
* Create a new Cluster * Create a new Cluster
@ -21,19 +21,22 @@ Before performing any operations with OpenStack environment, Sahara validates
user input. user input.
There are two types of validations, that are done: There are two types of validations, that are done:
* Check that a request contains all necessary fields and that the request does not violate * Check that a request contains all necessary fields and that the request does
any constraints like unique naming, etc. not violate any constraints like unique naming, etc.
* Plugin check (optional). The provisioning Plugin may also perform any specific checks like a Cluster topology validation check. * Plugin check (optional). The provisioning Plugin may also perform any
specific checks like a Cluster topology validation check.
If any of the validations fails during creating, the Cluster object will If any of the validations fails during creating, the Cluster object will still
still be kept in the database with an ``Error`` status. If any validations fails during scaling the ``Active`` Cluster, it will be be kept in the database with an ``Error`` status. If any validations fails
kept with an ``Active`` status. during scaling the ``Active`` Cluster, it will be kept with an ``Active``
In both cases status description will contain error messages about the reasons of failure. status. In both cases status description will contain error messages about the
reasons of failure.
2. InfraUpdating 2. InfraUpdating
~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~
This status means that the Provisioning plugin is performing some infrastructure updates. This status means that the Provisioning plugin is performing some
infrastructure updates.
3. Spawning 3. Spawning
~~~~~~~~~~~ ~~~~~~~~~~~
@ -49,22 +52,23 @@ so sahara will wait until all of the VMs are in an ``Active`` state.
4. Waiting 4. Waiting
~~~~~~~~~~ ~~~~~~~~~~
Sahara waits while VMs' operating systems boot up and all internal infrastructure Sahara waits while VMs' operating systems boot up and all internal
components like networks and volumes are attached and ready to use. infrastructure components like networks and volumes are attached and ready to
use.
5. Preparing 5. Preparing
~~~~~~~~~~~~ ~~~~~~~~~~~~
Sahara prepares a Cluster for starting. This step includes generating the ``/etc/hosts`` Sahara prepares a Cluster for starting. This step includes generating the
file, so that all instances can access each other by a hostname. Also Sahara ``/etc/hosts`` file, so that all instances can access each other by a hostname.
updates the ``authorized_keys`` file on each VM, so that VMs can communicate without Also Sahara updates the ``authorized_keys`` file on each VM, so that VMs can
passwords. communicate without passwords.
6. Configuring 6. Configuring
~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
Sahara pushes service configurations to VMs. Both XML and JSON based configurations and Sahara pushes service configurations to VMs. Both XML and JSON based
environmental variables are set on this step. configurations and environmental variables are set on this step.
7. Starting 7. Starting
~~~~~~~~~~~ ~~~~~~~~~~~
@ -74,7 +78,8 @@ Sahara is starting Hadoop services on Cluster's VMs.
8. Active 8. Active
~~~~~~~~~ ~~~~~~~~~
Active status means that a Cluster has started successfully and is ready to run EDP Jobs. Active status means that a Cluster has started successfully and is ready to run
EDP Jobs.
Scaling/Shrinking an existing Cluster Scaling/Shrinking an existing Cluster
@ -84,7 +89,8 @@ Scaling/Shrinking an existing Cluster
~~~~~~~~~~~~~ ~~~~~~~~~~~~~
Sahara checks the scale/shrink request for validity. The Plugin method called Sahara checks the scale/shrink request for validity. The Plugin method called
for performing Plugin specific checks is different from the validation method in creation. for performing Plugin specific checks is different from the validation method
in creation.
2. Scaling 2. Scaling
~~~~~~~~~~ ~~~~~~~~~~
@ -95,22 +101,23 @@ and creating new ones to join the existing Node Groups.
3. Adding Instances 3. Adding Instances
~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
Status is similar to ``Spawning`` in Custer creation. Sahara adds required amount Status is similar to ``Spawning`` in Custer creation. Sahara adds required
of VMs to the existing Node Groups and creates new Node Groups. amount of VMs to the existing Node Groups and creates new Node Groups.
4. Configuring 4. Configuring
~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
Status is similar to ``Configuring`` in Cluster creation. New instances are being configured Status is similar to ``Configuring`` in Cluster creation. New instances are
in the same manner as already existing ones. The VMs in the existing Cluster are also updated being configured in the same manner as already existing ones. The VMs in the
with a new ``/etc/hosts`` file. existing Cluster are also updated with a new ``/etc/hosts`` file.
5. Decommissioning 5. Decommissioning
~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~
Sahara stops Hadoop services on VMs that will be deleted from a Cluster. Sahara stops Hadoop services on VMs that will be deleted from a Cluster.
Decommissioning a Data Node may take some time because Hadoop rearranges data replicas Decommissioning a Data Node may take some time because Hadoop rearranges data
around the Cluster, so that no data will be lost after that Data Node is deleted. replicas around the Cluster, so that no data will be lost after that Data Node
is deleted.
6. Deleting Instances 6. Deleting Instances
~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~
@ -132,19 +139,22 @@ Deleting an existing Cluster
1. Deleting 1. Deleting
~~~~~~~~~~~ ~~~~~~~~~~~
The only step, that releases all Cluster's resources and removes it from the database. The only step, that releases all Cluster's resources and removes it from the
database.
Error State Error State
----------- -----------
If the Cluster creation fails, the Cluster will enter the ``Error`` state. If the Cluster creation fails, the Cluster will enter the ``Error`` state.
This status means the Cluster may not be able to perform any operations normally. This status means the Cluster may not be able to perform any operations
This cluster will stay in the database until it is manually deleted. The reason for normally. This cluster will stay in the database until it is manually deleted.
failure may be found in the sahara logs. Also, the status description will contain The reason for failure may be found in the sahara logs. Also, the status
information about the error. description will contain information about the error.
If an error occurs during the ``Adding Instances`` operation, Sahara will first If an error occurs during the ``Adding Instances`` operation, Sahara will first
try to rollback this operation. If a rollback is impossible or fails itself, then try to rollback this operation. If a rollback is impossible or fails itself,
the Cluster will also go into an ``Error`` state. If a rollback was successful, Cluster will get into an ``Active`` state then the Cluster will also go into an ``Error`` state. If a rollback was
and status description will contain a short message about the reason of ``Adding Instances`` failure. successful, Cluster will get into an ``Active`` state and status description
will contain a short message about the reason of ``Adding Instances`` failure.

48
doc/source/userdoc/upgrade.guide.rst
View File

@ -1,9 +1,8 @@
Sahara Upgrade Guide Sahara Upgrade Guide
==================== ====================
This page contains details about upgrading sahara between releases such This page contains details about upgrading sahara between releases such as
as configuration file updates, database migrations, and architectural configuration file updates, database migrations, and architectural changes.
changes.
Icehouse -> Juno Icehouse -> Juno
---------------- ----------------
@ -18,11 +17,11 @@ All-In-One sahara is desired.
Authentication middleware changes Authentication middleware changes
+++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++
The custom auth_token middleware has been deprecated in favor of the The custom auth_token middleware has been deprecated in favor of the keystone
keystone middleware. This change requires an update to the sahara middleware. This change requires an update to the sahara configuration file. To
configuration file. To update your configuration file you should replace update your configuration file you should replace the following parameters from
the following parameters from the ``[DEFAULT]`` section with the new the ``[DEFAULT]`` section with the new parameters in the
parameters in the ``[keystone_authtoken]`` section: ``[keystone_authtoken]`` section:
+----------------------+--------------------+ +----------------------+--------------------+
| Old parameter name | New parameter name | | Old parameter name | New parameter name |
@ -49,12 +48,11 @@ The oslo based code from sahara.openstack.common.db has been replaced by
the usage of the oslo.db package. This change does not require any the usage of the oslo.db package. This change does not require any
update to sahara's configuration file. update to sahara's configuration file.
Additionally, the usage of SQLite databases has been deprecated. Please Additionally, the usage of SQLite databases has been deprecated. Please use
use MySQL or PostgreSQL databases for sahara. SQLite has been MySQL or PostgreSQL databases for sahara. SQLite has been deprecated because it
deprecated because it does not, and is not going to, support the does not, and is not going to, support the ``ALTER COLUMN`` and ``DROP COLUMN``
``ALTER COLUMN`` and ``DROP COLUMN`` commands required for migrations commands required for migrations between versions. For more information please
between versions. For more information please see see http://www.sqlite.org/omitted.html
http://www.sqlite.org/omitted.html
Sahara integration into OpenStack Dashboard Sahara integration into OpenStack Dashboard
+++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++
@ -82,10 +80,9 @@ name for instances created by sahara using HEAT was always 'ec2-user'. As
of Juno, the user name is taken from the image registry as described in of Juno, the user name is taken from the image registry as described in
the :doc:`registering_image` document. the :doc:`registering_image` document.
This change breaks backward compatibility for clusters created using the This change breaks backward compatibility for clusters created using the HEAT
HEAT infrastructure engine prior to the Juno release. Clusters will infrastructure engine prior to the Juno release. Clusters will continue to
continue to operate, but we do not recommended using the scaling operations operate, but we do not recommended using the scaling operations with them.
with them.
Anti affinity implementation changed Anti affinity implementation changed
++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++++++++++++++++++
@ -126,13 +123,14 @@ Kilo -> Liberty
Direct engine deprecation Direct engine deprecation
+++++++++++++++++++++++++ +++++++++++++++++++++++++
In the Liberty release the direct infrastructure engine has been deprecated In the Liberty release the direct infrastructure engine has been deprecated and
and the heat infrastructure engine is now default. This means, that it is preferable the heat infrastructure engine is now default. This means, that it is
to use heat engine instead now. In the Liberty release you can continue to preferable to use heat engine instead now. In the Liberty release you can
operate clusters with the direct engine (create, delete, scale). Using heat engine only continue to operate clusters with the direct engine (create, delete, scale).
the delete operation is available on clusters that were created by the direct engine. Using heat engine only the delete operation is available on clusters that were
After the Liberty release the direct engine will be removed, this means that you will created by the direct engine. After the Liberty release the direct engine will
only be able to delete clusters created with the direct engine. be removed, this means that you will only be able to delete clusters created
with the direct engine.
Policy namespace changed (policy.json) Policy namespace changed (policy.json)
++++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++++++++++++++++++++

12
doc/source/userdoc/vanilla_imagebuilder.rst
View File

@ -26,13 +26,14 @@ Elements for building vanilla images are stored in the
To create vanilla images follow these steps: To create vanilla images follow these steps:
1. Clone repository "https://github.com/openstack/sahara-image-elements" locally. 1. Clone repository "https://github.com/openstack/sahara-image-elements"
locally.
2. Use tox to build images. 2. Use tox to build images.
You can run the command below in sahara-image-elements You can run the command below in sahara-image-elements
directory to build images. By default this script will attempt to create cloud directory to build images. By default this script will attempt to create
images for all versions of supported plugins and all operating systems cloud images for all versions of supported plugins and all operating systems
(subset of Ubuntu, Fedora, and CentOS depending on plugin). (subset of Ubuntu, Fedora, and CentOS depending on plugin).
.. sourcecode:: console .. sourcecode:: console
@ -40,7 +41,10 @@ To create vanilla images follow these steps:
tox -e venv -- sahara-image-create -u tox -e venv -- sahara-image-create -u
Tox will create a virtualenv and install required python packages in it, Tox will create a virtualenv and install required python packages in it,
clone the repositories "https://github.com/openstack/diskimage-builder" and "https://github.com/openstack/sahara-image-elements" and export necessary parameters. clone the repositories "https://github.com/openstack/diskimage-builder" and
"https://github.com/openstack/sahara-image-elements" and export necessary
parameters.
* ``DIB_HADOOP_VERSION`` - version of Hadoop to install * ``DIB_HADOOP_VERSION`` - version of Hadoop to install
* ``JAVA_DOWNLOAD_URL`` - download link for JDK (tarball or bin) * ``JAVA_DOWNLOAD_URL`` - download link for JDK (tarball or bin)
* ``OOZIE_DOWNLOAD_URL`` - download link for OOZIE (we have built * ``OOZIE_DOWNLOAD_URL`` - download link for OOZIE (we have built

7
tox.ini
View File

@ -41,18 +41,13 @@ downloadcache = ~/cache/pip
[testenv:pep8] [testenv:pep8]
commands = commands =
flake8 {posargs} flake8 {posargs}
doc8 doc/source
# Check that .po and .pot files are valid: # Check that .po and .pot files are valid:
bash -c "find sahara -type f -regex '.*\.pot?' -print0|xargs -0 -n 1 msgfmt --check-format -o /dev/null" bash -c "find sahara -type f -regex '.*\.pot?' -print0|xargs -0 -n 1 msgfmt --check-format -o /dev/null"
# Run bashate checks # Run bashate checks
bash -c "find sahara -iname '*.sh' -print0 | xargs -0 bashate -v" bash -c "find sahara -iname '*.sh' -print0 | xargs -0 bashate -v"
bash -c "find devstack -not -name README.rst -and -not -name \*.json -type f -print0 | xargs -0 bashate -v" bash -c "find devstack -not -name README.rst -and -not -name \*.json -type f -print0 | xargs -0 bashate -v"
[testenv:doc8]
deps =
-r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands = doc8 doc/source
[testenv:venv] [testenv:venv]
commands = {posargs} commands = {posargs}