Browse Source

doc: refer to the split plugin documentation

Remove the detailed information about the plugins from the
core documentation and redirect to the plugins documentation instead.

A redirect has been set to not break the existing links.

Change-Id: Ief8593b02242748a5ffd4f55515975faebd19523
Luigi Toscano 1 month ago
parent
commit
bb5f75db7f

+ 1
- 0
doc/source/_extra/.htaccess View File

@@ -4,3 +4,4 @@ redirectmatch 301 ^/sahara/([^/]+)/contributor/launchpad.html$ /sahara/$1/contri
4 4
 redirectmatch 301 ^/sahara/(?!ocata|pike|queens)([^/]+)/user/vanilla-imagebuilder.html$ /sahara/$1/user/vanilla-plugin.html
5 5
 redirectmatch 301 ^/sahara/(?!ocata|pike|queens)([^/]+)/user/cdh-imagebuilder.html$ /sahara/$1/user/cdh-plugin.html
6 6
 redirectmatch 301 ^/sahara/(?!ocata|pike|queens)([^/]+)/user/guest-requirements.html$ /sahara/$1/user/building-guest-images.html
7
+redirectmatch 301 ^/sahara/([^/]+)/user/([^-]+)-plugin.html$ /sahara-plugin-$2/$1/

+ 6
- 0
doc/source/conf.py View File

@@ -60,6 +60,12 @@ openstack_projects = [
60 60
     'neutron',
61 61
     'nova',
62 62
     'oslo.middleware',
63
+    'sahara-plugin-ambari',
64
+    'sahara-plugin-cdh',
65
+    'sahara-plugin-mapr',
66
+    'sahara-plugin-spark',
67
+    'sahara-plugin-storm',
68
+    'sahara-plugin-vanilla',
63 69
     'tooz'
64 70
 ]
65 71
 

+ 0
- 161
doc/source/user/ambari-plugin.rst View File

@@ -1,161 +0,0 @@
1
-
2
-Ambari Plugin
3
-=============
4
-The Ambari sahara plugin provides a way to provision
5
-clusters with Hortonworks Data Platform on OpenStack using templates in a
6
-single click and in an easily repeatable fashion. The sahara controller serves
7
-as the glue between Hadoop and OpenStack. The Ambari plugin mediates between
8
-the sahara controller and Apache Ambari in order to deploy and configure Hadoop
9
-on OpenStack. Core to the HDP Plugin is Apache Ambari
10
-which is used as the orchestrator for deploying HDP on OpenStack. The Ambari
11
-plugin uses Ambari Blueprints for cluster provisioning.
12
-
13
-Apache Ambari Blueprints
14
-------------------------
15
-Apache Ambari Blueprints is a portable document definition, which provides a
16
-complete definition for an Apache Hadoop cluster, including cluster topology,
17
-components, services and their configurations. Ambari Blueprints can be
18
-consumed by the Ambari plugin to instantiate a Hadoop cluster on OpenStack. The
19
-benefits of this approach is that it allows for Hadoop clusters to be
20
-configured and deployed using an Ambari native format that can be used with as
21
-well as outside of OpenStack allowing for clusters to be re-instantiated in a
22
-variety of environments.
23
-
24
-Images
25
-------
26
-
27
-For cluster provisioning, prepared images should be used.
28
-
29
-.. list-table:: Support matrix for the `ambari` plugin
30
-   :widths: 15 15 20 15 35
31
-   :header-rows: 1
32
-
33
-   * - Version
34
-       (image tag)
35
-     - Distribution
36
-     - Build method
37
-     - Version
38
-       (build parameter)
39
-     - Notes
40
-
41
-   * - 2.6
42
-     - Ubuntu 16.04, CentOS 7
43
-     - sahara-image-pack
44
-     - 2.6
45
-     - uses Ambari 2.6
46
-
47
-   * - 2.5
48
-     - Ubuntu 16.04, CentOS 7
49
-     - sahara-image-pack
50
-     - 2.5
51
-     - uses Ambari 2.6
52
-
53
-   * - 2.4
54
-     - Ubuntu 14.04, CentOS 7
55
-     - sahara-image-pack
56
-     - 2.4
57
-     - uses Ambari 2.6
58
-
59
-   * - 2.4
60
-     - Ubuntu 14.04, CentOS 7
61
-     - sahara-image-create
62
-     - 2.4
63
-     - uses Ambari 2.2.1.0
64
-
65
-   * - 2.3
66
-     - Ubuntu 14.04, CentOS 7
67
-     - sahara-image-pack
68
-     - 2.3
69
-     - uses Ambari 2.4
70
-
71
-   * - 2.3
72
-     - Ubuntu 14.04, CentOS 7
73
-     - sahara-image-create
74
-     - 2.3
75
-     - uses Ambari 2.2.0.0
76
-
77
-For more information about building image, refer to
78
-:doc:`building-guest-images`.
79
-
80
-HDP plugin requires an image to be tagged in sahara Image Registry with two
81
-tags: 'ambari' and '<plugin version>' (e.g. '2.5').
82
-
83
-The image requires a username. For more information, refer to the
84
-:doc:`registering-image` section.
85
-
86
-To speed up provisioning, the HDP packages can be pre-installed on the image
87
-used. The packages' versions depend on the HDP version required.
88
-
89
-High Availability for HDFS and YARN
90
------------------------------------
91
-High Availability (Using the Quorum Journal Manager) can be
92
-deployed automatically with the Ambari plugin. You can deploy High Available
93
-cluster through UI by selecting ``NameNode HA`` and/or ``ResourceManager HA``
94
-options in general configs of cluster template.
95
-
96
-The NameNode High Availability is deployed using 2 NameNodes, one active and
97
-one standby. The NameNodes use a set of JournalNodes and Zookepeer Servers to
98
-ensure the necessary synchronization. In case of ResourceManager HA 2
99
-ResourceManagers should be enabled in addition.
100
-
101
-A typical Highly available Ambari cluster uses 2 separate NameNodes, 2 separate
102
-ResourceManagers and at least 3 JournalNodes and at least 3 Zookeeper Servers.
103
-
104
-HDP Version Support
105
--------------------
106
-The HDP plugin currently supports deployment of HDP 2.3, 2.4 and 2.5.
107
-
108
-Cluster Validation
109
-------------------
110
-Prior to Hadoop cluster creation, the HDP plugin will perform the following
111
-validation checks to ensure a successful Hadoop deployment:
112
-
113
-* Ensure the existence of Ambari Server process in the cluster;
114
-* Ensure the existence of a NameNode, Zookeeper, ResourceManagers processes
115
-  HistoryServer and App TimeLine Server in the cluster
116
-
117
-Enabling Kerberos security for cluster
118
---------------------------------------
119
-
120
-If you want to protect your clusters using MIT Kerberos security you have to
121
-complete a few steps below.
122
-
123
-* If you would like to create a cluster protected by Kerberos security you
124
-  just need to enable Kerberos by checkbox in the ``General Parameters``
125
-  section of the cluster configuration. If you prefer to use the OpenStack CLI
126
-  for cluster creation, you have to put the data below in the
127
-  ``cluster_configs`` section:
128
-
129
-  .. sourcecode:: console
130
-
131
-     "cluster_configs": {
132
-       "Enable Kerberos Security": true,
133
-     }
134
-
135
-  Sahara in this case will correctly prepare KDC server and will create
136
-  principals along with keytabs to enable authentication for Hadoop services.
137
-
138
-* Ensure that you have the latest hadoop-openstack jar file distributed
139
-  on your cluster nodes. You can download one at
140
-  ``https://tarballs.openstack.org/sahara-extra/dist/``
141
-
142
-* Sahara will create principals along with keytabs for system users
143
-  like ``oozie``, ``hdfs`` and ``spark`` so that you will not have to
144
-  perform additional auth operations to execute your jobs on top of the
145
-  cluster.
146
-
147
-Adjusting Ambari Agent Package Installation timeout Parameter
148
--------------------------------------------------------------
149
-
150
-For a cluster with large number of nodes or slow connectivity to HDP repo
151
-server, a Sahara HDP Cluster creation  may fail due to ambari agent
152
-reaching the timeout threshold while installing the packages in the nodes.
153
-
154
-Such failures will occur during the "cluster start"  stage which can be
155
-monitored from Cluster Events tab of Sahara Dashboard. The timeout error will
156
-be visible from the Ambari Dashboard as well.
157
-
158
-* To avoid the package installation timeout by ambari agent you need to change
159
-  the default value of ``Ambari Agent Package Install timeout`` parameter which
160
-  can be found in the ``General Parameters`` section of the cluster template
161
-  configuration.

+ 0
- 190
doc/source/user/cdh-plugin.rst View File

@@ -1,190 +0,0 @@
1
-Cloudera Plugin
2
-===============
3
-
4
-The Cloudera plugin is a Sahara plugin which allows the user to
5
-deploy and operate a cluster with Cloudera Manager.
6
-
7
-The Cloudera plugin is enabled in Sahara by default. You can manually
8
-modify the Sahara configuration file (default /etc/sahara/sahara.conf) to
9
-explicitly enable or disable it in "plugins" line.
10
-
11
-Images
12
-------
13
-
14
-For cluster provisioning, prepared images should be used.
15
-
16
-.. list-table:: Support matrix for the `vanilla` plugin
17
-   :widths: 15 15 20 15 35
18
-   :header-rows: 1
19
-
20
-   * - Version
21
-       (image tag)
22
-     - Distribution
23
-     - Build method
24
-     - Version
25
-       (build parameter)
26
-     - Notes
27
-
28
-   * - 5.13.0
29
-     - Ubuntu 16.04, CentOS 7
30
-     - sahara-image-pack
31
-     - 5.13.0
32
-     -
33
-
34
-   * - 5.11.0
35
-     - Ubuntu 16.04, CentOS 7
36
-     - sahara-image-pack, sahara-image-create
37
-     - 5.11.0
38
-     -
39
-
40
-   * - 5.9.0
41
-     - Ubuntu 14.04, CentOS 7
42
-     - sahara-image-pack, sahara-image-create
43
-     - 5.9.0
44
-     -
45
-
46
-   * - 5.7.0
47
-     - Ubuntu 14.04, CentOS 7
48
-     - sahara-image-pack, sahara-image-create
49
-     - 5.7.0
50
-     -
51
-
52
-For more information about building image, refer to
53
-:doc:`building-guest-images`.
54
-
55
-The cloudera plugin requires an image to be tagged in Sahara Image Registry
56
-with two tags: 'cdh' and '<cloudera version>' (e.g. '5.13.0', '5.11.0',
57
-'5.9.0', etc).
58
-
59
-The default username specified for these images is different for each
60
-distribution. For more information, refer to the
61
-:doc:`registering-image` section.
62
-
63
-Build settings
64
-~~~~~~~~~~~~~~
65
-
66
-It is possible to specify minor versions of CDH when ``sahara-image-create``
67
-is used.
68
-If you want to use a minor versions, export ``DIB_CDH_MINOR_VERSION``
69
-before starting the build command, e.g.:
70
-
71
-   .. sourcecode:: console
72
-
73
-      export DIB_CDH_MINOR_VERSION=5.7.1
74
-
75
-Services Supported
76
-------------------
77
-
78
-Currently below services are supported in both versions of Cloudera plugin:
79
-HDFS, Oozie, YARN, Spark, Zookeeper, Hive, Hue, HBase. 5.3.0 version of
80
-Cloudera Plugin also supported following services: Impala, Flume, Solr, Sqoop,
81
-and Key-value Store Indexer. In version 5.4.0 KMS service support was added
82
-based on version 5.3.0. Kafka 2.0.2 was added for CDH 5.5 and higher.
83
-
84
-.. note::
85
-
86
-    Sentry service is enabled in Cloudera plugin. However, as we do not enable
87
-    Kerberos authentication in the cluster for CDH version < 5.5 (which is
88
-    required for Sentry functionality) then using Sentry service will not
89
-    really take any effect, and other services depending on Sentry will not do
90
-    any authentication too.
91
-
92
-High Availability Support
93
--------------------------
94
-
95
-Currently HDFS NameNode High Availability is supported beginning with
96
-Cloudera 5.4.0 version.  You can refer to :doc:`features` for the detail
97
-info.
98
-
99
-YARN ResourceManager High Availability is supported beginning with Cloudera
100
-5.4.0 version. This feature adds redundancy in the form of an Active/Standby
101
-ResourceManager pair to avoid the failure of single RM. Upon failover, the
102
-Standby RM become Active so that the applications can resume from their last
103
-check-pointed state.
104
-
105
-Cluster Validation
106
-------------------
107
-
108
-When the user performs an operation on the cluster using a Cloudera plugin, the
109
-cluster topology requested by the user is verified for consistency.
110
-
111
-The following limitations are required in the cluster topology for all
112
-cloudera plugin versions:
113
-
114
-+ Cluster must contain exactly one manager.
115
-+ Cluster must contain exactly one namenode.
116
-+ Cluster must contain exactly one secondarynamenode.
117
-+ Cluster must contain at least ``dfs_replication`` datanodes.
118
-+ Cluster can contain at most one resourcemanager and this process is also
119
-  required by nodemanager.
120
-+ Cluster can contain at most one jobhistory and this process is also
121
-  required for resourcemanager.
122
-+ Cluster can contain at most one oozie and this process is also required
123
-  for EDP.
124
-+ Cluster can't contain oozie without datanode.
125
-+ Cluster can't contain oozie without nodemanager.
126
-+ Cluster can't contain oozie without jobhistory.
127
-+ Cluster can't contain hive on the cluster without the following services:
128
-  metastore, hive server, webcat and resourcemanager.
129
-+ Cluster can contain at most one hue server.
130
-+ Cluster can't contain hue server without hive service and oozie.
131
-+ Cluster can contain at most one spark history server.
132
-+ Cluster can't contain spark history server without resourcemanager.
133
-+ Cluster can't contain hbase master service without at least one zookeeper
134
-  and at least one hbase regionserver.
135
-+ Cluster can't contain hbase regionserver without at least one hbase maser.
136
-
137
-In case of 5.3.0, 5.4.0, 5.5.0, 5.7.x or 5.9.x version of Cloudera Plugin
138
-there are few extra limitations in the cluster topology:
139
-
140
-+ Cluster can't contain flume without at least one datanode.
141
-+ Cluster can contain at most one sentry server service.
142
-+ Cluster can't contain sentry server service without at least one zookeeper
143
-  and at least one datanode.
144
-+ Cluster can't contain solr server without at least one zookeeper and at
145
-  least one datanode.
146
-+ Cluster can contain at most one sqoop server.
147
-+ Cluster can't contain sqoop server without at least one datanode,
148
-  nodemanager and jobhistory.
149
-+ Cluster can't contain hbase indexer without at least one datanode,
150
-  zookeeper, solr server and hbase master.
151
-+ Cluster can contain at most one impala catalog server.
152
-+ Cluster can contain at most one impala statestore.
153
-+ Cluster can't contain impala catalogserver without impala statestore,
154
-  at least one impalad service, at least one datanode, and metastore.
155
-+ If using Impala, the daemons must be installed on every datanode.
156
-
157
-In case of version 5.5.0, 5.7.x or 5.9.x of Cloudera Plugin additional
158
-services in the cluster topology are available:
159
-
160
-+ Cluster can have the kafka service and several kafka brokers.
161
-
162
-Enabling Kerberos security for cluster
163
---------------------------------------
164
-
165
-If you want to protect your clusters using MIT Kerberos security you have to
166
-complete a few steps below.
167
-
168
-* If you would like to create a cluster protected by Kerberos security you
169
-  just need to enable Kerberos by checkbox in the ``General Parameters``
170
-  section of the cluster configuration. If you prefer to use the OpenStack CLI
171
-  for cluster creation, you have to put the data below in the
172
-  ``cluster_configs`` section:
173
-
174
-  .. sourcecode:: console
175
-
176
-     "cluster_configs": {
177
-       "Enable Kerberos Security": true,
178
-     }
179
-
180
-  Sahara in this case will correctly prepare KDC server and will create
181
-  principals along with keytabs to enable authentication for Hadoop services.
182
-
183
-* Ensure that you have the latest hadoop-openstack jar file distributed
184
-  on your cluster nodes. You can download one at
185
-  ``https://tarballs.openstack.org/sahara-extra/dist/``
186
-
187
-* Sahara will create principals along with keytabs for system users
188
-  like ``hdfs`` and ``spark`` so that you will not have to
189
-  perform additional auth operations to execute your jobs on top of the
190
-  cluster.

+ 0
- 6
doc/source/user/index.rst View File

@@ -24,12 +24,6 @@ Plugins
24 24
    :maxdepth: 2
25 25
 
26 26
    plugins
27
-   vanilla-plugin
28
-   ambari-plugin
29
-   spark-plugin
30
-   storm-plugin
31
-   cdh-plugin
32
-   mapr-plugin
33 27
 
34 28
 
35 29
 Elastic Data Processing

+ 0
- 128
doc/source/user/mapr-plugin.rst View File

@@ -1,128 +0,0 @@
1
-MapR Distribution Plugin
2
-========================
3
-The MapR Sahara plugin allows to provision MapR clusters on
4
-OpenStack in an easy way and do it, quickly, conveniently and simply.
5
-
6
-Operation
7
----------
8
-The MapR Plugin performs the following four primary functions during cluster
9
-creation:
10
-
11
-1. MapR components deployment - the plugin manages the deployment of the
12
-   required software to the target VMs
13
-2. Services Installation - MapR services are installed according to provided
14
-   roles list
15
-3. Services Configuration - the plugin combines default settings with user
16
-   provided settings
17
-4. Services Start - the plugin starts appropriate services according to
18
-   specified roles
19
-
20
-Images
21
-------
22
-The Sahara MapR plugin can make use of either minimal (operating system only)
23
-images or pre-populated MapR images. The base requirement for both is that the
24
-image is cloud-init enabled and contains a supported operating system (see
25
-http://maprdocs.mapr.com/home/InteropMatrix/r_os_matrix.html).
26
-
27
-The advantage of a pre-populated image is that provisioning time is reduced, as
28
-packages do not need to be downloaded which make up the majority of the time
29
-spent in the provisioning cycle. In addition, provisioning large clusters will
30
-put a burden on the network as packages for all nodes need to be downloaded
31
-from the package repository.
32
-
33
-
34
-.. list-table:: Support matrix for the `mapr` plugin
35
-   :widths: 15 15 20 15 35
36
-   :header-rows: 1
37
-
38
-   * - Version
39
-       (image tag)
40
-     - Distribution
41
-     - Build method
42
-     - Version
43
-       (build parameter)
44
-     - Notes
45
-
46
-   * - 5.2.0.mrv2
47
-     - Ubuntu 14.04, CentOS 7
48
-     - sahara-image-pack
49
-     - 5.2.0.mrv2
50
-     -
51
-
52
-   * - 5.2.0.mrv2
53
-     - Ubuntu 14.04, CentOS 7
54
-     - sahara-image-create
55
-     - 5.2.0
56
-     -
57
-
58
-For more information about building image, refer to
59
-:doc:`building-guest-images`.
60
-
61
-MapR plugin needs an image to be tagged in Sahara Image Registry with
62
-two tags: 'mapr' and '<MapR version>' (e.g. '5.2.0.mrv2').
63
-
64
-The default username specified for these images is different for each
65
-distribution. For more information, refer to the
66
-:doc:`registering-image` section.
67
-
68
-Hadoop Version Support
69
-----------------------
70
-The MapR plugin currently supports Hadoop 2.7.0 (5.2.0.mrv2).
71
-
72
-Cluster Validation
73
-------------------
74
-When the user creates or scales a Hadoop cluster using a mapr plugin, the
75
-cluster topology requested by the user is verified for consistency.
76
-
77
-Every MapR cluster must contain:
78
-
79
-* at least 1 *CLDB* process
80
-* exactly 1 *Webserver* process
81
-* odd number of *ZooKeeper* processes but not less than 1
82
-* *FileServer* process on every node
83
-* at least 1 ephemeral drive (then you need to specify the ephemeral drive in
84
-  the flavor not on the node group template creation) or 1 Cinder volume
85
-  per instance
86
-
87
-Every Hadoop cluster must contain exactly 1 *Oozie* process
88
-
89
-Every MapReduce v1 cluster must contain:
90
-
91
-* at least 1 *JobTracker* process
92
-* at least 1 *TaskTracker* process
93
-
94
-Every MapReduce v2 cluster must contain:
95
-
96
-* exactly 1 *ResourceManager* process
97
-* exactly 1 *HistoryServer* process
98
-* at least 1 *NodeManager* process
99
-
100
-Every Spark cluster must contain:
101
-
102
-* exactly 1 *Spark Master* process
103
-* exactly 1 *Spark HistoryServer* process
104
-* at least 1 *Spark Slave* (worker) process
105
-
106
-HBase service is considered valid if:
107
-
108
-* cluster has at least 1 *HBase-Master* process
109
-* cluster has at least 1 *HBase-RegionServer* process
110
-
111
-Hive service is considered valid if:
112
-
113
-* cluster has exactly 1 *HiveMetastore* process
114
-* cluster has exactly 1 *HiveServer2* process
115
-
116
-Hue service is considered valid if:
117
-
118
-* cluster has exactly 1 *Hue* process
119
-* *Hue* process resides on the same node as *HttpFS* process
120
-
121
-HttpFS service is considered valid if cluster has exactly 1 *HttpFS* process
122
-
123
-Sqoop service is considered valid if cluster has exactly 1 *Sqoop2-Server*
124
-process
125
-
126
-The MapR Plugin
127
----------------
128
-For more information, please contact MapR.

+ 14
- 6
doc/source/user/plugins.rst View File

@@ -6,12 +6,20 @@ enables sahara to deploy a specific data processing framework (for example,
6 6
 Hadoop) or distribution, and allows configuration of topology and
7 7
 management/monitoring tools.
8 8
 
9
-* :doc:`vanilla-plugin` - deploys Vanilla Apache Hadoop
10
-* :doc:`ambari-plugin` - deploys Hortonworks Data Platform
11
-* :doc:`spark-plugin` - deploys Apache Spark with Cloudera HDFS
12
-* :doc:`storm-plugin` - deploys Apache Storm
13
-* :doc:`mapr-plugin` - deploys MapR plugin with MapR File System
14
-* :doc:`cdh-plugin` - deploys Cloudera Hadoop
9
+The plugins currently developed as part of the official Sahara project are:
10
+
11
+* :sahara-plugin-ambari-doc:`Ambari Plugin <>` -
12
+  deploys Hortonworks Data Platform
13
+* :sahara-plugin-cdh-doc:`CDH Plugin <>` -
14
+  deploys Cloudera Hadoop
15
+* :sahara-plugin-mapr-doc:`MapR Plugin <>` -
16
+  deploys MapR plugin with MapR File System
17
+* :sahara-plugin-spark-doc:`Spark Plugin <>` -
18
+  deploys Apache Spark with Cloudera HDFS
19
+* :sahara-plugin-storm-doc:`Storm Plugin <>` -
20
+  deploys Apache Storm
21
+* :sahara-plugin-vanilla-doc:`Vanilla Plugin <>` -
22
+  deploys Vanilla Apache Hadoop
15 23
 
16 24
 Managing plugins
17 25
 ----------------

+ 0
- 91
doc/source/user/spark-plugin.rst View File

@@ -1,91 +0,0 @@
1
-Spark Plugin
2
-============
3
-
4
-The Spark plugin for sahara provides a way to provision Apache Spark clusters
5
-on OpenStack in a single click and in an easily repeatable fashion.
6
-
7
-Currently Spark is installed in standalone mode, with no YARN or Mesos
8
-support.
9
-
10
-Images
11
-------
12
-
13
-For cluster provisioning, prepared images should be used.
14
-
15
-.. list-table:: Support matrix for the `spark` plugin
16
-   :widths: 15 15 20 15 35
17
-   :header-rows: 1
18
-
19
-   * - Version
20
-       (image tag)
21
-     - Distribution
22
-     - Build method
23
-     - Version
24
-       (build parameter)
25
-     - Notes
26
-
27
-   * - 2.3
28
-     - Ubuntu 16.04
29
-     - sahara-image-create
30
-     - 2.3.0
31
-     - based on CDH 5.11
32
-
33
-   * - 2.2
34
-     - Ubuntu 16.04
35
-     - sahara-image-create
36
-     - 2.2.0
37
-     - based on CDH 5.11
38
-
39
-For more information about building image, refer to
40
-:doc:`building-guest-images`.
41
-
42
-The Spark plugin requires an image to be tagged in the sahara image registry
43
-with two tags: 'spark' and '<Spark version>' (e.g. '1.6.0').
44
-
45
-The image requires a username. For more information, refer to the
46
-:doc:`registering-image` section.
47
-
48
-Note that the Spark cluster is deployed using the scripts available in the
49
-Spark distribution, which allow the user to start all services (master and
50
-slaves), stop all services and so on. As such (and as opposed to CDH HDFS
51
-daemons), Spark is not deployed as a standard Ubuntu service and if the
52
-virtual machines are rebooted, Spark will not be restarted.
53
-
54
-Build settings
55
-~~~~~~~~~~~~~~
56
-
57
-When ``sahara-image-create`` is used, you can override few settings
58
-by exporting the corresponding environment variables
59
-before starting the build command:
60
-
61
-* ``SPARK_DOWNLOAD_URL`` - download link for Spark
62
-
63
-Spark configuration
64
--------------------
65
-
66
-Spark needs few parameters to work and has sensible defaults. If needed they
67
-can be changed when creating the sahara cluster template. No node group
68
-options are available.
69
-
70
-Once the cluster is ready, connect with ssh to the master using the `ubuntu`
71
-user and the appropriate ssh key. Spark is installed in `/opt/spark` and
72
-should be completely configured and ready to start executing jobs. At the
73
-bottom of the cluster information page from the OpenStack dashboard, a link to
74
-the Spark web interface is provided.
75
-
76
-Cluster Validation
77
-------------------
78
-
79
-When a user creates an Hadoop cluster using the Spark plugin, the cluster
80
-topology requested by user is verified for consistency.
81
-
82
-Currently there are the following limitations in cluster topology for the
83
-Spark plugin:
84
-
85
-+ Cluster must contain exactly one HDFS namenode
86
-+ Cluster must contain exactly one Spark master
87
-+ Cluster must contain at least one Spark slave
88
-+ Cluster must contain at least one HDFS datanode
89
-
90
-The tested configuration co-locates the NameNode with the master and a
91
-DataNode with each slave to maximize data locality.

+ 0
- 82
doc/source/user/storm-plugin.rst View File

@@ -1,82 +0,0 @@
1
-Storm Plugin
2
-============
3
-
4
-The Storm plugin for sahara provides a way to provision Apache Storm clusters
5
-on OpenStack in a single click and in an easily repeatable fashion.
6
-
7
-Currently Storm is installed in standalone mode, with no YARN support.
8
-
9
-Images
10
-------
11
-
12
-For cluster provisioning, prepared images should be used.
13
-
14
-.. list-table:: Support matrix for the `storm` plugin
15
-   :widths: 15 15 20 15 35
16
-   :header-rows: 1
17
-
18
-   * - Version
19
-       (image tag)
20
-     - Distribution
21
-     - Build method
22
-     - Version
23
-       (build parameter)
24
-     - Notes
25
-
26
-   * - 1.2
27
-     - Ubuntu 16.04
28
-     - sahara-image-create
29
-     - 1.2.1, 1.2.0
30
-     - both versions are supported by the same image tag
31
-
32
-   * - 1.1.0
33
-     - Ubuntu 16.04
34
-     - sahara-image-create
35
-     - 1.1.1, 1.1.0
36
-     - both versions are supported by the same image tag
37
-
38
-For more information about building image, refer to
39
-:doc:`building-guest-images`.
40
-
41
-The Storm plugin requires an image to be tagged in the sahara image registry
42
-with two tags: 'storm' and '<Storm version>' (e.g. '1.1.0').
43
-
44
-The image requires a username. For more information, refer to the
45
-:doc:`registering-image` section.
46
-
47
-Note that the Storm cluster is deployed using the scripts available in the
48
-Storm distribution, which allow the user to start all services (nimbus,
49
-supervisors and zookeepers), stop all services and so on. As such Storm is not
50
-deployed as a standard Ubuntu service and if the virtual machines are rebooted,
51
-Storm will not be restarted.
52
-
53
-Storm configuration
54
--------------------
55
-
56
-Storm needs few parameters to work and has sensible defaults. If needed they
57
-can be changed when creating the sahara cluster template. No node group
58
-options are available.
59
-
60
-Once the cluster is ready, connect with ssh to the master using the `ubuntu`
61
-user and the appropriate ssh key. Storm is installed in `/usr/local/storm` and
62
-should be completely configured and ready to start executing jobs. At the
63
-bottom of the cluster information page from the OpenStack dashboard, a link to
64
-the Storm web interface is provided.
65
-
66
-Cluster Validation
67
-------------------
68
-
69
-When a user creates a Storm cluster using the Storm plugin, the cluster
70
-topology requested by user is verified for consistency.
71
-
72
-Currently there are the following limitations in cluster topology for the
73
-Storm plugin:
74
-
75
-+ Cluster must contain exactly one Storm nimbus
76
-+ Cluster must contain at least one Storm supervisor
77
-+ Cluster must contain at least one Zookeeper node
78
-
79
-The tested configuration has nimbus, supervisor, and Zookeeper processes each
80
-running on their own nodes.
81
-Another possible configuration is one node with nimbus alone, and additional
82
-nodes each with supervisor and Zookeeper processes together.

+ 0
- 111
doc/source/user/vanilla-plugin.rst View File

@@ -1,111 +0,0 @@
1
-Vanilla Plugin
2
-==============
3
-
4
-The vanilla plugin is a reference implementation which allows users to operate
5
-a cluster with Apache Hadoop.
6
-
7
-Since the Newton release Spark is integrated into the Vanilla plugin so you
8
-can launch Spark jobs on a Vanilla cluster.
9
-
10
-Images
11
-------
12
-
13
-For cluster provisioning, prepared images should be used.
14
-
15
-.. list-table:: Support matrix for the `vanilla` plugin
16
-   :widths: 15 15 20 15 35
17
-   :header-rows: 1
18
-
19
-   * - Version
20
-       (image tag)
21
-     - Distribution
22
-     - Build method
23
-     - Version
24
-       (build parameter)
25
-     - Notes
26
-
27
-   * - 2.8.2
28
-     - Ubuntu 16.04, CentOS 7
29
-     - sahara-image-create
30
-     - 2.8.2
31
-     - Hive 2.3.2, Oozie 4.3.0
32
-
33
-   * - 2.7.5
34
-     - Ubuntu 16.04, CentOS 7
35
-     - sahara-image-create
36
-     - 2.7.5
37
-     - Hive 2.3.2, Oozie 4.3.0
38
-
39
-   * - 2.7.1
40
-     - Ubuntu 16.04, CentOS 7
41
-     - sahara-image-create
42
-     - 2.7.1
43
-     - Hive 0.11.0, Oozie 4.2.0
44
-
45
-For more information about building image, refer to
46
-:doc:`building-guest-images`.
47
-
48
-Vanilla plugin requires an image to be tagged in Sahara Image Registry with
49
-two tags: 'vanilla' and '<hadoop version>' (e.g. '2.7.1').
50
-
51
-The image requires a username. For more information, refer to the
52
-:doc:`registering-image` section.
53
-
54
-Build settings
55
-~~~~~~~~~~~~~~
56
-
57
-When ``sahara-image-create`` is used, you can override few settings
58
-by exporting the corresponding environment variables
59
-before starting the build command:
60
-
61
-* ``DIB_HADOOP_VERSION`` - version of Hadoop to install
62
-* ``HIVE_VERSION`` - version of Hive to install
63
-* ``OOZIE_DOWNLOAD_URL`` - download link for Oozie (we have built
64
-  Oozie libs here: https://tarballs.openstack.org/sahara-extra/dist/oozie/)
65
-* ``SPARK_DOWNLOAD_URL`` - download link for Spark
66
-
67
-Vanilla Plugin Requirements
68
----------------------------
69
-
70
-The image building tools described in :ref:`building-guest-images-label`
71
-add the required software to the image and their usage is strongly suggested.
72
-Nevertheless, here are listed the software that should be pre-loaded
73
-on the guest image so that it can be used to create Vanilla clusters:
74
-
75
-* ssh-client installed
76
-* Java (version >= 7)
77
-* Apache Hadoop installed
78
-* 'hadoop' user created
79
-
80
-See :doc:`hadoop-swift` for information on using Swift with your sahara cluster
81
-(for EDP support Swift integration is currently required).
82
-
83
-To support EDP, the following components must also be installed on the guest:
84
-
85
-* Oozie version 4 or higher
86
-* mysql/mariadb
87
-* hive
88
-
89
-Cluster Validation
90
-------------------
91
-
92
-When user creates or scales a Hadoop cluster using a Vanilla plugin,
93
-the cluster topology requested by user is verified for consistency.
94
-
95
-Currently there are the following limitations in cluster topology for Vanilla
96
-plugin:
97
-
98
-For Vanilla Hadoop version 2.x.x:
99
-
100
-+ Cluster must contain exactly one namenode
101
-+ Cluster can contain at most one resourcemanager
102
-+ Cluster can contain at most one secondary namenode
103
-+ Cluster can contain at most one historyserver
104
-+ Cluster can contain at most one oozie and this process is also required
105
-  for EDP
106
-+ Cluster can't contain oozie without resourcemanager and without
107
-  historyserver
108
-+ Cluster can't have nodemanager nodes if it doesn't have resourcemanager
109
-+ Cluster can have at most one hiveserver node.
110
-+ Cluster can have at most one spark history server and this process is also
111
-  required for Spark EDP (Spark is available since the Newton release).

+ 2
- 0
doc/test/redirect-tests.txt View File

@@ -5,3 +5,5 @@
5 5
 /sahara/latest/user/cdh-imagebuilder.html 301 /sahara/latest/user/cdh-plugin.html
6 6
 /sahara/latest/user/guest-requirements.html 301 /sahara/latest/user/building-guest-images.html
7 7
 /sahara/rocky/user/guest-requirements.html 301 /sahara/rocky/user/building-guest-images.html
8
+/sahara/latest/user/vanilla-plugin.html 301 /sahara-plugin-vanilla/latest/
9
+/sahara/stein/user/storm-plugin.html 301 /sahara-plugin-storm/stein/

Loading…
Cancel
Save