Merge "Restructure the documentation according the new spec"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
146521b9a6
|
|
@ -0,0 +1,95 @@
|
||||||
|
.. _diskimage-builder-label:
|
||||||
|
|
||||||
|
Building Images for Vanilla Plugin
|
||||||
|
==================================
|
||||||
|
|
||||||
|
In this document you will find instruction on how to build Ubuntu, Fedora, and
|
||||||
|
CentOS images with Apache Hadoop version 2.x.x.
|
||||||
|
|
||||||
|
As of now the vanilla plugin works with images with pre-installed versions of
|
||||||
|
Apache Hadoop. To simplify the task of building such images we use
|
||||||
|
`Disk Image Builder <https://github.com/openstack/diskimage-builder>`_.
|
||||||
|
|
||||||
|
`Disk Image Builder` builds disk images using elements. An element is a
|
||||||
|
particular set of code that alters how the image is built, or runs within the
|
||||||
|
chroot to prepare the image.
|
||||||
|
|
||||||
|
Elements for building vanilla images are stored in the
|
||||||
|
`Sahara image elements repository <https://github.com/openstack/sahara-image-elements>`_
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
Sahara requires images with cloud-init package installed:
|
||||||
|
|
||||||
|
* `For Fedora <http://pkgs.fedoraproject.org/cgit/cloud-init.git/>`_
|
||||||
|
* `For Ubuntu 14 <http://packages.ubuntu.com/trusty/cloud-init>`_
|
||||||
|
* `For CentOS 6 <http://mirror.centos.org/centos/6/extras/x86_64/Packages/cloud-init-0.7.5-10.el6.centos.2.x86_64.rpm>`_
|
||||||
|
* `For CentOS 7 <http://mirror.centos.org/centos/7/extras/x86_64/Packages/cloud-init-0.7.5-10.el7.centos.1.x86_64.rpm>`_
|
||||||
|
|
||||||
|
To create vanilla images follow these steps:
|
||||||
|
|
||||||
|
1. Clone repository "https://github.com/openstack/sahara-image-elements"
|
||||||
|
locally.
|
||||||
|
|
||||||
|
2. Use tox to build images.
|
||||||
|
|
||||||
|
You can run the command below in sahara-image-elements
|
||||||
|
directory to build images. By default this script will attempt to create
|
||||||
|
cloud images for all versions of supported plugins and all operating systems
|
||||||
|
(subset of Ubuntu, Fedora, and CentOS depending on plugin).
|
||||||
|
|
||||||
|
.. sourcecode:: console
|
||||||
|
|
||||||
|
tox -e venv -- sahara-image-create -u
|
||||||
|
|
||||||
|
If you want to build Vanilla 2.7.1 image with centos 7 just execute:
|
||||||
|
|
||||||
|
.. sourcecode:: console
|
||||||
|
|
||||||
|
tox -e venv -- sahara-image-create -p vanilla -v 2.7.1 -i centos7
|
||||||
|
|
||||||
|
Tox will create a virtualenv and install required python packages in it,
|
||||||
|
clone the repositories "https://github.com/openstack/diskimage-builder" and
|
||||||
|
"https://github.com/openstack/sahara-image-elements" and export necessary
|
||||||
|
parameters.
|
||||||
|
|
||||||
|
* ``DIB_HADOOP_VERSION`` - version of Hadoop to install
|
||||||
|
* ``JAVA_DOWNLOAD_URL`` - download link for JDK (tarball or bin)
|
||||||
|
* ``OOZIE_DOWNLOAD_URL`` - download link for OOZIE (we have built
|
||||||
|
Oozie libs here: ``http://sahara-files.mirantis.com/oozie-4.2.0-hadoop-2.7.1.tar.gz``)
|
||||||
|
* ``SPARK_DOWNLOAD_URL`` - download link for Spark
|
||||||
|
* ``HIVE_VERSION`` - version of Hive to install
|
||||||
|
(currently supports only 0.11.0)
|
||||||
|
* ``ubuntu_image_name``
|
||||||
|
* ``fedora_image_name``
|
||||||
|
* ``DIB_IMAGE_SIZE`` - parameter that specifies a volume of hard disk
|
||||||
|
of instance. You need to specify it only for Fedora because Fedora
|
||||||
|
doesn't use all available volume
|
||||||
|
* ``DIB_COMMIT_ID`` - latest commit id of diskimage-builder project
|
||||||
|
* ``SAHARA_ELEMENTS_COMMIT_ID`` - latest commit id of
|
||||||
|
sahara-image-elements project
|
||||||
|
|
||||||
|
NOTE: If you don't want to use default values, you should set your values
|
||||||
|
of parameters.
|
||||||
|
|
||||||
|
Then it will create required cloud images using image elements that install
|
||||||
|
all the necessary packages and configure them. You will find created images
|
||||||
|
in the parent directory.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
Disk Image Builder will generate QCOW2 images, used with the default
|
||||||
|
OpenStack Qemu/KVM hypervisors. If your OpenStack uses a different
|
||||||
|
hypervisor, the generated image should be converted to an appropriate
|
||||||
|
format.
|
||||||
|
|
||||||
|
VMware Nova backend requires VMDK image format. You may use qemu-img
|
||||||
|
utility to convert a QCOW2 image to VMDK.
|
||||||
|
|
||||||
|
.. sourcecode:: console
|
||||||
|
|
||||||
|
qemu-img convert -O vmdk <original_image>.qcow2 <converted_image>.vmdk
|
||||||
|
|
||||||
|
|
||||||
|
For finer control of diskimage-create.sh see the `official documentation
|
||||||
|
<https://github.com/openstack/sahara-image-elements/blob/master/diskimage-create/README.rst>`_
|
||||||
|
|
@ -0,0 +1,56 @@
|
||||||
|
Vanilla Plugin
|
||||||
|
==============
|
||||||
|
|
||||||
|
The vanilla plugin is a reference implementation which allows users to operate
|
||||||
|
a cluster with Apache Hadoop.
|
||||||
|
|
||||||
|
Since the Newton release Spark is integrated into the Vanilla plugin so you
|
||||||
|
can launch Spark jobs on a Vanilla cluster.
|
||||||
|
|
||||||
|
For cluster provisioning prepared images should be used. They already have
|
||||||
|
Apache Hadoop 2.7.1 installed.
|
||||||
|
|
||||||
|
You may build images by yourself using :doc:`vanilla-imagebuilder` or you could
|
||||||
|
download prepared images from http://sahara-files.mirantis.com/images/upstream
|
||||||
|
|
||||||
|
Vanilla plugin requires an image to be tagged in Sahara Image Registry with
|
||||||
|
two tags: 'vanilla' and '<hadoop version>' (e.g. '2.7.1').
|
||||||
|
|
||||||
|
The default username specified for these images is different
|
||||||
|
for each distribution:
|
||||||
|
|
||||||
|
+--------------+------------+
|
||||||
|
| OS | username |
|
||||||
|
+==============+============+
|
||||||
|
| Ubuntu 14 | ubuntu |
|
||||||
|
+--------------+------------+
|
||||||
|
| Fedora 20 | fedora |
|
||||||
|
+--------------+------------+
|
||||||
|
| CentOS 6 | cloud-user |
|
||||||
|
+--------------+------------+
|
||||||
|
| CentOS 7 | centos |
|
||||||
|
+--------------+------------+
|
||||||
|
|
||||||
|
Cluster Validation
|
||||||
|
------------------
|
||||||
|
|
||||||
|
When user creates or scales a Hadoop cluster using a Vanilla plugin,
|
||||||
|
the cluster topology requested by user is verified for consistency.
|
||||||
|
|
||||||
|
Currently there are the following limitations in cluster topology for Vanilla
|
||||||
|
plugin:
|
||||||
|
|
||||||
|
For Vanilla Hadoop version 2.x.x:
|
||||||
|
|
||||||
|
+ Cluster must contain exactly one namenode
|
||||||
|
+ Cluster can contain at most one resourcemanager
|
||||||
|
+ Cluster can contain at most one secondary namenode
|
||||||
|
+ Cluster can contain at most one historyserver
|
||||||
|
+ Cluster can contain at most one oozie and this process is also required
|
||||||
|
for EDP
|
||||||
|
+ Cluster can't contain oozie without resourcemanager and without
|
||||||
|
historyserver
|
||||||
|
+ Cluster can't have nodemanager nodes if it doesn't have resourcemanager
|
||||||
|
+ Cluster can have at most one hiveserver node.
|
||||||
|
+ Cluster can have at most one spark history server and this process is also
|
||||||
|
required for Spark EDP (Spark is available since the Newton release).
|
||||||
Loading…
Reference in New Issue