doc: restructure the image building documentation
Main goal: consolidate the information about image
building under the same documentation page, and move
plugin-specific details inside plugin pages.
No plugin-specific information should live outside
those pages.
More details:
- move the detailed documentation about sahara-image-pack
from the contributor guide to the new dedicated page
in the user manual;
- remove the vanilla and cdh pages which describes building
images with sahara-image-create, and move the common
information to new sahara-image-create page
in the user manual;
- add the matrix of supported plugin versions and
supported building technology for each plugin inside
the respective <plugin>-plugin.rst;
- add the redirects for the removed pages (only for master
and rocky, where this change should be backported).
- remove few details not really needed (e.g. how to convert
to VMDK images, location of cloud-init packages, etc,
which do not really belong here).
Change-Id: I8398a7ad625276d8f11d743688ba71902a7e1adc
(cherry picked from commit 4a44010f31
)
This commit is contained in:
parent
1dccb84d3c
commit
ddb518fd81
@ -1,2 +1,6 @@
|
||||
# renamed after the switch to Storyboard
|
||||
redirectmatch 301 ^/sahara/([^/]+)/contributor/launchpad.html$ /sahara/$1/contributor/project.html
|
||||
# renamed after some documentation reshuffling
|
||||
redirectmatch 301 ^/sahara/(?!ocata|pike|queens)([^/]+)/user/vanilla-imagebuilder.html$ /sahara/$1/user/vanilla-plugin.html
|
||||
redirectmatch 301 ^/sahara/(?!ocata|pike|queens)([^/]+)/user/cdh-imagebuilder.html$ /sahara/$1/user/cdh-plugin.html
|
||||
redirectmatch 301 ^/sahara/(?!ocata|pike|queens)([^/]+)/user/guest-requirements.html$ /sahara/$1/user/building-guest-images.html
|
||||
|
@ -512,7 +512,7 @@ sahara have the latest version of the Hadoop swift filesystem plugin
|
||||
installed. The sources for this plugin can be found in the
|
||||
`sahara extra repository`_. For more information on images or swift
|
||||
integration see the sahara documentation sections
|
||||
:ref:`diskimage-builder-label` and :ref:`swift-integration-label`.
|
||||
:ref:`building-guest-images-label` and :ref:`swift-integration-label`.
|
||||
|
||||
.. _Sahara extra repository: http://git.openstack.org/cgit/openstack/sahara-extra
|
||||
|
||||
|
@ -24,82 +24,8 @@ The key user-facing interface to this feature is the CLI script
|
||||
``sahara-image-pack``. This script will be installed with all other Sahara
|
||||
binaries.
|
||||
|
||||
Interface
|
||||
~~~~~~~~~
|
||||
|
||||
The script ``sahara-image-pack`` takes the following primary arguments:
|
||||
|
||||
::
|
||||
|
||||
--config-file PATH Path to a config file to use. Multiple config files
|
||||
can be specified, with values in later files taking
|
||||
precedence. Defaults to None.
|
||||
--image IMAGE The path to an image to modify. This image will be
|
||||
modified in-place: be sure to target a copy if you
|
||||
wish to maintain a clean master image.
|
||||
--root-filesystem ROOT_FS
|
||||
The filesystem to mount as the root volume on the
|
||||
image. Novalue is required if only one filesystem is
|
||||
detected.
|
||||
--test-only If this flag is set, no changes will be made to the
|
||||
image; instead, the script will fail if discrepancies
|
||||
are found between the image and the intended state.
|
||||
|
||||
After these arguments, the script takes ``PLUGIN`` and ``VERSION`` arguments.
|
||||
These arguments will allow any plugin and version combination which supports
|
||||
the image packing feature. Plugins may require their own arguments at specific
|
||||
versions; use the ``--help`` feature with ``PLUGIN`` and ``VERSION`` to see
|
||||
the appropriate argument structure.
|
||||
|
||||
|
||||
a plausible command-line invocation would be:
|
||||
|
||||
::
|
||||
|
||||
sahara-image-pack --image CentOS.qcow2 \
|
||||
--config-file etc/sahara/sahara.conf \
|
||||
cdh 5.7.0 [cdh 5.7.0 specific arguments, if any]
|
||||
|
||||
This script will modify the target image in-place. Please copy your image
|
||||
if you want a backup or if you wish to create multiple images from a single
|
||||
base image.
|
||||
|
||||
This CLI will automatically populate the set of available plugins and
|
||||
versions from the plugin set loaded in Sahara, and will show any plugin for
|
||||
which the image packing feature is available. The next sections of this guide
|
||||
will first describe how to modify an image packing specification for one
|
||||
of the plugins, and second, how to enable the image packing feature for new
|
||||
or existing plugins.
|
||||
|
||||
Note: In case of a RHEL 7 images, it is necessary to register the image before
|
||||
starting to pack it, also enable some required repos.
|
||||
|
||||
::
|
||||
|
||||
virt-customize -v -a $SAHARA_RHEL_IMAGE --sm-register \
|
||||
--sm-credentials ${REG_USER}:password:${REG_PASSWORD} --sm-attach \
|
||||
pool:${REG_POOL_ID} --run-command 'subscription-manager repos \
|
||||
--disable=* --enable=$REPO_A \ --enable=$REPO_B \ --enable=$REPO_C'
|
||||
|
||||
Dev notes on the CLI itself
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The script depends on a python library which is not packaged
|
||||
in pip, but is available through yum, dnf, and apt. If you have installed
|
||||
Sahara through yum, dnf, or apt, you should have appropriate dependencies,
|
||||
but if you wish to use the script but are working with Sahara from source,
|
||||
run whichever of the following is appropriate to your OS:
|
||||
|
||||
::
|
||||
|
||||
sudo yum install libguestfs python-libguestfs libguestfs-tools
|
||||
sudo dnf install libguestfs python-libguestfs libguestfs-tools
|
||||
sudo apt-get install libguestfs python-guestfs libguestfs-tools
|
||||
|
||||
If you are using tox to create virtual environments for your Sahara work,
|
||||
please use the ``images`` environment to run sahara-image-pack. This
|
||||
environment is configured to use system site packages, and will thus
|
||||
be able to find its dependency on python-libguestfs.
|
||||
The usage of the CLI script ``sahara-image-pack`` is documented in
|
||||
the :ref:`sahara-image-pack-label` section of the user guide.
|
||||
|
||||
|
||||
The Image Manifest
|
||||
|
@ -23,10 +23,59 @@ variety of environments.
|
||||
|
||||
Images
|
||||
------
|
||||
The sahara Ambari plugin is using minimal (operating system only) images.
|
||||
|
||||
For more information about Ambari images, refer to
|
||||
`<https://git.openstack.org/cgi/openstack/sahara-image-elements>`_.
|
||||
For cluster provisioning, prepared images should be used.
|
||||
|
||||
.. list-table:: Support matrix for the `ambari` plugin
|
||||
:widths: 15 15 20 15 35
|
||||
:header-rows: 1
|
||||
|
||||
* - Version
|
||||
(image tag)
|
||||
- Distribution
|
||||
- Build method
|
||||
- Version
|
||||
(build parameter)
|
||||
- Notes
|
||||
|
||||
* - 2.6
|
||||
- Ubuntu 16.04, CentOS 7
|
||||
- sahara-image-pack
|
||||
- 2.6
|
||||
- uses Ambari 2.6
|
||||
|
||||
* - 2.5
|
||||
- Ubuntu 16.04, CentOS 7
|
||||
- sahara-image-pack
|
||||
- 2.5
|
||||
- uses Ambari 2.6
|
||||
|
||||
* - 2.4
|
||||
- Ubuntu 14.04, CentOS 7
|
||||
- sahara-image-pack
|
||||
- 2.4
|
||||
- uses Ambari 2.6
|
||||
|
||||
* - 2.4
|
||||
- Ubuntu 14.04, CentOS 7
|
||||
- sahara-image-create
|
||||
- 2.4
|
||||
- uses Ambari 2.2.1.0
|
||||
|
||||
* - 2.3
|
||||
- Ubuntu 14.04, CentOS 7
|
||||
- sahara-image-pack
|
||||
- 2.3
|
||||
- uses Ambari 2.4
|
||||
|
||||
* - 2.3
|
||||
- Ubuntu 14.04, CentOS 7
|
||||
- sahara-image-create
|
||||
- 2.3
|
||||
- uses Ambari 2.2.0.0
|
||||
|
||||
For more information about building image, refer to
|
||||
:doc:`building-guest-images`.
|
||||
|
||||
HDP plugin requires an image to be tagged in sahara Image Registry with two
|
||||
tags: 'ambari' and '<plugin version>' (e.g. '2.5').
|
||||
@ -34,6 +83,9 @@ tags: 'ambari' and '<plugin version>' (e.g. '2.5').
|
||||
The image requires a username. For more information, refer to the
|
||||
:doc:`registering-image` section.
|
||||
|
||||
To speed up provisioning, the HDP packages can be pre-installed on the image
|
||||
used. The packages' versions depend on the HDP version required.
|
||||
|
||||
High Availability for HDFS and YARN
|
||||
-----------------------------------
|
||||
High Availability (Using the Quorum Journal Manager) can be
|
||||
|
49
doc/source/user/building-guest-images.rst
Normal file
49
doc/source/user/building-guest-images.rst
Normal file
@ -0,0 +1,49 @@
|
||||
.. _building-guest-images-label:
|
||||
|
||||
Building guest images
|
||||
=====================
|
||||
|
||||
Sahara plugins represent different Hadoop or other Big Data platforms
|
||||
and requires specific guest images.
|
||||
|
||||
While it is possible to use cloud images which only contain the basic
|
||||
software requirements (also called *plain images*), their usage slows down
|
||||
the cluster provisioning process and was not throughly tested recently.
|
||||
|
||||
It is strongly advised to build images which contain
|
||||
the software required to create the clusters for the various plugins
|
||||
and use them instead of *plain images*.
|
||||
|
||||
Sahara currently provides two different tools for building
|
||||
guest images:
|
||||
- ``sahara-image-pack`` is newer and support more recent images;
|
||||
- ``sahara-image-create`` is the older tool.
|
||||
|
||||
Both tools are described in the details in the next sections.
|
||||
|
||||
The documentation of each plugin describes which method is supported
|
||||
for the various versions. If both are supported, ``sahara-image-pack``
|
||||
is recommended.
|
||||
|
||||
General requirements for guest images
|
||||
-------------------------------------
|
||||
|
||||
There are few common requirements for all guest images,
|
||||
which must be based on GNU/Linux distributions.
|
||||
|
||||
* cloud-init must be installed
|
||||
* the ssh server must be installed
|
||||
* the firewall, if enabled, must allow connections on port 22 (ssh)
|
||||
|
||||
The cloud images provided by the GNU/Linux distributions respect
|
||||
those requirements.
|
||||
|
||||
Each plugin specifies additional requirements.
|
||||
The image building tools provided by Sahara take care of preparing the images
|
||||
with those additional requirements.
|
||||
|
||||
.. toctree::
|
||||
|
||||
building-guest-images/sahara-image-pack
|
||||
building-guest-images/sahara-image-create
|
||||
building-guest-images/baremetal
|
14
doc/source/user/building-guest-images/baremetal.rst
Normal file
14
doc/source/user/building-guest-images/baremetal.rst
Normal file
@ -0,0 +1,14 @@
|
||||
.. _building-baremetal-images-label:
|
||||
|
||||
Bare metal images
|
||||
-----------------
|
||||
|
||||
Images that can be used for bare metal deployment through Ironic
|
||||
can be generated using both image building tools:
|
||||
|
||||
sahara-image-create:
|
||||
pass the -b parameters to the command
|
||||
|
||||
sahara-image-pack:
|
||||
use `virt-get-kernel` on the generated image to extract the kernel and
|
||||
the initramfs file
|
@ -0,0 +1,80 @@
|
||||
sahara-image-create
|
||||
-------------------
|
||||
|
||||
The historical tool for building images, ``sahara-image-create``, is based on
|
||||
`Disk Image Builder <https://git.openstack.org/cgit/openstack/diskimage-builder>`_.
|
||||
|
||||
`Disk Image Builder` builds disk images using elements. An element is a
|
||||
particular set of code that alters how the image is built, or runs within the
|
||||
chroot to prepare the image.
|
||||
|
||||
The additional elements required by Sahara images and the ``sahara-image-create``
|
||||
command itself are stored in the
|
||||
`Sahara image elements repository <https://git.openstack.org/cgit/openstack/sahara-image-elements>`_
|
||||
|
||||
To create images for a specific plugin follow these steps:
|
||||
|
||||
1. Clone repository "https://git.openstack.org/cgit/openstack/sahara-image-elements"
|
||||
locally.
|
||||
|
||||
2. Use tox to build images.
|
||||
|
||||
You can run the command below in sahara-image-elements
|
||||
directory to build images. By default this script will attempt to create
|
||||
cloud images for all versions of supported plugins and all operating systems
|
||||
(subset of Ubuntu, Fedora, and CentOS depending on plugin).
|
||||
|
||||
.. sourcecode:: console
|
||||
|
||||
tox -e venv -- sahara-image-create -u
|
||||
|
||||
If you want to build a image for ``<plugin>`` with ``<version>`` on a specific
|
||||
``<distribution>`` just execute:
|
||||
|
||||
.. sourcecode:: console
|
||||
|
||||
tox -e venv -- sahara-image-create -p <plugin> -v <version> -i <distribution>
|
||||
|
||||
Tox will create a virtualenv and install required python packages in it,
|
||||
clone the repositories "https://git.openstack.org/cgit/openstack/diskimage-builder" and
|
||||
"https://git.openstack.org/cgit/openstack/sahara-image-elements" and export necessary
|
||||
parameters.
|
||||
|
||||
The valid values for the ``<distribution>`` argument are:
|
||||
|
||||
- Ubuntu (all versions): ``ubuntu``
|
||||
- CentOS 7: ``centos7``
|
||||
- Fedora: ``fedora``
|
||||
|
||||
``sahara-image-create`` will then create the required cloud images
|
||||
using image elements that install all the necessary packages
|
||||
and configure them.
|
||||
You will find created images in the parent directory.
|
||||
|
||||
Variables
|
||||
~~~~~~~~~
|
||||
|
||||
The following environment variables can be used to change the behavior of the
|
||||
image building:
|
||||
|
||||
* ``JAVA_DOWNLOAD_URL`` - download link for JDK (tarball or bin)
|
||||
* ``DIB_IMAGE_SIZE`` - parameter that specifies a volume of hard disk
|
||||
of instance. You need to specify it only for Fedora because Fedora
|
||||
doesn't use all available volume
|
||||
|
||||
The following variables can be used to change the name of the output
|
||||
image:
|
||||
|
||||
* ``centos7_image_name``
|
||||
* ``ubuntu_image_name``
|
||||
* ``fedora_image_name``
|
||||
|
||||
.. note::
|
||||
|
||||
Disk Image Builder will generate QCOW2 images, used with the default
|
||||
OpenStack Qemu/KVM hypervisors. If your OpenStack uses a different
|
||||
hypervisor, the generated image should be converted to an appropriate
|
||||
format.
|
||||
|
||||
For finer control of ``sahara-image-create`` see the `official documentation
|
||||
<https://git.openstack.org/cgit/openstack/sahara-image-elements/tree/diskimage-create/README.rst>`_
|
90
doc/source/user/building-guest-images/sahara-image-pack.rst
Normal file
90
doc/source/user/building-guest-images/sahara-image-pack.rst
Normal file
@ -0,0 +1,90 @@
|
||||
.. _sahara-image-pack-label:
|
||||
|
||||
sahara-image-pack
|
||||
-----------------
|
||||
|
||||
The CLI command ``sahara-image-pack`` operates in-place on an existing image
|
||||
and installs and configures the software required for the plugin.
|
||||
|
||||
The script ``sahara-image-pack`` takes the following primary arguments:
|
||||
|
||||
::
|
||||
|
||||
--config-file PATH Path to a config file to use. Multiple config files
|
||||
can be specified, with values in later files taking
|
||||
precedence. Defaults to None.
|
||||
--image IMAGE The path to an image to modify. This image will be
|
||||
modified in-place: be sure to target a copy if you
|
||||
wish to maintain a clean master image.
|
||||
--root-filesystem ROOT_FS
|
||||
The filesystem to mount as the root volume on the
|
||||
image. Novalue is required if only one filesystem is
|
||||
detected.
|
||||
--test-only If this flag is set, no changes will be made to the
|
||||
image; instead, the script will fail if discrepancies
|
||||
are found between the image and the intended state.
|
||||
|
||||
After these arguments, the script takes ``PLUGIN`` and ``VERSION`` arguments.
|
||||
These arguments will allow any plugin and version combination which supports
|
||||
the image packing feature. Plugins may require their own arguments at specific
|
||||
versions; use the ``--help`` feature with ``PLUGIN`` and ``VERSION`` to see
|
||||
the appropriate argument structure.
|
||||
|
||||
|
||||
a plausible command-line invocation would be:
|
||||
|
||||
::
|
||||
|
||||
sahara-image-pack --image CentOS.qcow2 \
|
||||
--config-file etc/sahara/sahara.conf \
|
||||
cdh 5.7.0 [cdh 5.7.0 specific arguments, if any]
|
||||
|
||||
This script will modify the target image in-place. Please copy your image
|
||||
if you want a backup or if you wish to create multiple images from a single
|
||||
base image.
|
||||
|
||||
This CLI will automatically populate the set of available plugins and
|
||||
versions from the plugin set loaded in Sahara, and will show any plugin for
|
||||
which the image packing feature is available. The next sections of this guide
|
||||
will first describe how to modify an image packing specification for one
|
||||
of the plugins, and second, how to enable the image packing feature for new
|
||||
or existing plugins.
|
||||
|
||||
Note: In case of a RHEL 7 images, it is necessary to register the image before
|
||||
starting to pack it, also enable some required repos.
|
||||
|
||||
::
|
||||
|
||||
virt-customize -v -a $SAHARA_RHEL_IMAGE --sm-register \
|
||||
--sm-credentials ${REG_USER}:password:${REG_PASSWORD} --sm-attach \
|
||||
pool:${REG_POOL_ID} --run-command 'subscription-manager repos \
|
||||
--disable=* --enable=$REPO_A \ --enable=$REPO_B \ --enable=$REPO_C'
|
||||
|
||||
Installation and developer notes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The script is part of of the Sahara repository, but it does not depend
|
||||
on the Sahara services. In order to use its development version,
|
||||
clone the `Sahara repository <https://git.openstack.org/openstack/sahara>`_,
|
||||
check out the branch which matches the Sahara version used, and
|
||||
install the repository in a virtualenv.
|
||||
|
||||
The script is also provided by binary distributions of OpenStack.
|
||||
For example, RDO ships it in the ``openstack-sahara-image-pack`` package.
|
||||
|
||||
The script depends on a python library which is not packaged
|
||||
in pip, but is available through yum, dnf, and apt. If you have installed
|
||||
Sahara through yum, dnf, or apt, you should have appropriate dependencies,
|
||||
but if you wish to use the script but are working with Sahara from source,
|
||||
run whichever of the following is appropriate to your OS:
|
||||
|
||||
::
|
||||
|
||||
sudo yum install libguestfs python-libguestfs libguestfs-tools
|
||||
sudo dnf install libguestfs python-libguestfs libguestfs-tools
|
||||
sudo apt-get install libguestfs python-guestfs libguestfs-tools
|
||||
|
||||
If you are using tox to create virtual environments for your Sahara work,
|
||||
please use the ``images`` environment to run sahara-image-pack. This
|
||||
environment is configured to use system site packages, and will thus
|
||||
be able to find its dependency on python-libguestfs.
|
@ -1,89 +0,0 @@
|
||||
.. _cdh-diskimage-builder-label:
|
||||
|
||||
Building Images for Cloudera Plugin
|
||||
===================================
|
||||
|
||||
In this document you will find instructions on how to build Ubuntu and CentOS
|
||||
images with Cloudera Express (now only versions {5.0.0, 5.3.0 5.4.0, 5.5.0,
|
||||
5.7.x, 5.9.x} are supported).
|
||||
|
||||
To simplify the task of building such images we use
|
||||
`Disk Image Builder <https://git.openstack.org/cgit/openstack/diskimage-builder>`_.
|
||||
|
||||
`Disk Image Builder` builds disk images using elements. An element is a
|
||||
particular set of code that alters how the image is built, or runs within the
|
||||
chroot to prepare the image.
|
||||
|
||||
Elements for building Cloudera images are stored in
|
||||
`Sahara extra repository <https://git.openstack.org/cgit/openstack/sahara-image-elements>`_
|
||||
|
||||
.. note::
|
||||
|
||||
Sahara requires images with cloud-init package installed:
|
||||
|
||||
* `For CentOS 7 <http://mirror.centos.org/centos/7/os/x86_64/Packages/cloud-init-0.7.9-24.el7.centos.x86_64.rpm>`_
|
||||
* `For Ubuntu 14.04 <http://packages.ubuntu.com/trusty/cloud-init>`_
|
||||
* `For Ubuntu 16.04 <http://packages.ubuntu.com/xenial/cloud-init>`_
|
||||
|
||||
To create cloudera images follow these steps:
|
||||
|
||||
1. Clone repository "https://git.openstack.org/cgit/openstack/sahara-image-elements" locally.
|
||||
|
||||
2. Use tox to build images.
|
||||
|
||||
You can run "tox -e venv -- sahara-image-create" command in
|
||||
sahara-image-elements directory to build images. By default this script will
|
||||
attempt to create cloud images for all versions of supported plugins and all
|
||||
operating systems (subset of Ubuntu, Fedora, and CentOS depending on
|
||||
plugin). To only create Cloudera images, you should use the "-p cloudera"
|
||||
parameter in the command line. If you want to create the image only for a
|
||||
specific operating system, you should use the "-i ubuntu|centos|centos7"
|
||||
parameter to assign the operating system (the cloudera plugin only supports
|
||||
Ubuntu and Centos). If you want to create the image only for a specific
|
||||
Cloudera version, you should use the "-v 5.0|5.3|5.4|5.5|5.7|5.9" parameter
|
||||
to assign the version. Note that Centos 7 can only be used with CDH 5.5 and
|
||||
higher. Below is an example to create Cloudera images for both Ubuntu and
|
||||
CentOS with Cloudera Express 5.5.0 version.
|
||||
|
||||
.. sourcecode:: console
|
||||
|
||||
tox -e venv -- sahara-image-create -p cloudera -v 5.5
|
||||
|
||||
If you want to create only an Ubuntu image, you may use following example
|
||||
for that:
|
||||
|
||||
.. sourcecode:: console
|
||||
|
||||
tox -e venv -- sahara-image-create -p cloudera -i ubuntu -v 5.5
|
||||
|
||||
For CDH 5.7 and higher we support minor versions. If you want to build a minor
|
||||
version just export DIB_CDH_MINOR_VERSION before sahara-image-create launch, e.g.:
|
||||
|
||||
.. sourcecode:: console
|
||||
|
||||
export DIB_CDH_MINOR_VERSION=5.7.1
|
||||
|
||||
NOTE: If you don't want to use default values, you should explicitly set the
|
||||
values of your required parameters.
|
||||
|
||||
The script will create required cloud images using image elements that
|
||||
install all the necessary packages and configure them. You will find the
|
||||
created images in the parent directory.
|
||||
|
||||
.. note::
|
||||
|
||||
Disk Image Builder will generate QCOW2 images, used with the default
|
||||
OpenStack Qemu/KVM hypervisors. If your OpenStack uses a different
|
||||
hypervisor, the generated image should be converted to an appropriate
|
||||
format.
|
||||
|
||||
The VMware Nova backend requires the VMDK image format. You may use
|
||||
qemu-img utility to convert a QCOW2 image to VMDK.
|
||||
|
||||
.. sourcecode:: console
|
||||
|
||||
qemu-img convert -O vmdk <original_image>.qcow2 <converted_image>.vmdk
|
||||
|
||||
|
||||
For finer control of diskimage-create.sh see the `official documentation
|
||||
<https://git.openstack.org/cgit/openstack/sahara-image-elements/tree/diskimage-create/README.rst>`_
|
@ -8,17 +8,70 @@ The Cloudera plugin is enabled in Sahara by default. You can manually
|
||||
modify the Sahara configuration file (default /etc/sahara/sahara.conf) to
|
||||
explicitly enable or disable it in "plugins" line.
|
||||
|
||||
You need to build images using :doc:`cdh-imagebuilder` to produce images used
|
||||
to provision cluster.
|
||||
Images
|
||||
------
|
||||
|
||||
For cluster provisioning, prepared images should be used.
|
||||
|
||||
.. list-table:: Support matrix for the `vanilla` plugin
|
||||
:widths: 15 15 20 15 35
|
||||
:header-rows: 1
|
||||
|
||||
* - Version
|
||||
(image tag)
|
||||
- Distribution
|
||||
- Build method
|
||||
- Version
|
||||
(build parameter)
|
||||
- Notes
|
||||
|
||||
* - 5.13.0
|
||||
- Ubuntu 16.04, CentOS 7
|
||||
- sahara-image-pack
|
||||
- 5.13.0
|
||||
-
|
||||
|
||||
* - 5.11.0
|
||||
- Ubuntu 16.04, CentOS 7
|
||||
- sahara-image-pack, sahara-image-create
|
||||
- 5.11.0
|
||||
-
|
||||
|
||||
* - 5.9.0
|
||||
- Ubuntu 14.04, CentOS 7
|
||||
- sahara-image-pack, sahara-image-create
|
||||
- 5.9.0
|
||||
-
|
||||
|
||||
* - 5.7.0
|
||||
- Ubuntu 14.04, CentOS 7
|
||||
- sahara-image-pack, sahara-image-create
|
||||
- 5.7.0
|
||||
-
|
||||
|
||||
For more information about building image, refer to
|
||||
:doc:`building-guest-images`.
|
||||
|
||||
The cloudera plugin requires an image to be tagged in Sahara Image Registry
|
||||
with two tags: 'cdh' and '<cloudera version>' (e.g. '5', '5.3.0', '5.4.0',
|
||||
'5.5.0', '5.7.0', '5.9.0' or '5.9.1', here '5' stands for '5.0.0').
|
||||
with two tags: 'cdh' and '<cloudera version>' (e.g. '5.13.0', '5.11.0',
|
||||
'5.9.0', etc).
|
||||
|
||||
The default username specified for these images is different for each
|
||||
distribution. For more information, refer to the
|
||||
:doc:`registering-image` section.
|
||||
|
||||
Build settings
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
It is possible to specify minor versions of CDH when ``sahara-image-create``
|
||||
is used.
|
||||
If you want to use a minor versions, export ``DIB_CDH_MINOR_VERSION``
|
||||
before starting the build command, e.g.:
|
||||
|
||||
.. sourcecode:: console
|
||||
|
||||
export DIB_CDH_MINOR_VERSION=5.7.1
|
||||
|
||||
Services Supported
|
||||
------------------
|
||||
|
||||
|
@ -1,57 +0,0 @@
|
||||
Requirements for Guests
|
||||
=======================
|
||||
|
||||
Sahara manages guests of various platforms (for example Ubuntu, Fedora, RHEL,
|
||||
and CentOS) with various versions of the Hadoop ecosystem projects installed.
|
||||
There are common requirements for all guests, and additional requirements based
|
||||
on the plugin that is used for cluster deployment.
|
||||
|
||||
Common Requirements
|
||||
-------------------
|
||||
|
||||
* The operating system must be Linux
|
||||
* cloud-init must be installed
|
||||
* ssh-server must be installed
|
||||
|
||||
+ if a firewall is active it must allow connections on port 22 to enable ssh
|
||||
|
||||
Vanilla Plugin Requirements
|
||||
---------------------------
|
||||
|
||||
If the Vanilla Plugin is used for cluster deployment the guest is required to
|
||||
have
|
||||
|
||||
* ssh-client installed
|
||||
* Java (version >= 6)
|
||||
* Apache Hadoop installed
|
||||
* 'hadoop' user created
|
||||
|
||||
See :doc:`hadoop-swift` for information on using Swift with your sahara cluster
|
||||
(for EDP support Swift integration is currently required).
|
||||
|
||||
To support EDP, the following components must also be installed on the guest:
|
||||
|
||||
* Oozie version 4 or higher
|
||||
* mysql
|
||||
* hive
|
||||
|
||||
See :doc:`vanilla-imagebuilder` for instructions on building images for this
|
||||
plugin.
|
||||
|
||||
Hortonworks Plugin Requirements
|
||||
-------------------------------
|
||||
|
||||
This plugin does not have any additional requirements. Currently, only the
|
||||
CentOS Linux and Ubuntu distributions are supported but other distributions
|
||||
will be supported in the future.
|
||||
To speed up provisioning, the HDP packages can be pre-installed on the image
|
||||
used. The packages' versions depend on the HDP version being used.
|
||||
|
||||
Cloudera Plugin Requirements
|
||||
----------------------------
|
||||
|
||||
Cloudera Plugin does not have any additional requirements, just build a CDH
|
||||
image to deploy the cluster.
|
||||
|
||||
See :doc:`cdh-imagebuilder` for instructions on building images for this
|
||||
plugin.
|
@ -48,7 +48,5 @@ Guest Images
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
guest-requirements
|
||||
building-guest-images
|
||||
hadoop-swift
|
||||
vanilla-imagebuilder
|
||||
cdh-imagebuilder
|
||||
|
@ -30,12 +30,33 @@ spent in the provisioning cycle. In addition, provisioning large clusters will
|
||||
put a burden on the network as packages for all nodes need to be downloaded
|
||||
from the package repository.
|
||||
|
||||
For more information about MapR images, refer to
|
||||
`<https://git.openstack.org/cgit/openstack/sahara-image-elements>`_.
|
||||
|
||||
There are VM images provided for use with the MapR Plugin, that can also be
|
||||
built using the tools available in sahara-image-elements:
|
||||
https://s3-us-west-2.amazonaws.com/sahara-images/index.html
|
||||
.. list-table:: Support matrix for the `mapr` plugin
|
||||
:widths: 15 15 20 15 35
|
||||
:header-rows: 1
|
||||
|
||||
* - Version
|
||||
(image tag)
|
||||
- Distribution
|
||||
- Build method
|
||||
- Version
|
||||
(build parameter)
|
||||
- Notes
|
||||
|
||||
* - 5.2.0.mrv2
|
||||
- Ubuntu 14.04, CentOS 7
|
||||
- sahara-image-pack
|
||||
- 5.2.0.mrv2
|
||||
-
|
||||
|
||||
* - 5.2.0.mrv2
|
||||
- Ubuntu 14.04, CentOS 7
|
||||
- sahara-image-create
|
||||
- 5.2.0
|
||||
-
|
||||
|
||||
For more information about building image, refer to
|
||||
:doc:`building-guest-images`.
|
||||
|
||||
MapR plugin needs an image to be tagged in Sahara Image Registry with
|
||||
two tags: 'mapr' and '<MapR version>' (e.g. '5.2.0.mrv2').
|
||||
|
@ -44,11 +44,9 @@ generated Ubuntu vanilla image, referred to as
|
||||
``sahara-vanilla-latest-ubuntu.qcow2``,
|
||||
and the latest version of vanilla plugin as an example.
|
||||
|
||||
* Download a pre-built image
|
||||
|
||||
* Build the image using `sahara-image-elements
|
||||
<https://git.openstack.org/cgit/openstack/sahara-image-elements/tree/README.rst>`_
|
||||
|
||||
Build an image which works for the specific plugin.
|
||||
Please refer to :ref:`building-guest-images-label` and
|
||||
to the plugin-specific documentation.
|
||||
|
||||
Upload the generated image into the OpenStack Image service:
|
||||
|
||||
|
@ -19,8 +19,8 @@ from the bare metal performance with self-service resource provisioning.
|
||||
|
||||
4. Build the Sahara image and prepare it for uploading to Glance:
|
||||
|
||||
- Build an image for Sahara plugin with the ``-b`` flag. Use sahara image elements
|
||||
when building the image. See `<https://git.openstack.org/cgit/openstack/sahara-image-elements>`_
|
||||
- Build an image for Sahara plugin which supports baremetal deployment.
|
||||
Refer to the :ref:`building-baremetal-images-label` section.
|
||||
|
||||
- Convert the qcow2 image format to the raw format. For example:
|
||||
|
||||
|
@ -10,14 +10,34 @@ support.
|
||||
Images
|
||||
------
|
||||
|
||||
For cluster provisioning, prepared images should be used. The Spark plugin
|
||||
has been developed and tested with the images generated by
|
||||
sahara-image-elements:
|
||||
For cluster provisioning, prepared images should be used.
|
||||
|
||||
* https://git.openstack.org/cgit/openstack/sahara-image-elements
|
||||
.. list-table:: Support matrix for the `spark` plugin
|
||||
:widths: 15 15 20 15 35
|
||||
:header-rows: 1
|
||||
|
||||
The latest Ubuntu images generated by sahara-image-elements have Cloudera
|
||||
CDH 5.4.0 HDFS and Apache Spark installed.
|
||||
* - Version
|
||||
(image tag)
|
||||
- Distribution
|
||||
- Build method
|
||||
- Version
|
||||
(build parameter)
|
||||
- Notes
|
||||
|
||||
* - 2.3
|
||||
- Ubuntu 16.04
|
||||
- sahara-image-create
|
||||
- 2.3.0
|
||||
- based on CDH 5.11
|
||||
|
||||
* - 2.2
|
||||
- Ubuntu 16.04
|
||||
- sahara-image-create
|
||||
- 2.2.0
|
||||
- based on CDH 5.11
|
||||
|
||||
For more information about building image, refer to
|
||||
:doc:`building-guest-images`.
|
||||
|
||||
The Spark plugin requires an image to be tagged in the sahara image registry
|
||||
with two tags: 'spark' and '<Spark version>' (e.g. '1.6.0').
|
||||
@ -31,6 +51,15 @@ slaves), stop all services and so on. As such (and as opposed to CDH HDFS
|
||||
daemons), Spark is not deployed as a standard Ubuntu service and if the
|
||||
virtual machines are rebooted, Spark will not be restarted.
|
||||
|
||||
Build settings
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
When ``sahara-image-create`` is used, you can override few settings
|
||||
by exporting the corresponding environment variables
|
||||
before starting the build command:
|
||||
|
||||
* ``SPARK_DOWNLOAD_URL`` - download link for Spark
|
||||
|
||||
Spark configuration
|
||||
-------------------
|
||||
|
||||
|
@ -9,11 +9,34 @@ Currently Storm is installed in standalone mode, with no YARN support.
|
||||
Images
|
||||
------
|
||||
|
||||
For cluster provisioning, prepared images should be used. The Storm plugin
|
||||
has been developed and tested with the images generated by
|
||||
sahara-image-elements:
|
||||
For cluster provisioning, prepared images should be used.
|
||||
|
||||
* https://git.openstack.org/cgit/openstack/sahara-image-elements
|
||||
.. list-table:: Support matrix for the `storm` plugin
|
||||
:widths: 15 15 20 15 35
|
||||
:header-rows: 1
|
||||
|
||||
* - Version
|
||||
(image tag)
|
||||
- Distribution
|
||||
- Build method
|
||||
- Version
|
||||
(build parameter)
|
||||
- Notes
|
||||
|
||||
* - 1.2
|
||||
- Ubuntu 16.04
|
||||
- sahara-image-create
|
||||
- 1.2.1, 1.2.0
|
||||
- both versions are supported by the same image tag
|
||||
|
||||
* - 1.1.0
|
||||
- Ubuntu 16.04
|
||||
- sahara-image-create
|
||||
- 1.1.1, 1.1.0
|
||||
- both versions are supported by the same image tag
|
||||
|
||||
For more information about building image, refer to
|
||||
:doc:`building-guest-images`.
|
||||
|
||||
The Storm plugin requires an image to be tagged in the sahara image registry
|
||||
with two tags: 'storm' and '<Storm version>' (e.g. '1.1.0').
|
||||
|
@ -1,94 +0,0 @@
|
||||
.. _diskimage-builder-label:
|
||||
|
||||
Building Images for Vanilla Plugin
|
||||
==================================
|
||||
|
||||
In this document you will find instruction on how to build Ubuntu, Fedora, and
|
||||
CentOS images with Apache Hadoop version 2.x.x.
|
||||
|
||||
As of now the vanilla plugin works with images with pre-installed versions of
|
||||
Apache Hadoop. To simplify the task of building such images we use
|
||||
`Disk Image Builder <https://git.openstack.org/cgit/openstack/diskimage-builder>`_.
|
||||
|
||||
`Disk Image Builder` builds disk images using elements. An element is a
|
||||
particular set of code that alters how the image is built, or runs within the
|
||||
chroot to prepare the image.
|
||||
|
||||
Elements for building vanilla images are stored in the
|
||||
`Sahara image elements repository <https://git.openstack.org/cgit/openstack/sahara-image-elements>`_
|
||||
|
||||
.. note::
|
||||
|
||||
Sahara requires images with cloud-init package installed:
|
||||
|
||||
* `For Ubuntu 16.04 <http://packages.ubuntu.com/xenial/cloud-init>`_
|
||||
* `For CentOS 7 <http://mirror.centos.org/centos/7/os/x86_64/Packages/cloud-init-0.7.9-24.el7.centos.x86_64.rpm>`_
|
||||
* `For Fedora <https://src.fedoraproject.org/rpms/cloud-init>`_
|
||||
|
||||
To create vanilla images follow these steps:
|
||||
|
||||
1. Clone repository "https://git.openstack.org/cgit/openstack/sahara-image-elements"
|
||||
locally.
|
||||
|
||||
2. Use tox to build images.
|
||||
|
||||
You can run the command below in sahara-image-elements
|
||||
directory to build images. By default this script will attempt to create
|
||||
cloud images for all versions of supported plugins and all operating systems
|
||||
(subset of Ubuntu, Fedora, and CentOS depending on plugin).
|
||||
|
||||
.. sourcecode:: console
|
||||
|
||||
tox -e venv -- sahara-image-create -u
|
||||
|
||||
If you want to build Vanilla 2.7.1 image with centos 7 just execute:
|
||||
|
||||
.. sourcecode:: console
|
||||
|
||||
tox -e venv -- sahara-image-create -p vanilla -v 2.7.1 -i centos7
|
||||
|
||||
Tox will create a virtualenv and install required python packages in it,
|
||||
clone the repositories "https://git.openstack.org/cgit/openstack/diskimage-builder" and
|
||||
"https://git.openstack.org/cgit/openstack/sahara-image-elements" and export necessary
|
||||
parameters.
|
||||
|
||||
* ``DIB_HADOOP_VERSION`` - version of Hadoop to install
|
||||
* ``JAVA_DOWNLOAD_URL`` - download link for JDK (tarball or bin)
|
||||
* ``OOZIE_DOWNLOAD_URL`` - download link for OOZIE (we have built
|
||||
Oozie libs here: ``https://tarballs.openstack.org/sahara-extra/dist/oozie/oozie-4.2.0-hadoop-2.7.1.tar.gz``)
|
||||
* ``SPARK_DOWNLOAD_URL`` - download link for Spark
|
||||
* ``HIVE_VERSION`` - version of Hive to install
|
||||
(currently supports only 0.11.0)
|
||||
* ``ubuntu_image_name``
|
||||
* ``fedora_image_name``
|
||||
* ``DIB_IMAGE_SIZE`` - parameter that specifies a volume of hard disk
|
||||
of instance. You need to specify it only for Fedora because Fedora
|
||||
doesn't use all available volume
|
||||
* ``DIB_COMMIT_ID`` - latest commit id of diskimage-builder project
|
||||
* ``SAHARA_ELEMENTS_COMMIT_ID`` - latest commit id of
|
||||
sahara-image-elements project
|
||||
|
||||
NOTE: If you don't want to use default values, you should set your values
|
||||
of parameters.
|
||||
|
||||
Then it will create required cloud images using image elements that install
|
||||
all the necessary packages and configure them. You will find created images
|
||||
in the parent directory.
|
||||
|
||||
.. note::
|
||||
|
||||
Disk Image Builder will generate QCOW2 images, used with the default
|
||||
OpenStack Qemu/KVM hypervisors. If your OpenStack uses a different
|
||||
hypervisor, the generated image should be converted to an appropriate
|
||||
format.
|
||||
|
||||
VMware Nova backend requires VMDK image format. You may use qemu-img
|
||||
utility to convert a QCOW2 image to VMDK.
|
||||
|
||||
.. sourcecode:: console
|
||||
|
||||
qemu-img convert -O vmdk <original_image>.qcow2 <converted_image>.vmdk
|
||||
|
||||
|
||||
For finer control of diskimage-create.sh see the `official documentation
|
||||
<https://git.openstack.org/cgit/openstack/sahara-image-elements/tree/diskimage-create/README.rst>`_
|
@ -7,10 +7,43 @@ a cluster with Apache Hadoop.
|
||||
Since the Newton release Spark is integrated into the Vanilla plugin so you
|
||||
can launch Spark jobs on a Vanilla cluster.
|
||||
|
||||
For cluster provisioning prepared images should be used. They already have
|
||||
Apache Hadoop 2.7.1 installed.
|
||||
Images
|
||||
------
|
||||
|
||||
You may build images by yourself using :doc:`vanilla-imagebuilder`.
|
||||
For cluster provisioning, prepared images should be used.
|
||||
|
||||
.. list-table:: Support matrix for the `vanilla` plugin
|
||||
:widths: 15 15 20 15 35
|
||||
:header-rows: 1
|
||||
|
||||
* - Version
|
||||
(image tag)
|
||||
- Distribution
|
||||
- Build method
|
||||
- Version
|
||||
(build parameter)
|
||||
- Notes
|
||||
|
||||
* - 2.8.2
|
||||
- Ubuntu 16.04, CentOS 7
|
||||
- sahara-image-create
|
||||
- 2.8.2
|
||||
- Hive 2.3.2, Oozie 4.3.0
|
||||
|
||||
* - 2.7.5
|
||||
- Ubuntu 16.04, CentOS 7
|
||||
- sahara-image-create
|
||||
- 2.7.5
|
||||
- Hive 2.3.2, Oozie 4.3.0
|
||||
|
||||
* - 2.7.1
|
||||
- Ubuntu 16.04, CentOS 7
|
||||
- sahara-image-create
|
||||
- 2.7.1
|
||||
- Hive 0.11.0, Oozie 4.2.0
|
||||
|
||||
For more information about building image, refer to
|
||||
:doc:`building-guest-images`.
|
||||
|
||||
Vanilla plugin requires an image to be tagged in Sahara Image Registry with
|
||||
two tags: 'vanilla' and '<hadoop version>' (e.g. '2.7.1').
|
||||
@ -18,6 +51,41 @@ two tags: 'vanilla' and '<hadoop version>' (e.g. '2.7.1').
|
||||
The image requires a username. For more information, refer to the
|
||||
:doc:`registering-image` section.
|
||||
|
||||
Build settings
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
When ``sahara-image-create`` is used, you can override few settings
|
||||
by exporting the corresponding environment variables
|
||||
before starting the build command:
|
||||
|
||||
* ``DIB_HADOOP_VERSION`` - version of Hadoop to install
|
||||
* ``HIVE_VERSION`` - version of Hive to install
|
||||
* ``OOZIE_DOWNLOAD_URL`` - download link for Oozie (we have built
|
||||
Oozie libs here: https://tarballs.openstack.org/sahara-extra/dist/oozie/)
|
||||
* ``SPARK_DOWNLOAD_URL`` - download link for Spark
|
||||
|
||||
Vanilla Plugin Requirements
|
||||
---------------------------
|
||||
|
||||
The image building tools described in :ref:`building-guest-images-label`
|
||||
add the required software to the image and their usage is strongly suggested.
|
||||
Nevertheless, here are listed the software that should be pre-loaded
|
||||
on the guest image so that it can be used to create Vanilla clusters:
|
||||
|
||||
* ssh-client installed
|
||||
* Java (version >= 7)
|
||||
* Apache Hadoop installed
|
||||
* 'hadoop' user created
|
||||
|
||||
See :doc:`hadoop-swift` for information on using Swift with your sahara cluster
|
||||
(for EDP support Swift integration is currently required).
|
||||
|
||||
To support EDP, the following components must also be installed on the guest:
|
||||
|
||||
* Oozie version 4 or higher
|
||||
* mysql/mariadb
|
||||
* hive
|
||||
|
||||
Cluster Validation
|
||||
------------------
|
||||
|
||||
|
@ -1,3 +1,7 @@
|
||||
/sahara/pike/contributor/launchpad.html 301 /sahara/pike/contributor/project.html
|
||||
/sahara/queens/contributor/launchpad.html 301 /sahara/queens/contributor/project.html
|
||||
/sahara/latest/contributor/launchpad.html 301 /sahara/latest/contributor/project.html
|
||||
/sahara/latest/user/vanilla-imagebuilder.html 301 /sahara/latest/user/vanilla-plugin.html
|
||||
/sahara/latest/user/cdh-imagebuilder.html 301 /sahara/latest/user/cdh-plugin.html
|
||||
/sahara/latest/user/guest-requirements.html 301 /sahara/latest/user/building-guest-images.html
|
||||
/sahara/rocky/user/guest-requirements.html 301 /sahara/rocky/user/building-guest-images.html
|
||||
|
Loading…
Reference in New Issue
Block a user