openstack/sahara-specs
Code Issues Proposed changes

Retire Sahara: remove repo content

Browse Source 
Sahara project is retiring
- https://review.opendev.org/c/openstack/governance/+/919374

this commit remove the content of this project repo

Depends-On: https://review.opendev.org/c/openstack/project-config/+/919376
Change-Id: Idfba6aefa444d9dce49d357c5c0a73e44153f0e9
This commit is contained in:
Ghanshyam Mann 2024-05-10 17:28:55 -07:00
parent 2b7e44b7ec
commit 47f81c66ce
162 changed files with 8 additions and 23893 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
.gitignore vendored
View File

@ -1,11 +0,0 @@
AUTHORS
ChangeLog
build
.tox
.venv
*.egg*
*.swp
*.swo
*.pyc
.testrepository
.stestr

2
.stestr.conf
View File

@ -1,2 +0,0 @@
[DEFAULT]
test_path=tests

9
.zuul.yaml
View File

@ -1,9 +0,0 @@
- project:
templates:
- openstack-specs-jobs
check:
jobs:
- openstack-tox-py36
gate:
jobs:
- openstack-tox-py36

19
CONTRIBUTING.rst
View File

@ -1,19 +0,0 @@
The source repository for this project can be found at:
https://opendev.org/openstack/sahara-specs
Pull requests submitted through GitHub are not monitored.
To start contributing to OpenStack, follow the steps in the contribution guide
to set up and use Gerrit:
https://docs.openstack.org/contributors/code-and-documentation/quick-start.html
Bugs should be filed on Storyboard:
https://storyboard.openstack.org/#!/project/openstack/sahara-specs
For more specific information about contributing to this repository, see the
sahara contributor guide:
https://docs.openstack.org/sahara/latest/contributor/contributing.html

3
LICENSE
View File

@ -1,3 +0,0 @@
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode

89
README.rst
View File

@ -1,83 +1,10 @@
========================
Team and repository tags
========================
This project is no longer maintained.
.. image:: https://governance.openstack.org/tc/badges/sahara-specs.svg
:target: https://governance.openstack.org/tc/reference/tags/index.html
The contents of this repository are still available in the Git
source code management system. To see the contents of this
repository before it reached its end of life, please check out the
previous commit with "git checkout HEAD^1".
.. Change things from this point on
===============================
OpenStack Sahara Specifications
===============================
This git repository is used to hold approved design specifications for additions
to the Sahara project. Reviews of the specs are done in gerrit, using a similar
workflow to how we review and merge changes to the code itself.
The layout of this repository for Sahara specifications is::
specs/<release>/
The layout of this repository for Sahara Client is::
specs/saharaclient/
When a new version of Sahara Client is released the implemented blueprints
will be moved to a directory specific to that release::
specs/saharaclient/<release>/
The layout of this repository for Sahara Tests is::
specs/sahara-tests/
You can find an example spec in ``specs/template.rst``.
For specifications that have been reviewed and approved but have not been
implemented::
specs/backlog/
Specifications in this directory indicate the original author has either
become unavailable, or has indicated that they are not going to implement the
specification. The specifications found here are available as projects for
people looking to get involved with Sahara. If you are interested in
claiming a spec, start by posting a review for the specification that moves it
from this directory to the next active release. Please set yourself as the new
`primary assignee` and maintain the original author in the `other contributors`
list.
Specifications are proposed for a given release by adding them to the
``specs/<release>`` directory and posting it for review. Not all approved
blueprints will get fully implemented. The implementation status of a
blueprint for a given release can be found by looking at the tasks associated
to the corresponding story in Storyboard.
Incomplete specifications have to be re-proposed for every release. The review
may be quick, but even if something was previously approved, it should be
re-reviewed to make sure it still makes sense as written.
Prior to the Juno development cycle and for the Juno-1 development milestone,
this repository was not used for spec reviews. Reviews prior to Juno were
completed entirely through Launchpad blueprints::
https://blueprints.launchpad.net/sahara
Launchpad blueprints are no more used for tracking the
current status of blueprints. For historical information, see::
https://wiki.openstack.org/wiki/Blueprints
For more information about working with gerrit, see::
https://docs.openstack.org/infra/manual/developers.html#development-workflow
To validate that the specification is syntactically correct (i.e. get more
confidence in the Jenkins result), please execute the following command::
$ tox
After running ``tox``, the documentation will be available for viewing in HTML
format in the ``doc/build/`` directory.
For any further questions, please email
openstack-discuss@lists.openstack.org or join #openstack-dev on
OFTC.

3
doc/requirements.txt
View File

@ -1,3 +0,0 @@
openstackdocstheme>=2.2.1 # Apache-2.0
sphinx>=2.0.0,!=2.1.0 # BSD
yasfb>=0.8.0

266
doc/source/conf.py
View File

@ -1,266 +0,0 @@
# -*- coding: utf-8 -*-
#
# Tempest documentation build configuration file, created by
# sphinx-quickstart on Tue May 21 17:43:32 2013.
#
# This file is execfile()d with the current directory set to its containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import datetime
import sys
import os
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('.'))
# -- General configuration -----------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['redirect',
'sphinx.ext.todo',
'openstackdocstheme',
'yasfb',
]
# Feed configuration for yasfb
feed_base_url = 'https://specs.openstack.org/openstack/sahara-specs'
feed_author = 'OpenStack Sahara Team'
todo_include_todos = True
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = 'Sahara Specs'
copyright = '%s, OpenStack Sahara Team' % datetime.date.today().year
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = [
'_build',
'**/template.rst',
]
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
add_module_names = False
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'native'
# A list of ignored prefixes for module index sorting.
modindex_common_prefix = ['sahara-specs.']
# Neither version nor release number for specs
version = ''
release = ''
# -- Options for man page output ----------------------------------------------
man_pages = []
# -- Options for HTML output ---------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'openstackdocs'
openstackdocs_repo_name = 'openstack/sahara-specs'
openstackdocs_pdf_link = True
openstackdocs_auto_name = False
openstackdocs_use_storyboard = True
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
html_domain_indices = False
# If false, no index is generated.
html_use_index = False
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'Sahara-Specsdoc'
# -- Options for LaTeX output --------------------------------------------------
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass [howto/manual]).
latex_documents = [
('index', 'doc-sahara-specs.tex', 'Sahara Specs',
'Sahara Team', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# If true, show page references after internal links.
#latex_show_pagerefs = False
# If true, show URL addresses after external links.
#latex_show_urls = False
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_domain_indices = True
smartquotes_excludes = {'builders': ['latex']}
# -- Options for Texinfo output ------------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'Sahara-specs', 'Sahara Design Specs',
'OpenStack Sahara Team', 'sahara-specs', 'Design specifications for the Sahara project.',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
#texinfo_appendices = []
# If false, no module index is generated.
#texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'
# -- Options for Epub output ---------------------------------------------------
# Bibliographic Dublin Core info.
epub_title = 'Sahara Specs'
epub_author = 'OpenStack Sahara Team'
epub_publisher = 'OpenStack Sahara Team'
epub_copyright = '2014, OpenStack Sahara Team'
# The language of the text. It defaults to the language option
# or en if the language is not set.
#epub_language = ''
# The scheme of the identifier. Typical schemes are ISBN or URL.
#epub_scheme = ''
# The unique identifier of the text. This can be a ISBN number
# or the project homepage.
#epub_identifier = ''
# A unique identification for the text.
#epub_uid = ''
# A tuple containing the cover image and cover page html template filenames.
#epub_cover = ()
# HTML files that should be inserted before the pages created by sphinx.
# The format is a list of tuples containing the path and title.
#epub_pre_files = []
# HTML files shat should be inserted after the pages created by sphinx.
# The format is a list of tuples containing the path and title.
#epub_post_files = []
# A list of files that should not be packed into the epub file.
#epub_exclude_files = []
# The depth of the table of contents in toc.ncx.
#epub_tocdepth = 3
# Allow duplicate toc entries.
#epub_tocdup = True

59
doc/source/index.rst
View File

@ -1,59 +0,0 @@
.. sahara-specs documentation master file
===============================================
Data Processing Service (Sahara) Specifications
===============================================
Implemented specs by release:
.. toctree::
:glob:
:maxdepth: 2
specs/rocky_idx.rst
specs/queens_idx.rst
specs/pike_idx.rst
specs/ocata_idx.rst
specs/newton_idx.rst
specs/mitaka_idx.rst
specs/liberty_idx.rst
specs/kilo_idx.rst
specs/juno_idx.rst
Unimplemented backlog specs:
.. toctree::
:glob:
:maxdepth: 2
specs/backlog_idx.rst
Other components:
.. toctree::
:glob:
:maxdepth: 2
specs/saharaclient_idx.rst
.. toctree::
:glob:
:maxdepth: 2
specs/sahara-image-elements_idx.rst
.. toctree::
:glob:
:maxdepth: 2
specs/sahara-tests_idx.rst
.. only:: html
Indices and tables
==================
* :ref:`search`

52
doc/source/redirect.py
View File

@ -1,52 +0,0 @@
# A simple sphinx plugin which creates HTML redirections from old names
# to new names. It does this by looking for files named "redirect" in
# the documentation source and using the contents to create simple HTML
# redirection pages for changed filenames.
# Stolen from openstack/nova-specs
import os.path
from sphinx.util import logging
LOG = logging.getLogger(__name__)
def setup(app):
from sphinx import application
if not isinstance(app, application.Sphinx):
return
app.connect('build-finished', emit_redirects)
def process_redirect_file(app, path, ent):
parent_path = path.replace(app.builder.srcdir, app.builder.outdir)
with open(os.path.join(path, ent)) as redirects:
for line in redirects.readlines():
from_path, to_path = line.rstrip().split(' ')
from_path = from_path.replace('.rst', '.html')
to_path = to_path.replace('.rst', '.html')
redirected_filename = os.path.join(parent_path, from_path)
redirected_directory = os.path.dirname(redirected_filename)
if not os.path.exists(redirected_directory):
os.makedirs(redirected_directory)
with open(redirected_filename, 'w') as f:
f.write('<html><head><meta http-equiv="refresh" content="0; '
'url=%s" /></head></html>'
% to_path)
def emit_redirects(app, exc):
LOG.info('scanning %s for redirects...', app.builder.srcdir)
def process_directory(path):
for ent in os.listdir(path):
p = os.path.join(path, ent)
if os.path.isdir(p):
process_directory(p)
elif ent == 'redirects':
LOG.info(' found redirects at %s' % p)
process_redirect_file(app, path, ent)
process_directory(app.builder.srcdir)
LOG.info('...done')

1
doc/source/specs
View File

@ -1 +0,0 @@
../../specs

9
requirements.txt
View File

@ -1,9 +0,0 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
pbr!=2.1.0,>=2.0.0 # Apache-2.0
stestr>=1.0.0 # Apache-2.0
testtools>=2.2.0 # MIT
yasfb>=0.8.0

12
setup.cfg
View File

@ -1,12 +0,0 @@
[metadata]
name = sahara-specs
summary = OpenStack Sahara Project Development Specs
description_file =
README.rst
author = OpenStack
author_email = openstack-discuss@lists.openstack.org
home_page = https://specs.openstack.org/openstack/sahara-specs/
classifier =
Intended Audience :: Developers
License :: OSI Approved :: Apache Software License
Operating System :: POSIX :: Linux

22
setup.py
View File

@ -1,22 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT
import setuptools
setuptools.setup(
setup_requires=['pbr'],
pbr=True)

159
specs/backlog/boot-from-volume.rst
View File

@ -1,159 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
============================
Support for Boot from Volume
============================
This specification proposes to add boot from volume capability to Sahara.
Problem description
===================
Sahara engine provisions VMs using a Glance image directly. In most
installations this means that the image is copied as a local file the
Nova-compute host and used as a root disk for the VM.
Having the root disk as a plain file introduces some limitations:
* Nova VM live migration (or Host evacuation) is not possible.
* Root disk performance may significantly suffer if QCOW2 format is used.
Having root disk replaced with a bootable Volume backed by a distributed
backend or local disk storage will solve the limitation listed above.
Proposed change
===============
The ability to boot from volume still requires an image to serve as source.
This means that Volume base provisioning will require 2 steps.
* Create a bootable volume from a registered Sahara image.
* Boot a VM with the block_device_mapping parameter pointing to the created
volume.
Volume based provisioning requires the volume size to be set explicitly. This
means that Sahara should take care about this parameter. The proposed way to
set the bootable volume size is take the root disk size of the flavor being
used for the VM.
If the user selects the Node Group to be provisioned from the volume, he should
set the boot_from_volume flag to True.
The volume based provisioning is different from image based and implies the
following change.
The image parameter from the instance template should be removed.
The instance Heat template should have a new section with the block device
mapping.
.. code-block:: yaml
block_device_mapping: [{
device_name: "vda",
volume_id : {
get_resource : bootable_volume },
delete_on_termination : "true" }
]
The resource group definition should have a volume added by the following
template:
.. code-block:: yaml
bootable_volume:
type: OS::Cinder::Volume
properties:
size: <size derived from the flavor>
image: <regular Sahara image>
Alternatives
------------
Alternatively the user may be allowed to chose an existing volume to boot from.
This however cannot guarantee that the provided volume is suitable for the
cluster installation. Sahara also requires a username metadata to be able to
log into VM. This metadata is only stored in images right now.
Data model impact
-----------------
Node Group Template, Node Group and Templates relation objects should now
have a boolean boot_from_volume field.
REST API impact
---------------
Convert and Boot from Volume flag should be added to all endpoints responsible
for Node Group manipulation.
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
An option should be added to the Node Group create and update forms.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Nikita Konovalov nkonovalov@mirantis.com
Work Items
----------
* Implement boot from volume support at the backend with the db migration and
Heat templates update.
* Add boot_from_volume flag support in python-saharaclient
* Add boot_from_volume flag support in sahara-dashboard
Dependencies
============
None
Testing
=======
* Unit test coverage in sahara and python-saharaclient repositories.
* Integration test coverage in sahara-tests framework.
Documentation Impact
====================
* REST API documents should be updated.
* General user documentation should describe the behavior introduced by the
boot_from_volume flag.
References
==========
None

104
specs/backlog/chronic-edp-revival.rst
View File

@ -1,104 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Revival of "Chronic EDP" work
==========================================
https://blueprints.launchpad.net/sahara/+spec/enable-scheduled-edp-jobs
https://blueprints.launchpad.net/sahara/+spec/recurrence-edp-job
https://blueprints.launchpad.net/sahara/+spec/add-suspend-resume-ability-for-edp-jobs
Problem description
===================
Three specs about time-related EDP job submission are partially completed.
The work could be revived.
Proposed change
===============
With reference to existing specs/patches, revive any or all of the 3 specs.
Alternatives
------------
Keep the specs in their current state, either abandoned or partially complete.
Data model impact
-----------------
Refer to existing specs.
REST API impact
---------------
Refer to existing specs.
Other end user impact
---------------------
Refer to existing specs.
Deployer impact
---------------
Refer to existing specs.
Developer impact
----------------
Refer to existing specs.
Image Packing impact
--------------------
Refer to existing specs.
Sahara-dashboard / Horizon impact
---------------------------------
Refer to existing specs.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
None
Other contributors:
None
Work Items
----------
* Enumerate and verify what has already been done
* Decide what work should be revived
* Clean up doc of exisitng work
* Follow steps outlined in existing specs for new work
Dependencies
============
None
Testing
=======
Refer to existing specs.
Documentation Impact
====================
Refer to existing specs.
References
==========
None

128
specs/backlog/heat-two-step-scaling.rst
View File

@ -1,128 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=================================
Two step scaling with Heat engine
=================================
https://blueprints.launchpad.net/sahara/+spec/heat-two-step-scaling
In case of failure during cluster scaling Heat will rollback deletion of
all resources. After that Sahara will ask Heat to delete them anyway.
"Rollback deletion and delete after that" step looks unnecessary.
Problem description
===================
The following scenario happens when Sahara with Heat engine failed to scale
cluster:
1. User requested cluster scaling
2. Sahara runs hadoop nodes decommissioning
3. Sahara runs heat stack update with both added and removed nodes and
rollback_on_failure=True
4. If 3 failed Heat returns all deleted nodes back
5. Sahara runs heat stack update with removed nodes only
6. Heat removes nodes one more time
So, at step 4 Heat restores nodes that will be deleted later anyway.
Proposed change
===============
The described problem could be avoided by scaling in two steps. So, resulting
flow will look like this:
1. User requested cluster scaling
2. Sahara runs hadoop nodes decommissioning
3. Sahara runs heat stack update with removed resources only and
rollback_on_failure=False
4. Sahara runs heat stack update with new resources and
rollback_on_failure=True
In this case if step 4 failed Heat will not try to restore deleted resources.
It will rollback to the state where resources are already deleted.
If step 3 fails there is nothing Sahara can do. Cluster will be moved to
'Error' state.
Alternatives
------------
Do nothing since issue appears only in rare scenario of failed scaling down.
Data model impact
-----------------
None.
REST API impact
---------------
None.
Other end user impact
---------------------
None.
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
alazarev (Andrew Lazarev)
Other contributors:
None
Work Items
----------
* Perform changes
* Test that rollback on cluster scale failure works as expected
Dependencies
============
None.
Testing
=======
Manually.
Documentation Impact
====================
None.
References
==========
None.

19
specs/backlog_idx.rst
View File

@ -1,19 +0,0 @@
Backlog
^^^^^^^
This directory is for specifications that have been approved but not yet
implemented. If you are interested in implementing one of these specifications
submit a review request moving it to the appropriate release.
The updated review should reflect the new primary assignee. Please maintain
all previous assignees in the list of Other Contributors.
If a specification has been partially implemented, the document in the backlog
will contain information of what has been completed.
.. toctree::
:glob:
:maxdepth: 1
backlog/*

0
specs/juno/.gitignore vendored
View File

129
specs/juno/anti-affinity-via-server-groups.rst
View File

@ -1,129 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==================================================
Make anti affinity working via server groups
==================================================
https://blueprints.launchpad.net/sahara/+spec/anti-affinity-via-server-groups
Server groups is an openstack way to implement anti affinity. Anti affinity
was implemented in Sahara before server groups were introduced in nova. Now it
is time to replace custom solution with the common one.
Problem description
===================
Direct engine uses manual scheduler hints for anti affinity.
Heat engine has limited anti affinity support and also uses scheduler hints
(https://bugs.launchpad.net/sahara/+bug/1268610).
Nova has generic mechanism for this purpose.
Proposed change
===============
Proposed solution is to switch both engines to implementation that uses server
groups.
Current server group implementation has limitation that each server could
belong to a single server group only. We can handle this constraint by having
one server group per cluster. In this case each instance with affected
processes will be included to this server group. So, with such implementation
there will be no several affected instances on the same host even if they
don't have common processes. Such implementation is fully compliant with all
documentation we have about anti affinity.
We need to keep backward compatibility for direct engine. Users should be
able to scale clusters deployed on Icehouse release. Sahara should update
already spawned VMs accordingly. Proposed solution - Sahara should check
server group existence during scaling and update whole cluster if server group
is missing.
We don't care about backward compatibility for heat engine. It was in beta
state for Icehouse and there are other changes that break it.
Alternatives
------------
We can implement anti affinity via server groups in heat engine only. We will
stop support of direct engine somewhere in the future. So, we can freeze
current behavior in direct engine and don't change it until it is deprecated
and removed.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
Anti affinity behavior changes in case of several involved processes. Before
the change there was possible situation when several instances with affected
(but different) processes are spawned on the same host. After the change all
instances with affected processes will be scheduled to different hosts.
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
alazarev
Work Items
----------
Implement change
Dependencies
============
None
Testing
=======
Will be covered by current integration tests.
Documentation Impact
====================
Need note in upgrade notes about anti affinity behavior change.
References
==========
* Team meeting minutes: http://eavesdrop.openstack.org/meetings/sahara/2014/sahara.2014-08-21-18.02.html

107
specs/juno/append-to-remote-file.rst
View File

@ -1,107 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
================================
Append to a remote existing file
================================
https://blueprints.launchpad.net/sahara/+spec/append-to-remote-file
Sahara utils remote can only create a new file and write to it or replace a
line for a new one, but it can't append to an existing file. This bp aims to
implement this feature.
Problem description
===================
When managing remote files, sahara can only create new files and replace lines
from existing one. The feature to append to an existing file doesn't exist
and it is necessary.
Proposed change
===============
Implement this feature following the idea of the write_to_file method
The code is basically the same, the change will be the method of opening
the file. Write uses 'w' we need to use 'a'.
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
* tellesmvn
Work Items
----------
The implementation is very basic, the idea is similar to the write_file,
the necessary change is to open the remote file in append mode.
Dependencies
============
None
Testing
=======
None for now.
Documentation Impact
====================
None
References
==========
None

157
specs/juno/cdh-plugin.rst
View File

@ -1,157 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
====================================
Plugin for CDH with Cloudera Manager
====================================
https://blueprints.launchpad.net/sahara/+spec/cdh-plugin
This specification proposes to add CDH plugin with Cloudera Distribution of
Hadoop and Cloudera Manager in Sahara.
Problem description
===================
Cloudera is open-source Apache Hadoop distribution, CDH (Cloudera Distribution
Including Apache Hadoop). CDH contains the main, core elements of Hadoop that
provide reliable, scalable distributed data processing of large data sets
(chiefly MapReduce and HDFS), as well as other enterprise-oriented components
that provide security, high availability, and integration with hardware and
other software. Cloudera Manager is the industry's first end-to-end management
application for Apache Hadoop. Cloudera Manager provides many useful features
for monitoring the health and performance of the components of your cluster
(hosts, service daemons) as well as the performance and resource demands of
the user jobs running on your cluster. [1]
Proposed change
===============
CDH plugin implementation will support Cloudera Manager version 5 and CDH
version 5.
Plugin will support key Sahara features:
* Cinder integration
* Cluster scaling
* EDP
* Cluster topology validation
* Integration with Swift
* Data locality
Plugin will be able to install following services:
* Cloudera Manager
* HDFS
* YARN
* Oozie
CDH plugin will support the following OS: Ubuntu 12.04 and CentOS 6.5.
CDH provisioning plugin will support mirrors with CDH and CM packages.
By default CDH doesn't support Hadoop Swift library. Integration with Swift
should be added to CDH plugin. CDH maven repository contains Hadoop Swift
library. [2]
CDH plugin will support the following processes:
* MANAGER - Cloudera Manager, master process
* NAMENODE - HDFS NameNode, master process
* SECONDARYNAMENODE - HDFS SecondaryNameNode, master process
* RESOURCEMANAGER - YARN ResourceManager, master process
* JOBHISTORY - YARN JobHistoryServer, master process
* OOZIE - Oozie server, master process
* DATANODE - HDFS DataNode, worker process
* NODEMANAGER - YARN NodeManager, worker process
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
CDH plugin must be support vanilla images and images with Cloudera packages.
For building pre-installed images with Cloudera packages use specific CDH
elements.
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
sreshetniak
Other contributors:
iberezovskiy
Work Items
----------
* Add implementation of plugin
* Add jobs in Sahara-ci
* Add integration tests
* Add elements to Sahara-image-elements for building images with pre-installed
Cloudera packages
Dependencies
============
Depends on OpenStack requirements, needs a `cm_api` python library version
6.0.2, which is not present in OS requirements. [3] Need to add `cm_api` to
OS requirements. [4]
Testing
=======
* Add unit tests to Sahara to cover basic functionality of plugin
* Add integration tests to Sahara
Documentation Impact
====================
CDH plugin documentation should be added to the plugin section of Sahara docs.
References
==========
* [1] http://www.cloudera.com/content/cloudera/en/products-and-services/cdh.html
* [2] https://repository.cloudera.com/artifactory/repo/org/apache/hadoop/hadoop-openstack/
* [3] https://pypi.python.org/pypi/cm-api
* [4] https://review.openstack.org/#/c/106011/

131
specs/juno/ceilometer-integration.rst
View File

@ -1,131 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
====================================================
Specification for integration Sahara with Ceilometer
====================================================
https://blueprints.launchpad.net/sahara/+spec/ceilometer-integration
It is impossible to send notifications from Sahara to Ceilometer now. Sahara
should have an ability to send notifications about cluster modifications,
for example about creating/updating/destoroying cluster.
Problem description
===================
New feature will provide the following ability:
* Sending notifications to Ceilometer about cluster modifications, which
can help for user to achive some information about clusters which he have:
number of active clusters in each moment and etc.
Proposed change
===============
Change will consist of following modifications:
* Adding to Sahara an ability to send notifications.
* Adding to Sahara sending notificitions in such places, where cluster is
modified
* Adding to Ceilometer an ability to pull notifications from Sahara exchange
and to parse it.
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
vgridnev
Other contributors:
slukjanov
Work Items
----------
* Add notification sender in Sahara
* Add Ceilometer parser
* Add unit tests
Dependencies
============
Depends on OpenStack requirements
Testing
=======
There will be:
* unit tests in Sahara
* unit tests in Ceilometer
Documentation Impact
====================
Need modifications in Ceilometer documentation here:
* [1] http://docs.openstack.org/developer/ceilometer/measurements.html
* [2] http://docs.openstack.org/developer/ceilometer/install/manual.html
References
==========
* [1] https://review.openstack.org/#/c/108982/

105
specs/juno/cluster-persist-sahara-configuration.rst
View File

@ -1,105 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
================================================
Store Sahara configuration in cluster properties
================================================
https://blueprints.launchpad.net/sahara/+spec/cluster-persist-sahara-configuration
It is important to know Sahara version and configuration for proper cluster
migration and preventing dangerous operations.
Problem description
===================
Now Sahara has no way to know conditions when cluster was created. Cluster
operations after Openstack upgrade or switching infrastructure engine could be
dangerous and cause data loss.
Proposed change
===============
Store main Sahara properties (version, type and version of infrastructure
engine, etc) to cluster DB object. This will allow to prevent dangerous
operations and notify user gracefully.
Alternatives
------------
Don't store any information about Sahara settings. Always assume that settings
didn't change and we have current or previous release of OpenStack.
Data model impact
-----------------
New field in cluster object. Probably this should be dictionary with defined
keys.
REST API impact
---------------
We can expose information about Sahara settings to user. Or not.
Other end user impact
---------------------
Some error messages could become more descriptive.
Deployer impact
---------------
None
Developer impact
----------------
More options to handle errors.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
alazarev
Work Items
----------
Implement change
Dependencies
============
None
Testing
=======
Manual
Documentation Impact
====================
None
References
==========
None

124
specs/juno/cluster-secgroups.rst
View File

@ -1,124 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==============================================================
Security groups management in Sahara
==============================================================
https://blueprints.launchpad.net/sahara/+spec/cluster-secgroups
It is not acceptable for production use to require default security group with
all ports open. Sahara need more flexible way to work with security groups.
Problem description
===================
Now Sahara doesn't manage security groups and use default security group for
instances provisioning.
Proposed change
===============
Solution will consist of several parts:
1) Allow user to specify list of security groups for each of node groups.
2) Add support of automatic security group creation. Sahara knows everything
to create security group with required ports open. In the first iteration
this will be security group with all exposed ports open for all networks.
Alternatives
------------
Creation of security groups by Sahara could be done in several ways. Ideally
Sahara should support separation between different networks and configuration
on what to allow and what is not.
Data model impact
-----------------
1) List of security groups need to be saved in each node group.
2) Flag indicating that one of security groups is created by Sahara
3) List of ports to be opened. It need to be stored somewhere to provide this
information to provisioning engine.
REST API impact
---------------
Requests to create cluster, nodegroup, cluster template and nodegroup template
will be extended to receive security groups to use. Also option for
automatic security group creation will be added.
Other end user impact
---------------------
None
Deployer impact
---------------
In some cases there will be no need to configure default security group.
Developer impact
----------------
Plugin SPI will be extended with method to return required ports for node
group.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
New field to select security group in all create screens.
Implementation
==============
Assignee(s)
-----------
Primary Assignee:
Andrew Lazarev (alazarev)
Work Items
----------
* Allow user to specify security groups for node group
* Implement ability of security group creation by Sahara
Both items require the following steps:
* Implement in both engines (heat and direct engine)
* Test for nova network and neutron
* Update documentation
* Update UI
* Create integration test
Dependencies
============
None
Testing
=======
Feature need to be covered by integration tests both for engine and UI.
Documentation Impact
====================
Feature need to be documented.
References
==========
None

159
specs/juno/edp-move-examples.rst
View File

@ -1,159 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================================
Move the EDP examples from the sahara-extra repo to sahara
==========================================================
https://blueprints.launchpad.net/sahara/+spec/edp-move-examples
Moving the Sahara EDP examples from the sahara-extra repo to
the sahara repo accomplishes several things:
* It eliminates code duplication since the examples are actually
used in integration tests
* It removes an element from the sahara-extra repo, thereby moving
us closer to retiring that repo and simplifying our repo structure
* It puts examples where developers are more likely to find it, and
makes it simpler to potentially bundle the examples with a Sahara
distribution
Problem description
===================
The goal is to create one unified set of EDP jobs that can
be used to educate users and developers on how to create/run
jobs and can also be used as jobs submitted during integration
testing.
Proposed change
===============
Under the sahara root directory, we should create a new directory::
sahara/edp-examples
The directory structure should follow a standard pattern (names
are not important per se, this is just an illustration)::
subdirectory_for_each_example/
README.rst (what it is, how to compile, etc)
script_and_jar_files
src_for_jars/
how_to_run_from_node_command_line/ (optional)
expected_input_and_output/ (optional)
hadoop_1_specific_examples/
subdirectory_for_each_example
hadoop_2_specific_examples/
subdirectory_for_each_example
The integration tests should be modified to pull job files from
the sahara/edp-examples directory.
Here are some notes on equivalence for the current script and jar
files in ``sahara-extra/edp-examples`` against
``sahara/tests/integration/tests/resources``::
pig-job/example.pig == resources/edp-job.pig
pig-job/udf.jar == resources/edp-lib.jar
wordcount/edp-java.jar == resources/edp-java/edp-java.jar
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
Examples won't be found in the sahara-extra repo any longer.
We should perhaps put a README file there that says "We have
moved" for a release cycle.
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
None as yet
Work Items
----------
The problem has several components:
* Move the examples to the sahara repository
* Merge any jobs used by the integration tests into the new
examples directory to create one comprehensive set
* Provide source code and compilation instructions for any
examples that currently lack them
* Make the integration tests reference the new directory structure
* Delineate which, if any, examples work only with specific
Hadoop versions. Most examples work on both Hadoop 1 and Hadoop 2
but some do not. Version-specific examples should be in a subdirectory
named for the version
Dependencies
============
None
Testing
=======
Testing will be inherent in the integration tests. The change will be
deemed successful if the integration tests run successfully after the
merging of the EDP examples and the integration test jobs.
Documentation Impact
====================
If our current docs reference the EDP examples, those references should
change to the new location. If our current docs do not reference the
EDP examples, a reference should be added in the developer and/or user
guide.
References
==========
None

216
specs/juno/edp-refactor-job-manager.rst
View File

@ -1,216 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==============================================================
[EDP] Refactor job manager to support multiple implementations
==============================================================
https://blueprints.launchpad.net/sahara/+spec/edp-refactor-job-manager
The job manager at the core of Sahara EDP (Elastic Data Processing) was
originally written to execute and monitor jobs via an Oozie server. However,
the job manager should allow for alternative EDP implementations which
can support new cluster types or new job execution environments.
Problem description
===================
To date, the provisioning plugins released with Sahara all support deployment
of an Oozie server. Oozie was a logical choice for the early releases of
Sahara EDP because it provided commonality across several plugins and allowed
the rapid development of the EDP feature.
However, Oozie is built on top of Hadoop Mapreduce and not every cluster
configuration can or should support it (consider a Spark cluster, for example,
where Hadoop Mapreduce is not a necessary part of the install). As Sahara
supports additional cluster types and gains a wider install base, it's
important for EDP to have the flexibility to run on new cluster configurations
and new job execution paradigms.
The current implementation of the job_manager has hardcoded dependencies on
Oozie. These dependencies should be removed and the job manager should be
refactored to support the current Oozie implementation as well as new
implementations. The job manager should select the appropriate implementation
for EDP operations based on attributes of the cluster.
Proposed change
===============
Sahara EDP requires three basic operations on a job:
* Launch the job
* Poll job status
* Terminate the job
Currently these operations are implemented in job_manager.py with explicit
dependencies on the Oozie server and Oozie-style workflows.
To move these dependencies out of the job manager, we can define an abstract
class that contains the three operations::
@six.add_metaclass(abc.ABCMeta)
class JobEngine(object):
@abc.abstractmethod
def cancel_job(self, job_execution):
pass
@abc.abstractmethod
def get_job_status(self, job_execution):
pass
@abc.abstractmethod
def run_job(self, job_execution):
pass
For each EDP implementation Sahara supports, a class should be derived from
JobEngine that contains the details. Each implementation and its supporting
files should be contained in its own subdirectory.
Logic can be added to the job manager to allocate a JobEngine based on the
cluster and/or job type, and the job manager can call the appropriate method
on the allocated JobEngine.
As much as possible the JobEngine classes should be concerned only with
implementing the operations. They may read objects from the Sahara database as
needed but modifications to the job execution object should generally be
done in the job_manager.py (in some cases this may be difficult, or may require
optional abstract methods to be added to the JobEngine base class).
For example, the "cancel_job" sequence should look something like this:
1) "cancel_job(id)" is called in job_manager.py
2) The job manager retrieves the job execution object and the cluster
object and allocates an appropriate job engine
3) The job manager calls engine.cancel_job(job_execution)
4) The engine performs necessary steps to cancel the job
5) The job manager traps and logs any exceptions
6) The job manager calls engine.get_job_status(job_execution)
7) The engine returns the status for the job, if possible
8) The job manager updates the job execution object in the Sahara database
with the indicated status, if any, and returns
In this example, the job manager has no knowledge of operations in the engine,
only the high level logic to orchestrate the operation and update the status.
Alternatives
------------
It may be possible to support "true" plugins for EDP similar to the
implementation of the provisioning plugins. In this case, the plugins would be
discovered and loaded dynamically at runtime.
However, this requires much more work than a refactoring and introduction of
abstract classes and is probably not realistic for the Juno release. There are
several problems/questions that need to be solved:
* Should EDP interfaces simply be added as optional methods in the current
provisioning plugin interface, or should EDP plugins be separate entities?
* Do vendors that supply the provisioning plugins also supply the EDP plugins?
* If separate EDP plugins are chosen, a convention is required to associate an
EDP plugin with a provisioning plugin so that the proper EDP implementation
can be chosen at runtime for a particular cluster without any internal glue
* For clusters that are running an Oozie server, should the Oozie EDP
implementation always be the default if another implementation for the
cluster is not specified? Or should there be an explicitly designated
implementation for each cluster type?
* It requires not only a formal interface for the plugin, but a formal
interface for elements in Sahara that the plugin may require. For instance,
an EDP plugin will likely need a formal interface to the conductor so that it
can retrieve EDP objects (job execution objects, data sources, etc).
Data model impact
-----------------
This change should only cause one minor (optional) change to the current data
model. Currently, a JobExecution object contains a string field named
"oozie_job_id". While a job id field will almost certainly still be needed for
all implementations and can be probably be stored as a string, the name should
change to "job_id".
JobExecution objects also contain an "extra" field which in the case of the
Oozie implementation is used to store neutron connection info for the Oozie
server. Other implementations may need similar data, however since the "extra"
field is stored as a JsonDictType no change should be necessary.
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary Assignee:
Trevor McKay
Work Items
----------
Dependencies
============
None
Testing
=======
Testing will be done primarily through the current unit and integration tests.
Tests may be added that test the selection of the job engine.
Documentation Impact
====================
None
References
==========
None

191
specs/juno/edp-spark-job-type.rst
View File

@ -1,191 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
========================================================
[EDP] Add a Spark job type (instead of overloading Java)
========================================================
https://blueprints.launchpad.net/sahara/+spec/edp-spark-job-type
Spark EDP has been implemented initially using the Java job type. However,
the spark semantics are slightly different and Spark jobs will probably
continue to diverge from Java jobs during future development. Additionally,
a specific job type will help users distinguish between Spark apps and Java
mapreduce jobs in the Sahara database.
Problem description
===================
The work involves adding a Spark job type to the job type enumeration in
Sahara and extending the dashboard to allow the creation and submission
of Spark jobs. The Sahara client must be able to create and submit Spark
jobs as well (there may not be any new work in the client to support this).
Existing unit tests and integration tests must be repaired if the addition
of a new job type causes them to fail. Unit tests analagous to the tests
for current job types should be added for the Spark job type.
Integration tests for Spark clusters/jobs will be added as a separate effort.
Proposed change
===============
Changes in the Sahara-api code:
* Add the Spark job type to the enumeration
* Add validation methods for job creation and job execution creation
* Add unit tests for the Spark job type
* Add the Spark job type to the job types supported by the Spark plugin
+ Leave the Java type supported for Spark until the dashboard is changed
* Add config hints for the Spark type
+ These may be empty initially
Changes in the Sahara dashboard:
* Add the Spark job type as a selectable type on the job creation form.
+ Include the "Choose a main binary" input on the Create Job tab
+ Supporting libraries are optional, so the form should include the Libs tab
* Add a "Launch job" form for the Spark job type
+ The form should include the "Main class" input.
+ No data sources, as with Java jobs
+ Spark jobs will share the edp.java.main_class configuration with Java jobs.
Alternatively, Spark jobs could use a edp.spark.main_class config
+ There may be additional configuration parameters in the future, but none
are supported at present. The Configuration button may be included or
left out.
+ The arguments button should be present
Alternatives
------------
Overload the existing Java job type. It is similar enough to work as a
proof of concept, but long term this is probably not clear, desirable or
maintainable.
Data model impact
-----------------
None. Job type is stored as a string in the database, so there should be
no impact there.
REST API impact
---------------
None. The JSON schema will list "Spark" as a valid job type, but the API
calls themselves should not be affected.
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
Described under Proposed Change.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Trevor McKay (sahara-api)
Other contributors:
Chad Roberts (dashboard)
Work Items
----------
Additional notes on implementation of items described under Proposed Change:
* The simple addition of JOB_TYPE_SPARK to sahara.utils.edp.JOB_TYPES_ALL
did not cause existing unit tests to fail in an experiment
* Existing unit tests should be surveyed and analagous tests for the Spark job
type should be added as appropriate
* sahara.service.edp.job_manager.get_job_config_hints(job_type) needs to handle
the Spark job type. Currently all config hints are retrieved from the Oozie
job engine, this will not be correct.
+ Related, the use of JOB_TYPES_ALL should probably be modified in
workflow_creator.workflow_factory.get_possible_job_config() just to be safe
* sahara.service.edp.job_utils.get_data_sources() needs to treat Spark jobs
like Java jobs (there are no data sources, only arguments)
* service.validations.edp.job.check_main_libs() needs to require a main
application jar for Spark jobs and allow supporting libs as optional
* service.validations.edp.job_executor.check_job_executor()
+ Spark requires edp.java.main_class (or edp.spark.main_class)
+ check_edp_job_support() is called here and should be fine. The default is
an empty body and the Spark plugin does not override this method since the
Spark standalone deployment is part of a Spark image generated from DIB
* Use of EDP job types in sahara.service.edp.oozie.workflow_creator should not
be impacted since Spark jobs shouldn't be targeted to an Oozie engine by the
job manager (but see note on get_job_config_hints() and JOB_TYPES_ALL)
* The Sahara client does not appear to reference specific job type values so
there is likely no work to do in the client
Dependencies
============
This depends on https://blueprints.launchpad.net/sahara/+spec/edp-spark-standalone
Testing
=======
New unit tests will be added for the Spark job type, analogous to existing
tests for other job types. Existing unit and integration tests will ensure that
other job types have not been broken by the addition of a Spark type.
Integration tests for Spark clusters should be added in the following
blueprint, including tests for EDP with Spark job types
https://blueprints.launchpad.net/sahara/+spec/edp-spark-integration-tests
Documentation Impact
====================
The User Guide calls out details of the different job types for EDP.
Details of the Spark type will need to be added to this section.
References
==========
None.

239
specs/juno/edp-spark-standalone.rst
View File

@ -1,239 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=====================================================
[EDP] Add an engine for a Spark standalone deployment
=====================================================
https://blueprints.launchpad.net/sahara/+spec/edp-spark-standalone
The Spark provisioning plugin allows the creation of Spark standalone
clusters in Sahara. Sahara EDP should support running Spark
applications on those clusters.
Problem description
===================
The Spark plugin uses the *standalone* deployment mode for Spark
(as opposed to Spark on YARN or Mesos). An EDP implementation must be created
that supports the basic EDP functions using the facilities provided by the
operating system and the Spark standalone deployment.
The basic EDP functions are:
* **run_job()**
* **get_job_status()**
* **cancel_job()**
Proposed change
===============
The Sahara job manager has recently been refactored to allow provisioning
plugins to select or provide an EDP engine based on the cluster and job_type.
The plugin may return a custom EDP job engine object or choose a default engine
provided by Sahara.
A default job engine for Spark standalone clusters can be added to Sahara
that implements the basic EDP functions.
Note, there are no public APIs in Spark for job status or cancellation
beyond facilities that might be available through a SparkContext
object instantiated in a Scala program. However, it is possible to
provide basic functionality without developing Scala programs.
* **Engine selection criteria**
The Spark provisioning plugin must determine if the default Spark EDP
engine may be used to run a particular job on a particular cluster. The
following conditions must be true to use the engine
+ The default engine will use *spark-submit* for running jobs, therefore
a cluster must have at least Spark version 1.0.0 to use the engine.
+ The job type must be Java (initially)
* **Remote commands via ssh**
All operations should be implemented using ssh to run remote
commands, as opposed to writing a custom agent and client. Furthermore,
any long running commands should be run asynchronously.
* **run_job()**
The run_job() function will be implemented using the *spark-submit*
script provided by Spark.
+ *spark-submit* must be run using client deploy mode
+ The *spark-submit* command will be executed via ssh. Ssh must
return immediately and must return a process id (PID) that can be
used for checking status or cancellation. This implies that the
process is run in the background.
+ SIGINT must not be ignored by the process running *spark-submit*.
Care needs to be taken here, since the default behavior of a
process backgrounded from bash is to ignore SIGINT. (This can be
handled by running *spark-submit* as a subprocess from a wrapper
which first restores SIGINT, and launching the wrapper from ssh. In this
case the wrapper must be sure to propagate SIGINT to the child).
+ *spark-submit* requires that the main application jar be in
local storage on the node where it is run. Supporting jars may
be in local storage or hdfs. For simplicity, all jars will be uploaded
to local storage.
+ *spark-submit* should be run from a subdirectory on the
master node in a well-known location. The subdirectory naming
should incorporate the job name and job execution id from
Sahara to make locating the directory easy. Program files,
output, and logs should be written to this directory.
+ The exit status returned from *spark-submit* must be written
to a file in the working directory.
+ *stderr* and *stdout* from *spark-submit* should be redirected
and saved in the working directory. This will help debugging as well
as preserve results for apps like SparkPi which write the result to stdout.
+ Sahara will allow the user to specify arguments to the Spark
application. Any input and output data sources will be specified
as path arguments to the Spark app.
* **get_job_status()**
Job status can be determined by monitoring the PID returned
by run_job() via *ps* and reading the file containing the exit status
+ The initial job status is PENDING (the same for all Sahara jobs)
+ If the PID is present, the job status is RUNNING
+ If the PID is absent, check the exit status file
- If the exit status is 0, the job status is SUCCEEDED
- If the exit status is -2 or 130, the job status is KILLED (by SIGINT)
- For any other exist status, the job status is DONEWITHERROR
+ If the job fails in Sahara (ie, because of an exception), the job status
will be FAILED
* **cancel_job()**
A Spark application may be canceled by sending SIGINT to the
process running *spark-submit*.
+ cancel_job() should send SIGINT to the PID returned by run_job().
If the PID is the process id of a wrapper, the wrapper must
ensure that SIGINT is propagated to the child
+ If the command to send SIGINT is successful (ie, *kill* returns 0),
cancel_job() should call get_job_status() to update the job status
Alternatives
------------
The Ooyala job server is an alternative for implementing Spark EDP, but it's
a project of its own outside of OpenStack and introduces another dependency. It
would have to be installed by the Spark provisioning plugin, and Sahara
contributors would have to understand it thoroughly.
Other than Ooyala, there does not seem to be any existing client or API for
handling job submission, monitoring, and cancellation.
Data model impact
-----------------
There is no data model impact, but a few fields will be reused.
The *oozie_job_id* will store an id that allows the running application
to be operated on. The name of this field should be generalized at some
point in the future.
The job_execution.extra field may be used to store additional information
necessary to allow operations on the running application
REST API impact
---------------
None
Other end user impact
---------------------
None. Initially Spark jobs (jars) can be run using the Java job type.
At some point a specific Spark job type will be added (this will be
covered in a separate specification).
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Trevor McKay
Work Items
----------
* implement default spark engine selection in spark provisioning plugin
* implement run
* implement get_job_status
* implement cancel
* implement launch wrapper
* implement unit tests
Dependencies
============
None
Testing
=======
Unit tests will be added for the changes in Sahara.
Integration tests for Spark standalone clusters will be added in another
blueprint and specification.
Documentation Impact
====================
The Elastic Data Processing section of the User Guide should talk about
the ability to run Spark jobs and any restrictions.
References
==========

295
specs/juno/edp-swift-trust-authentication.rst
View File

@ -1,295 +0,0 @@
=====================================================
[EDP] Using trust delegation for Swift authentication
=====================================================
https://blueprints.launchpad.net/sahara/+spec/edp-swift-trust-authentication
Sahara currently stores and distributes credentials for access to Swift
objects. These credentials are username/password pairs that are stored in
Sahara's database. This blueprint describes a method for using Keystone’s
trust delegation mechanism in conjuction with temporary proxy users to remove
the storage of user credentials from Sahara's purview.
Problem description
===================
Sahara allows access to job binaries and data sources stored in Swift
containers by storing a user's credentials to those containers. The
credentials are stored in Sahara’s database and distributed to cluster
instances as part of a job’s workflow, or used by Sahara to access job
binaries. The storage of user credentials in Sahara's database represents a
security risk that can be avoided.
Proposed change
===============
A solution to using credentials for access to Swift objects is to generate a
Keystone trust between the user with access to those objects and a Sahara
proxy user. The trust would be established based on the user’s membership in
the project that contains the Swift objects. Using this trust the Sahara proxy
user could generate authentication tokens to access the Swift objects. When
access is no longer needed the trust can be revoked, and the proxy user
removed thus invalidating the tokens.
A proxy user will be created per job execution, and belong to a roleless
domain created by the stack administrator for the express purpose of
containing these new users. This domain will allow Sahara to create users as
needed for the purpose of delegating trust to access Swift objects.
Sahara will generate usernames and passwords for the proxy users when they are
created. These credentials will be used in conjuction with the trust to allow
Swift accress from the cluster instances.
General breakdown of the process:
1. On start Sahara confirms the existence of the proxy domain. If no proxy
domain is found and the user has configured Sahara to use one, then Sahara
will log an error and resume in backward compatibility mode.
2. When a new job that involves Swift objects is executed a trust is created
between the context user and a newly created proxy user. The proxy user's
name and password are created by Sahara and stored temporarily.
3. When an instance needs access to a Swift object it uses the proxy user's
credentials and the trust identifier to create the necessary authentication
token.
4. When the job has ended, the proxy user and the trust will be removed from
Keystone.
Detailed breakdown:
Step 1.
On start Sahara will confirm the existence of the proxy domain specified by
the administrator in the configuration file. If no domain can be found and the
user has configured Sahara to use one, then it will log an error and resume
in backward compatibility mode. The domain will be used to store proxy users
that will be created during job execution. It should explicitly have an SQL
identity backend, or another backend that will allow Sahara to create
users.(SQL is the Keystone default)
If the user has configured Sahara not to use a proxy domain then it will
fall back to using the backward compatible style of Swift authentication.
Requiring usernames and passwords for Swift access.
Step 2.
Whenever a new job execution is issued through Sahara a new proxy user will
be created in the proxy domain. This user will have a name generated based
on the job execution id, and a password generated randomly.
After creating the proxy user, Sahara will delegate a trust from the current
context user to the proxy user. This trust will grant a "Member" role by
default, but will allow configuration, and impersonate the context user. As
Sahara will not know the length of the job execution, the trust will be
generated with no expiration time and unlimited reuse.
Step 3.
During job execution, when an instance needs to access a Swift object, it
will use the proxy user's name and password in conjuction with the trust
identifier to create an authentication token. This token will then be used
to access the Swift object store.
Step 4.
After the job execution has finished, successfully or otherwise, the proxy
user and trust will be removed from Keystone.
A periodic task will check the proxy domain user list to ensure that none have
become stuck or abandoned after a job execution has been completed.
Alternatives
------------
Three alternatives have been discussed regarding this issue; using Swift’s
TempURL mechanism, encrypting the Swift credentials, and distributing tokens
from Sahara to the cluster instances.
Swift implements a feature named TempURL which allows the generation of
temporary URLs to allow public access to Swift objects. Using this feature
Sahara could create TempURLs for each Swift object that requires access and
then distribute these URLs to the cluster instances in the job workflow.
Although TempURLs would be low impact in terms of the work required to
implement they have a few major drawbacks for Sahara’s use case. When
creating a TempURL an expiration date must be associated with the URL. As
job lengths in Sahara cannot be predictively determined this would mean
creating indefinite expiration dates for the TempURLs. A solution to this
would be deleting the Swift object or changing the authentication identifier
associated with the creation of the TempURL. Both of these options present
implications that run beyond the boundaries of Sahara. In addition the
TempURLs would need to be passed to the instances as ciphertext to avoid
potential security breaches.
Another methodology discussed involves encrypting the Swift credentials and
allowing the cluster instances to decrypt them when access is required. Using
this method would involve Sahara generating a two part public/private key that
would be used to encrypt all credentials. The decryption, or private, part of
the key would be distributed to all cluster nodes. Upon job creation the Swift
credentials associated with the job would be encrypted and stored. The
encrypted credentials would be distributed to the cluster instances in the job
workflow. When access is needed to a Swift object, an instance would decrypt
the credentials using the locally stored key.
Encrypting the credentials poses a lower amount of change over using Keystone
trusts but perpetuates the current ideology of Sahara storing credentials for
Swift objects. In addition a new layer of security management becomes involved
in the form of Sahara needing to generate and store keys for use with the
credentials. This complexity adds another layer of management that could
instead be relegated to a more appropriate OpenStack service(i.e. Keystone).
Finally, another possibilty to achieve the removal of credentials from Sahara
would be for the main controller to distribute preauthorized tokens to the
cluster instances. These tokens would be used by the instances to validate
their Swift access. There are a few implementation problems with this approach
that have caused it to be abandoned by a previous version of this blueprint.
Keystone tokens have a default lifespan of one hour, this value can be
adjusted by the stack administrator. This implies that tokens must be updated
to the instances at least once an hour, possibly more frequently. The updating
process proves to add hindrances that are difficult to overcome. Update
frequency is one, but resource contention on the instances is another. A
further examination of this method shows that the updates to the Hadoop Swift
filesystem component will create a disonant design surrouding the injestion
of configuration data. In sum this methodology proves to be more fragile
than is acceptable.
Data model impact
-----------------
The job execution model currently stores username and password information in
a field that is a dictionary. There will be no changes to the model, but the
trust identifier will need to be stored in addition to the username and
password.
Once the credentials have been passed to the cluster the only values that
need be stored are the user id and trust identifier. These would need to be
used to destroy the trust and user after job execution. The proxy user's
password is only needed during the creation of the job execution and will
be distributed to the instances, but long term storage is not necessary.
REST API impact
---------------
The proxy usernames and trust identifiers should be sanitized from the
job execution output.
Other end user impact
---------------------
Users will no longer need to enter credentials when adding Swift data sources
to their jobs.
The user’s OpenStack credentials will need to have sufficient privileges to
access the Swift objects they add.
From the python-saharaclient, developers will no longer need to enter
credential_user or credential_pass when making a requests to create data
sources.
Keystone deployments that use LDAP backed domains will need to be configured
as recommended by the Keystone group, using domain based configurations. This
ensures that new domains created will be backed by SQL.
Deployer impact
---------------
A deployer will need to be aware of the Keystone configuration with respect
to the default identity backend. They will also need to create the proxy
domain and provide the Sahara service user with enough access to create new
users in that domain.
Developer impact
----------------
Developers will no longer need to pass credentials when creating data sources.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
For backward compatibility the username and password fields should be left in
the Swift data source forms and views, but they should allow the user to
enter blank data.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
* Michael McCune
Other contributors:
* Trevor McKay
Work Items
----------
* Domain detection and configuration option
* Proxy user creation/destruction
* Trust acquisition/revocation
* Workflow update
* Swift file system component update to use trust identifier
* Periodic proxy user removal task
* Documentation
* Tests
Dependencies
============
This feature will require the usage of Keystone v3 with the OS-TRUST mechanism
enabled.
The Horizon forms for Swift data sources will need to allow blank entries
for username and password.
The Hadoop Swift file system component will need to be updated as well.
Testing
=======
The current tests for Swift based objects will need to be modified to remove
the usage of username/password credentials. Otherwise these tests should prove
that the trust method is working properly.
Tests for situations where a user’s Keystone access do not permit permission
for the Swift objects they are adding should be implemented.
The Swift file system component will need it's tests modified to use
trust identifiers for scoping of the authentication tokens.
Documentation Impact
====================
The documentation for usage of Swift storage will need to have references to
the object credentials removed. Additionally there should be documentation
added about the impact of a user not having access to the Swift sources.
The proxy domain usage should be documented to give stack administrators a
clear understanding of Sahara's usage and needs. This should also note the
impact of Keystone configurations which do not provide default SQL identity
backends.
References
==========
Original bug report
https://bugs.launchpad.net/sahara/+bug/1321906
Keystone trust API reference
https://github.com/openstack/identity-api/blob/master/v3/src/markdown/identity-api-v3-os-trust-ext.md

154
specs/juno/error-handling-in-provisioning.rst
View File

@ -1,154 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==================================================
Improve error handling for provisioning operations
==================================================
https://blueprints.launchpad.net/sahara/+spec/error-handling-in-provisioning
Currently provisioning error handling is sprayed across the whole provisioning
code. This spec is to unify error handling and localize it in one place.
Problem description
===================
Currently we have two problems connected with error handling in provisioning
part:
1) The code incorrectly handles situations when cluster was deleted by user
during provisioning. In that case an arbitrary error might be raised in
many places.
2) The code performs rollback only in certain places, while it could be done
for any provisioning/scaling phase.
The following CR:
https://review.openstack.org/#/c/98556
mostly fixes issue #1, but it is full of duplicate code.
Proposed change
===============
The following solution is proposed instead which requires architectural
changes, but rather reliably fixes both problems:
1) For both cluster creation and scaling move error handling logic to the very
top functions inside ops.py file. Once exception is caught properly
process it:
a) if cluster object does not exists in DB, that means that user deleted
the cluster during provisioning; handle it and return
b) if cluster object exists, log it and perform rollback
2) Do not do any checks if cluster exists outside of ops.py, except places
where processing might hang indefinitely without the check.
We can employ the following rollback strategy:
For cluster creation: if anything went wrong, kill all VMs and move cluster
to the Error state.
For cluster scaling: that will be long. Cluster scaling has the following
stages:
1) decommission unneeded nodes (by plugin)
2) terminate unneeded nodes and create a new ones if needed (by engine). Note
that both scaling up and down could be run simultaneously but in
different node groups.
3) Configure and start nodes (by plugin)
My suggestion what to do if an exception occurred in the respective stage:
1) move cluster to Error state
2) kill unneeded nodes (finish scale down). Also kill new nodes, if they were
created for scale up.
3) move cluster to Error state
In cases #1 and #3 it is dangerous to delete not decommissioned or not
configured nodes as this can lead to data loss.
Alternatives
------------
Keep supporting current code. It is not elegant but works good enough.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
Data provisioning logic will be changed a lot. This could lead to behavior
change in case of errors on different stages.
Deployer impact
---------------
None
Developer impact
----------------
Provisioning engine API will be extended with "rollback_cluster" method.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
alazarev
Other contributors:
dmitrymex
Work Items
----------
1) Implement change
2) Test that cluster provisioning and rollback works on all feature matrix we
have.
Dependencies
============
None
Testing
=======
Provisioning could be tested manually and by CI.
It is much harder to test rollback. Even current code is not tested well (e.g.
https://bugs.launchpad.net/sahara/+bug/1337006).
Documentation Impact
====================
None
References
==========
None

118
specs/juno/move-rest-samples-to-docs.rst
View File

@ -1,118 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==============================================
Move Sahara REST API samples from /etc to docs
==============================================
https://blueprints.launchpad.net/sahara/+spec/move-rest-samples-to-docs
Initially this idea was raised during discussions about
moving/releasing/versioning of Sahara's subprojects.
REST samples are slightly outdated and common documentation
is a good place to have them there to keep them up-to-date.
Problem description
===================
Today REST API samples are outdated and don't reflect current state of
changes made in Sahara api since Havana release. Also it's not obvious to
find those samples in Sahara sources. The goal is to update current state
of samples and move it to http://docs.openstack.org/
Proposed change
===============
Create a new page in docs::
sahara/doc/source/restapi/rest_api_samples.rst
Move all JSON examples from sahara/etc/rest-api-samples to the new page.
Simple description for each example should be added before JSON code blocks.
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
Examples won't be found in the sahara/etc dir any longer.
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary Assignee:
Alexander Ignatov
Work Items
----------
The following steps should be done:
* Move REST samples from sahara/etc to a new page in docs
* Update samples in docs according to current state of Sahara api
* Remove sahara/etc/rest-api-samples directory in Sahara sources
Dependencies
============
None
Testing
=======
None
Documentation Impact
====================
New page with information about REST samples will appear in the Sahara docs.
References
==========
None

13
specs/juno/redirects
View File

@ -1,13 +0,0 @@
implemented/anti-affinity-via-server-groups.rst anti-affinity-via-server-groups.rst
implemented/append-to-remote-file.rst append-to-remote-file.rst
implemented/cdh-plugin.rst cdh-plugin.rst
implemented/ceilometer-integration.rst ceilometer-integration.rst
implemented/cluster-persist-sahara-configuration.rst cluster-persist-sahara-configuration.rst
implemented/cluster-secgroups.rst cluster-secgroups.rst
implemented/edp-move-examples.rst edp-move-examples.rst
implemented/edp-refactor-job-manager.rst edp-refactor-job-manager.rst
implemented/edp-spark-job-type.rst edp-spark-job-type.rst
implemented/edp-spark-standalone.rst edp-spark-standalone.rst
implemented/edp-swift-trust-authentication.rst edp-swift-trust-authentication.rst
implemented/error-handling-in-provisioning.rst error-handling-in-provisioning.rst
implemented/move-rest-samples-to-docs.rst move-rest-samples-to-docs.rst

8
specs/juno_idx.rst
View File

@ -1,8 +0,0 @@
Juno specs
^^^^^^^^^^
.. toctree::
:glob:
:maxdepth: 1
juno/*

151
specs/kilo/add-lib-subset-cm-api.rst
View File

@ -1,151 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==============================
Add CM API Library into Sahara
==============================
https://blueprints.launchpad.net/sahara/+spec/add-lib-subset-cm-api
This specification proposes to add CM API library into Sahara so that the
Cloudera plugin will not have to depend on a third party library support.
Problem description
===================
Now the Cloudera plugin is depending on a third party library cm_api (by
Cloudera). This library is not python3 compatible, and Cloudera has no
resource to fix it. Therefore, it is hard to put this library into
requirements.txt file. To fix this issue, we plan to implement a subset of
CM APIs and put them into Sahara project, so that the Cloudera plugin will not
depend on a third-party library, and we can enable this plugin by default.
Proposed change
===============
Now the CM APIs used in Cloudera plugin include (maybe not all included):
* ApiResource.create_cluster
* ApiResource.get_cluster
* ApiResource.get_all_hosts
* ApiResource.delete_host
* ApiResource.get_cloudera_manager
* ClouderaManager.create_mgmt_service
* ClouderaManager.hosts_start_roles
* ClouderaManager.get_service
* ApiCluster.start
* ApiCluster.remove_host
* ApiCluster.create_service
* ApiCluster.get_service
* ApiCluster.deploy_client_config
* ApiCluster.first_run
* ApiService.create_role
* ApiService.delete_role
* ApiService.refresh
* ApiService.deploy_client_config
* ApiService.start
* ApiService.restart
* ApiService.start_roles
* ApiService.format_hdfs
* ApiService.create_hdfs_tmp
* ApiService.create_yarn_job_history_dir
* ApiService.create_oozie_db
* ApiService.install_oozie_sharelib
* ApiService.create_hive_metastore_tables
* ApiService.create_hive_userdir
* ApiService.create_hive_warehouse
* ApiService.create_hbase_root
* ApiService.update_config
* ApiServiceSetupInfo.add_role_info
* ApiRole.update_config
Those APIs are what we need to implement in our CM APIs. We can create a
directory client in plugin cdh directory, and put the lib files in this
directory.
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
Deployers will no longer need to install cm_api package anymore.
Developer impact
----------------
When new version of CDH is released, if it is incompatible with current used
client, or use some new APIs, developer may need to update the client when
adding support for new CDH release.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
ken chen
Other contributors:
ken chen
Work Items
----------
The work items will include:
* Add a directory client in cdh plugin directory, and put lib files under
this directory.
* Change all current cm_api imports into using the new client.
Dependencies
============
None
Testing
=======
Sahara Integration test for CDH plugin is enough.
Documentation Impact
====================
Documents about CDH plugin prerequisites and enabling should be modified, for
cm_api is not required any more.
References
==========
https://pypi.python.org/pypi/cm-api/8.0.0

120
specs/kilo/add-more-cdh-services.rst
View File

@ -1,120 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Add More Services into CDH plugin
==========================================
https://blueprints.launchpad.net/sahara/+spec/add-cdh-more-services
This specification proposes to add below services into CDH plugins:
Flume, Sentry, Sqoop, SOLR, Key-Value Store Indexer, and Impala.
Those services can be added into CDH plugin by using CM API first_run to start
them, which can save much effort to prepare and start those services one by
one.
Problem description
===================
Now services supported in Sahara CDH plugin is still limited. We want to add
more services ASAP. They are Flume, Sentry, Sqoop, SOLR, Key-Value Store
Indexer, and Impala.
Proposed change
===============
Since we plan to use first_run to prepare and start those services, we will
not need to call other CM APIs for those services in start_cluster() method.
The implementation will need below changes on codes for each service:
* Add process names of the service in some places.
* Add service or process configuration, and network ports to open.
* Add service validation.
* Modify some utils methods, like get_service, to meet more services.
* Some other changes for a few specific services if needed.
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
ken chen
Other contributors:
ken chen
Work Items
----------
The work items will be:
* Change python codes in sahara/sahara/plugins/cdh.
* Add more service resource files in sahara/sahara/plugins/cdh/resources.
* Test and evaluate the change.
Dependencies
============
None
Testing
=======
Use an integration test to create a cluster.
Documentation Impact
====================
None
References
==========
None

122
specs/kilo/add-service-test-in-integration.rst
View File

@ -1,122 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Add check service test in integration test
==========================================
https://blueprints.launchpad.net/sahara/+spec/add-service-test-in-integration
This specification proposes to add check services tests in integration test
for CDH plugins.Currently we have added zookeeper,HBase,Flume,Sentry,Sqoop,
SOLR,Key-Value Store Indexer, and Impala services.
Problem description
===================
Currently we have enabled many new services in CDH plugin.And we want to
increase the coverage of the test cases.So we plan to add test cases in the
integration test, which will check the availability of those services by using
simple scripts like we did in map_reduce_testing.
Proposed change
===============
We plan to write test cases like the way we did in map_reduce_testing. First
copy the shell script to the node,then run this script, the script will run
basic usage of the services.
The implementation will need below changes on codes for each service:
* Add new cluster template (including all services process) in
test_gating_cdh.py
* Add check_services.py (check all services) to check all basic usage of
services
* Add shell scripts (check all services)
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
huichun lu
Other contributors:
huichun lu
Work Items
----------
The work items will be:
* Add python codes in ``sahara/sahara/tests/integration/tests/gating/
test_cdh_gating.py``.
* Add check services scripts files in ``sahara/sahara/tests/integrations/
tests/resources``.
* Add check_services.py in ``sahara/sahara/tests/integration/tests/``.
Dependencies
============
None
Testing
=======
Use an integration test to create a cluster.
Documentation Impact
====================
None
References
==========
None

156
specs/kilo/add-timeouts-for-polling.rst
View File

@ -1,156 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Add timeouts for infinite polling for smth
==========================================
https://blueprints.launchpad.net/sahara/+spec/add-timeouts-for-polling
We have infinite polling-processes for smth in sahara code.
It would be nice to add timeouts for its execution
Problem description
===================
When creating a cluster, cluster's status may be stuck in Waiting and will
always await network if the network configuration is wrong. We can add
configurable timeouts for all possible polling processes.
Proposed change
===============
Here you can find list of all polling processes in Sahara code:
For service module:
* volumes._await_attach_volumes
* volumes._create_attach_volume
* volumes._detach_volume
* engine._wait_until_accessible
* engine._await_networks
* direct_engine._await_active
* direct_engine._await_deleted
* edp.job_manager.cancel_job
For utils module:
* openstack.heat.wait_stack_completion
For cdh plugin:
* cloudera_utils.await_agents
* plugin_utils.start_cloudera_manager
For hdp plugin:
* _wait_for_async_request for both plugin versions
* wait_for_host_registrations for both plugin versions
* decommission_cluster_instances for 2.0.6 versionhandler
For spark plugin:
* scaling.decommission_dn
For vanilla plugin:
* hadoop2.scaling._check_decommission
* hadoop2.await_datanodes
* v1_2_1.scaling.decommission_dn
* v1_2_1.versionhandler._await_datanodes
Proposed change would consists of following steps:
* Add new module polling_utils in sahara/utils where would register
new timeouts options for polling processes from service and utils modules.
Also it would consist with a specific general util for polling.
As example it can be moved from
https://github.com/openstack/sahara/blob/master/sahara/utils/general.py#L175.
* Add new section in sahara.conf.sample which would consist with all
timeouts.
* All plugin specific options would be related only with this plugin and also
would be configurable. Also user would have ability to configure all plugin
specific options during cluster template creation.
Alternatives
------------
None
Data model impact
-----------------
This change doesn't require any data models modifications.
REST API impact
---------------
This change doesn't require any REST API modifications.
Other end user impact
---------------------
User would have ability to configure all timeouts separately. Some options
would be configurable via sahara.conf.sample, other would be configurable
from plugin during cluster template creation.
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
During cluster template creation user would have ability to configure
plugin specific options from UI.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
vgridnev
Work Items
----------
* Add general polling util to sahara/utils/polling_utils
* Apply this changes to all plugins and sahara engines.
Dependencies
============
Depends on current Openstack Data Processing Requirements.
Testing
=======
this change would require to add unit tests. Also this change would be tested
manually.
Documentation Impact
====================
Required to document this feature in sahara/userdoc/configuration.guide.
References
==========
[1] https://bugs.launchpad.net/sahara/+bug/1402903
[2] https://bugs.launchpad.net/sahara/+bug/1319079

183
specs/kilo/auth-policy.rst
View File

@ -1,183 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
============================
Authorization Policy Support
============================
https://blueprints.launchpad.net/sahara/+spec/auth-policy
Openstack components are supposed to check user privileges for performed
actions. Usually these checks are role-based. See
http://docs.openstack.org/developer/keystone/architecture.html#approach-to-authorization-policy.
Sahara need to support policies too.
Problem description
===================
OpenStack administrators may want to tune authorization policy for Sahara.
There should be a way to restrict some users to perform some of Sahara
operations.
Proposed change
===============
Add auth check for all Sahara API endpoints. This could be done in the same
way as in other openstack component. There is "policy" module in Oslo library
that may do all underlying work.
Proposed content of the policy file:
.. sourcecode:: json
{
"context_is_admin": "role:admin",
"admin_or_owner": "is_admin:True or project_id:%(project_id)s",
"default": "rule:admin_or_owner",
"clusters:get_all": "",
"clusters:create": "",
"clusters:scale": "",
"clusters:get": "",
"clusters:delete": "",
"cluster-templates:get_all": "",
"cluster-templates:create": "",
"cluster-templates:get": "",
"cluster-templates:modify": "",
"cluster-templates:delete": "",
"node-group-templates:get_all": "",
"node-group-templates:create": "",
"node-group-templates:get": "",
"node-group-templates:modify": "",
"node-group-templates:delete": "",
"plugins:get_all": "",
"plugins:get": "",
"plugins:get_version": "",
"plugins:convert_config": "",
"images:get_all": "",
"images:get": "",
"images:register": "",
"images:unregister": "",
"images:add_tags": "",
"images:remove_tags": "",
"job-executions:get_all": "",
"job-executions:get": "",
"job-executions:refresh_status": "",
"job-executions:cancel": "",
"job-executions:delete": "",
"data-sources:get_all": "",
"data-sources:get": "",
"data-sources:register": "",
"data-sources:delete": "",
"jobs:get_all": "",
"jobs:create": "",
"jobs:get": "",
"jobs:delete": "",
"jobs:get_config_hints": "",
"jobs:execute": "",
"job-binaries:get_all": "",
"job-binaries:create": "",
"job-binaries:get": "",
"job-binaries:delete": "",
"job-binaries:get_data": "",
"job-binary-internals:get_all": "",
"job-binary-internals:create": "",
"job-binary-internals:get": "",
"job-binary-internals:delete": "",
"job-binary-internals:get_data": ""
}
Separating Sahara users and operators could be the next step.
Alternatives
------------
None.
Data model impact
-----------------
None.
REST API impact
---------------
None.
Other end user impact
---------------------
None.
Deployer impact
---------------
None.
Developer impact
----------------
Adding new API will require changing policy rules.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
alazarev (Andrew Lazarev)
Work Items
----------
* Add policy.py from oslo
* Add config options to control policy file and settings
* Add policy check to all API calls
* Add unit tests
* Add documentation
Dependencies
============
* Policy module in Oslo.
Testing
=======
* Unit tests
* Manual testing
Documentation Impact
====================
* Feature need to be documented
References
==========
* http://docs.openstack.org/developer/keystone/architecture.html#approach-to-authorization-policy
* http://docs.openstack.org/developer/keystone/api/keystone.openstack.common.policy.html
* http://docs.openstack.org/developer/keystone/configuration.html#keystone-api-protection-with-role-based-access-control-rbac

122
specs/kilo/cdh-hbase-support.rst
View File

@ -1,122 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
CDH HBase Support
==========================================
https://blueprints.launchpad.net/sahara/+spec/cdh-hbase-support
This specification proposes to add HBase support for CDH Plugin in Sahara.
Problem description
===================
There is no HBase support in current cdh plugin, but Cloudera Manager
supports to install this service in a cluster. HBase is a non-relational,
distributed database model and can provide BigTable-like capabilities for
Hadoop. This service should be supported in Sahara cdh plugin.
Proposed change
===============
The implementation will support CDH 5.0.0.
Support Features:
* Install HBase processes in a CDH cluster using cm-api
* Zookeeper must be selected and launched in a cluster first
* Support HMaster and Multiple HRegion processes in a cluster
* Most Configuration Parameters Support in a CDH cluster
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
The end users need to select HMaster and HRegion process in node group
templates.
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
It will be required to install the necessary HBase package in the cdh image.
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
lu huichun
Other contributors:
weiting-chen
Work Items
----------
The work items can be divided to several parts:
* Investigate HBase service in a CDH cluster via Cloudera Manager(CM)
* Leverage CM-API Client to call functions to install Zookeep via CM
* Test and Evaluate the Concept
* Implement Source Code in Sahara cdh plugin
* Test the Code
Dependencies
============
Zookeeper process must be installed first in the cluster. HBase needs to use
Zookeeper service.
Testing
=======
Writing Unit Tests to basically test the configuration. It also required to
have an integration test with the cluster creation.
Documentation Impact
====================
Add HBase in the list and add more information about the configuration.
References
==========
* [1] http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-0-0/

166
specs/kilo/cdh-version-management.rst
View File

@ -1,166 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
============================================
Better Version Management in Cloudera Plugin
============================================
https://blueprints.launchpad.net/sahara/+spec/cdh-version-management
This specification proposes to manage different versions of CDH in Cloudera
plugin in a better solution.
Problem description
===================
Current situation of CDH version management in CDH plugin:
* We do not have version control for CDH plugin at all.
* Due to backward support requirement, all existing codes need to support
CDH version from 5.0.0 to latest (currently it is 5.3.0).
This will cause below issues:
* When new packages are released, we do not know whether our plugin still
can work. In fact nobody can ensure that.
* Our work have to be based on a non-static environment.
* Backward-compatibility is desired, but it is not ensured at all. For
example, we are not sure tomorrow whether a new released CDH package will
be compatible with old plugin codes or configuration files.
* CI-test results are not stable, so we cannot always turn it into voting.
* If new released package version bring issues, it can block all developers
even if they do not work on this version.
* When we do not want to support some obsolete versions, we cannot easily
remove it.
Proposed change
===============
* Add package version constraints, and give a key packages version list to
support CDH5.3.0 (or 5.2.0), and base our development only on this.
* Freeze the package to prevent later CDH release come in.
* Add sub-directories in plugin/cdh, like 5_3_0 for CDH5.3.0 to support
different versions of CDH. As we did for vanilla and hdp plugins.
* We also need to do some modifications in sahara-image-elements project to
support different versions of CDH image build.
* We need to change sahara-ci-config project to break current cdh tests into
several tests for different CDH versions. All new added versions should not
break old version tests. For example, current
gate-sahara-nova-direct-cdh_ubuntu-aio will be separated into
sahara-nova-direct-cdh_5.0_ubuntu-aio and
sahara-nova-direct-cdh_5.3.0_ubuntu-aio.
* Break sahara/tests/integration/tests/gating/test_cdh_gating.py into
several files, like test_cdh5_0_0_gating.py and test_cdh5_3_0_gating.py to
support different CDH versions.
* We need not ensure backward compatibility. E.g., CDH5.3.0 codes are not
required to work for CDH5.0.0. We may add some features and codes only for
later version of CDH, which was not supported by former CDH version.
* If we want to add more CDH versions in the future, we need to open a BP to
do this. For example, if we want to support CDH 5.4.0 in the future, below
works are required:
* Add a directory 5_3_0 including codes to support CDH 5.3.0 in
plugin/cdh.
* Modify scripts in sahara/image-elements/elements/hadoop_cloudera, to
add functions to install packages for CDH 5.4.0, while still keeping
functions installing packages for former CDH versions.
* Add a test_cdh5_4_0_gating.py in sahara/tests/integration/tests/gating
directory for integration test.
* Add a new ci-test item in sahara-ci-config, like
gate-sahara-nova-direct-cdh_5.4.0_ubuntu-aio. We can set this item as
non-voting at the beginning, and after it is tuned and verified ok, we
set it back to voting.
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
ken chen
Other contributors:
ken chen
Work Items
----------
The work items will include:
* Change directory structure in sahara/sahara/plugins/cdh to add 5_3_0 and
5_0_0 for CDH5.0.0 and 5.3.0 (We will support those two versions as the
first step).
* Retain common codes for each version in the original place, and move
version specific codes and files into their own directory.
* Add codes in sahara-image-elements/elements/hadoop-cloudera to support
installing different package groups for different CDH versions.
* Add different test items in sahara/tests/integration/tests/gating directory
to support different CDH versions
* Add item in sahara-ci-configs project for different CDH versions. At first
we mark it as non-voting. After it is verified, we can mark it as voting.
* Test and evaluate the change.
Dependencies
============
None
Testing
=======
Create clusters of different CDH versions one by one, and do integration tests
for each of them.
Documentation Impact
====================
None
References
==========
None

123
specs/kilo/cdh-zookeeper-support.rst
View File

@ -1,123 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
CDH Zookeeper Support
==========================================
https://blueprints.launchpad.net/sahara/+spec/cdh-zookeeper-support
This specification proposes to add Zookeeper support for CDH Plugin in Sahara.
Problem description
===================
Currently, Zookeeper isn't supported in the cdh plugin. Zookeeper is a
centralized service to provide functions like maintaining configuration,
distributed synchronization, and providing group services. It's important to
have a Zookeeper to prevent data loss and to avoid a single point failure(SPoF)
in a cluster. It has become a basic service to deploy a hadoop cluster in CDH
environment.
Proposed change
===============
The implementation will support CDH 5.0.0.
Support Features:
* Install Zookeeper Service in a CDH cluster
* Support to run Standalone Zookeeper in a cluster
* Support to run Replicated Zookeepers(Multiple Servers) in a cluster
* To have an option to select Zookeeper in the node group templates
* Most Configuration Parameters Support in a CDH cluster
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
The end users need to select Zookeeper process in their node group templates.
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
It will be required to put the necessary packages in the cdh image.
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
ken chen
Other contributors:
weiting-chen
Work Items
----------
The work items can be divided to several parts:
* Investigate Zookeeper service in a CDH cluster via Cloudera Manager(CM)
* Leverage CM-API Client to call functions to install Zookeep via CM
* Test and Evaluate the Concept
* Implement Source Code in Sahara cdh plugin
* Test the Code
Dependencies
============
None
Testing
=======
Writing Unit Tests to basically test the configuration. It is also required
to have an integration test with the cluster creation.
Documentation Impact
====================
Add Zookeeper in the list and add more information about the configuration.
References
==========
* [1] http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-0-0/

243
specs/kilo/default-templates.rst
View File

@ -1,243 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=================================
Default templates for each plugin
=================================
Blueprint: https://blueprints.launchpad.net/sahara/+spec/default-templates
In order to create a basic cluster for any plugin, a user currently has to
go through several steps to create an appropriate set of node group and
cluster templates. We believe that set of actions should be captured in a
default set of templates (per plugin) and loaded into the database ahead of
time so that users do not need to repeat some of the most basic steps to
provision a simple cluster.
Problem description
===================
Creating even a basic cluster currently requires several steps before the
user is able to launch their cluster. In an effort to reduce the amount of
effort and time to get a simple cluster running, we propose a set of default
templates for each plugin that will be pre-loaded and available for use.
Other potential issues/answers:
1) How do we make the set of templates available for all users/tenants?
Today, any given template is only read/writable by one tenant.
A) Implement ACL support for templates (SLukjanov plans to write the spec
for this). Having proper ACL support in place will allow us to manage
access for read/write across all tenants.
ACL support is not available yet. For the time being, templates will be
added per-tenant.
2) Do we allow editing of default templates?
I don't think we should allow editing of the default templates since they
are shared among all tenants. The flow to "edit" one would be to make a
copy of the template and work from there. I propose that each template will
be stored with a flag in the database that identifies each default template
as being a default template so that we can enforce that users cannot change
the default templates.
3) How do we avoid uploading the same default templates each time at startup
while still allowing for them to be updated as necessary?
We could use a numbering system in the template file names to indicate the
version number and store that in the database (perhaps instead of a boolean
flag indicating that the template is a default template,
we could store an int that is the version number). At startup time,
we would go through all of the template files for each plugin and compare
the version numbers to the version numbers that are stored in the database.
Sqlalchemy will detect when fields have changed and set the "updated at" field
accordingly. Therefore, we can simply attempt to update existing templates
whenever the script is run. If the template matches what is in the database,
there will be no update.
4) How do we make a json default cluster template reference a json default node
group template since we won't know the node group template IDs?
A) CLI util will operate them one-by-one starting with node group templates and
then cluster templates. In addition, we could create a few example jobs that
could be used for health check.
Proposed change
===============
1) Create sets of default template json files for each plugin.
A set of default templates will consist of node group templates and/or
cluster templates. Whether or not a template file defines a node group
template or a cluster template will be determined based on the presence of
required fields specified in the Sahara JSON validation code. Node group
templates require `flavor_id` and `node_processes`, so presence of these
fields implicitly identify a template as a node group template. If it is
not a node group template, it is a cluster template. This identification
avoids a naming scheme or some other kind of extra labeling to identify a
template type.
Default templates distributed with Sahara will be located in
`sahara/plugins/default_templates/`. The code that processes the default
templates should use this path as a default starting point but should allow
a different starting point to be specified. It should make no assumptions
about the directory structure beyond the following:
* All of the template files in a particular directory should be treated as
a set, and cluster templates may reference node group templates in the
same set by name.
* Directories may be nested for logical organization, so that a plugin may
define default template sets for each version. Therefore, the code should
recurse through subdirectories by default but it should be possible to
specify no recursion.
This design will allow code to process default templates decoupled from
explicit knowledge of Sahara plugins, plugin versions, or the structure of
plugin directories. The ability to specify a different starting point
will allow a user to process particular template sets if the entire set
is not desired (only some plugins enabled, for instance), or if an alternate
set at a different location is desired. In short, it keeps the processing
general and simple.
In practice, the directories under `sahara/plugins/default_templates` will
be named for plugins, and subdirectories will be created for different
versions as appropriate.
2) Add a CLI util that can be executed by cron with admin credentials to
create/update existing default templates. This utility needs to be able to
take some placeholders like "flavor" or "network" and make the appropriate
substitutions (either from configs or via command line args) at runtime.
The cron job can be optional if we want to force any updates to be
triggered explicitly.
The CLI will take an option to specify the starting directory (default
will be `sahara/plugins/default_templates`).
The CLI will take an option to disable recursion through subdirectories
(default will be to recurse).
At a minimum, the CLI will provide a command to create or update default
templates with processing beginning at the designated starting directory.
The CLI should use the "plugins" configuration parameter from the [database]
section to filter the templates that are processed. If the "plugin-name"
field of a template matches a plugin name in the "plugins" list, it will
be processed. If the "plugins" list is empty, all templates will be
processed. It should be possible to override the "plugins" configuration
from the command line with a "--plugin-name" option.
If there is an error during updating or creating templates in a particular
set, the CLI should attempt to undo any modifications or creations that
were done as part of that set.
The CLI should also provide a mechanism for deleting default templates,
since the `is_default` field will prevent that, should an admin for
some reason desire to remove default templates. This can be a simple
operation that will remove a single default template by ID. It is not
likely to be used very often.
Alternatives
------------
1) The loading process could be done via the REST API if we wanted to have
some sort of external process that manages the default templates. That might
require changing the API a bit to add endpoints for managing the default
templates and seems like a fair bit of unnecessary work since the management of
default templates should be something done only within Sahara.
2) Add a hook, possibly in plugins/base:PluginManager for
"load_default_templates". This method would be responsible for triggering
the loading of the defaults at startup time.
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Other end user impact
---------------------
End users will see the default templates show up just like any other
template that they may have created.
Deployer impact
---------------
N/A
Developer impact
----------------
N/A
Sahara-image-elements impact
----------------------------
N/A
Sahara-dashboard / Horizon impact
---------------------------------
N/A
The default templates will show-up in the UI and look like regular templates.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
croberts
Secondary assignee:
tmckay
Work Items
----------
1) Come up with a set of default templates for each plugin. These will
probably be json formatted files.
2) Come up with some sort of mechanism to load the templates or ensure that
they are already loaded when Sahara starts-up.
3) Update the Sahara documentation.
Dependencies
============
1) Implementation of the ACL for templates (spec still TBD). This will let
us give all users read access to the default templates while still possibly
allowing admins to edit the templates.
Testing
=======
Ideally, tests will be added to ensure that a functioning cluster can be
started based on each of the default template sets. If that is determined
to be too time-consuming per-run, then tests to ensure the validity of each set
of templates may be sufficient.
Documentation Impact
====================
The Sahara documentation should be updated to note that the default
templates are available for use. Additionally, any future plugins will be
expected to provide their own set of default templates.
References
==========
N/A

112
specs/kilo/drop-hadoop-2.3-support.rst
View File

@ -1,112 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
================================================
Remove support of Hadoop 2.3.0 in Vanilla plugin
================================================
https://blueprints.launchpad.net/sahara/+spec/drop-hadoop-2-3-support
At the ATL Design summit it was decided to have 1 OpenStack release cycle
management for support of previous Hadoop versions in Vanilla plugin.
So it's time to remove 2.3.0 plugin.
Problem description
===================
Current Sahara code contains Vanilla Hadoop 2.3.0 in sources.
Proposed change
===============
The proposed change is to remove all Hadoop 2.3 and any related mentions
from Sahara code and subprojects.
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
Users will not be able to deploy Hadoop 2.3
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
Hadoop 2.3 related elements should be removed
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary Assignee:
Sergey Reshetnyak (sreshetniak)
Other Assignees:
Sergey Lukjanov (slukjanov) - dib cleanup
Work Items
----------
* Remove Vanilla plugin 2.3.0 from Sahara sources
* Remove DIB stuff related to 2.3.0
* Clear unit/integration tests
* Replace EDP examples with newest version of hadoop-examples
* Documentation changes
Dependencies
============
None
Testing
=======
None
Documentation Impact
====================
* Documentation must to be updated accordingly
References
==========
None

236
specs/kilo/edp-add-hbase-sharelib.rst
View File

@ -1,236 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===============================================
Add a common HBase lib in hdfs on cluster start
===============================================
https://blueprints.launchpad.net/sahara/+spec/edp-add-hbase-lib
HBase applications written in Java require JARs on the HBase classpath.
Java actions launched from Oozie may reference JARs stored in hdfs
using the `oozie.libpath` configuration value, but there is no
standard HBase directory location in hdfs that is installed with Oozie.
Users can build their own HBase directory in hdfs manually from a cluster
node but it would be convenient if Sahara provided an option to build
the directory at cluster launch time.
Problem description
===================
HBase applications written in Java require the Hbase classpath. Typically,
a Java program will be run this way using /usr/bin/hbase to get the classpath::
java -cp `hbase classpath`:MyHbaseApp.jar MyHBaseApp
Java jobs launched from EDP are Oozie actions, and there is no way to set
an extra classpath value. Instead, the Oozie solution to this problem is
to create a directory of JAR files in hdfs and then set the
`oozie.libpath` configuration property on the job to that location.
This causes Oozie to make all of the jars in the directory available to the
job.
Sahara currently supports setting the `oozie.libpath` configuration on a job
but there is no existing collection of HBase JARs to reference. Users can log
into a cluster node and build an HBase directory in hdfs manually using bash or
Python. The steps are relatively simple:
* Run the ``hbase classpath`` command to retrieve the classpath as a string
* Separate the string on the **:** character
* Prune away all paths that do not end in **.jar**
* Upload all of the remaining paths to the designated directory in hdfs
However, it would be relatively simple for Sahara to do this optionally
at cluster creation time for clusters that include HBase services.
Note that the same idea of a shared hdfs directory is used in two
different but related ways in Oozie:
* the `Oozie sharelib` is a pre-packaged collection of JARs released and
supported as part of Oozie and referenced from a job by
setting the `oozie.use.system.libpath` configuration parameter to `True`.
Sahara already sets this option for all Ooozie-based jobs.
The official Oozie sharelib changes over time, and Oozie uses a timestamp
naming convention to support upgrades, multiple versions, etc.
* the ability to create an hdfs directory containing JAR files and reference
it from a job with the `oozie.libpath` configuration parameter is open to
anyone. This is what is being proposed here. This change in no way touches
the official `Oozie sharelib`. If Oozie ever adds HBase JARs to the
system sharelib, we probably will no longer need this feature.
Proposed change
===============
Create a class that can be shared by any provisioning plugins that support
installing the HBase service on a cluster. This class should provide a
method that runs remote commands on a cluster node to:
* Run the ``hbase classpath`` command to retrieve the classpath as a string
* Separate the string on the **:** character
* Prune away all paths that do not end in **.jar**
* Upload all of the remaining paths to pre-determined directory in hdfs
A code sample in the reference section below shows one method of doing this
in a Python script, which could be uploaded to the node and executed via
remote utilities.
The HBase hdfs directory can be fixed, it does not need to be configurable. For
example, it can be "/user/sahara-hbase-lib" or something similar. It should be
readable by the user that runs Hadoop jobs on the cluster. The EDP engine can
query this class for the location of the directory at runtime.
An option can be added to cluster configs that controls the creation of this
hdfs library. The default for this option should be True. If the config option
is True, and the cluster is provisioned with an HBase service, then the hdfs
HBase library should be created after the hdfs service is up and running and
before the cluster is moved to "Active".
A job needs to set the `oozie.libpath` value to reference the library.
Setting it directly presents a few problems:
* the hdfs location needs to be known to the end user
* it exposes more "Oozie-ness" to the end user. A lot of "Oozie-ness" has
leaked into Sahara's interfaces already but there is no reason to
add to it.
Instead, we should use an `edp.use_hbase_lib` boolean configuration
parameter to specify whether a job should use the HBase hdfs library. If this
configuration parameter is True, EDP can retrieve the hdfs location
from the utility class described above and set `oozie.libpath` accordingly.
Note that if for some reason an advanced user has already set `oozie.libpath`
to a value, the location of the HBase lib should be added to the value (which
may be a comma-separated list).
Alternatives
------------
* Do nothing. Let users make their own shared libraries.
* Support Oozie Shell actions in Sahara.
Shell actions are a much more general feature under consideration
for Sahara. Assuming they are supported, a Shell action provides
a way to launch a Java application from a script and the classpath
can be set directly without the need for an HBase hdfs library.
Using a Shell action would allow a user to run a script. In a script
a user would have complete control over how to launch a Java application
and could set the classpath appropriately.
The user experience would be a little different. Instead of just writing
a Java HBase application and launching it with `edp.use_hbase_lib` set to
True, a user would have to write a wrapper script and launch that as a
Shell action instead.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
We may want a simple checkbox option on the UI for Java actions
to set the `edp.use_hbase_libpath` config so that users don't need to
add it by hand.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
huichun
Other contributors:
tmckay
Work Items
----------
Dependencies
============
Testing
=======
An EDP integration test on a cluster with HBase installed would be great test
coverage for this since it involves cluster configuration.
Unit tests can verify that the oozie.libpath is set correctly.
Documentation Impact
====================
We need to document how to enable creation of the shared lib at cluster
creation time, and how to configure a job to reference it
References
==========
Here is a good blog entry on Oozie shared libraries in general.
http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
Here is a simple script that can be used to create the lib::
#!/usr/bin/python
import sys
import os
import subprocess
def main():
subprocess.Popen("hadoop fs -mkdir %s" % sys.argv[1], shell=True).wait()
cp, stderr = subprocess.Popen("hbase classpath",
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE).communicate()
paths = cp.split(':')
for p in paths:
if p.endswith(".jar"):
print(p)
subprocess.Popen("hadoop fs -put %s %s" % (os.path.realpath(p),
sys.argv[1]),
shell=True).wait()
if __name__ == "__main__":
main()

247
specs/kilo/edp-add-oozie-shell-action.rst
View File

@ -1,247 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=====================================
[EDP] Add Oozie Shell Action job type
=====================================
https://blueprints.launchpad.net/sahara/+spec/add-edp-shell-action
Oozie shell actions provide a great deal of flexibility and will
empower users to easily customize and extend the features of
Sahara EDP as needed. For example, a shell action could be used
to manage hdfs on the cluster, do pre or post processing for
another job launched from Sahara, or run a data processing job
from a specialized launcher that does extra configuration not
otherwise available from Sahara (ie, setting a special classpath
for a Java job).
Problem description
===================
Today if a user finds a limitation in Sahara EDP that prevents them
from performing desired processing they have a few choices:
* Log into the cluster nodes with ssh using the specified keypair
and perform processing by hand. This may not be available to all
users.
* Write a Java application to do the custom processing and run it
as a Java job in EDP. This can work, however we as developers know
that Java is not often used as a scripting language; it's a little
too heavyweight and not everyone knows it.
* Modify the Sahara source. A savvy user might extend Sahara EDP
themselves to get the desired functionality. However, not everyone
is a developer or has the time to understand Sahara enough to do this.
* Submit a bug or a blueprint and wait for the Sahara team to address it.
However, the existence of shell actions would empower users to easily
solve their own problems. With a shell action, a user can bundle files
with a script written in bash, Python, etc and execute it on the cluster.
Here is a real-world example of a case that could be easily solved
with a shell action:
https://blueprints.launchpad.net/sahara/+spec/edp-add-hbase-lib
In the above blueprint we are calling for additional features in Sahara
as a convenience for users, but with a shell action a user could solve this
problem on their own. A simple bash script can be written to launch a Java
application like this:
.. code::
#!/bin/bash
java -cp HBaseTest.jar:`hbase classpath` HBaseTest
In this case a user would associate the script and the Java application
with the shell job as job binaries and Sahara would execute the script.
In a similar case, Sahara EDP itself uses a Python wrapper around
spark-submit to run Spark jobs. A shell action makes these kinds of
launchers easily available to end users.
Proposed change
===============
Add a `Shell` job type that is implemented by the Oozie EDP engine.
(It is possible that other EDP engines, such as the Spark or Storm
engines, could share a basic shell command implementation for such
jobs but that would be another CR).
Shell jobs can use the existing `mains` and `libs` fields in a job
execution. The script identified in `mains` will be the script that
Sahara runs and the files in `libs` will be supporting files bundled
in the working directory by Oozie.
As with other job types, shell actions will support `configs` and
`args` passed in the existing `job_configs` field of a job execution.
Values in `configs` are specified in the Oozie workflow with
**<configuration>** tags and will be available in a file created by Oozie.
The `args` are specified with the **<argument>** tag and will be passed
to the shell script in order of specification.
In the reference section below there is a simple example of a shell
action workflow. There are three tags in the workflow that for Sahara's
purposes are unique to the `Shell` action and should be handled by
Sahara:
* **<exec>script</exec>**
This identifies the command that should be executed by the shell action.
The value specified here will be the name of the script identified in
`mains`.
Technically, this can be any command on the path but it is probably
simpler if we require it to be a script. Based on some experimentation,
there are subtleties of path evaluation that can be avoided if a script
is run from the working directory
* **<file>support.jar</file>**
This identifies a supporting file that will be included in the working
directory. There will be a <file> tag for every file named in `libs`
as well as the script identified in `mains`.
(Note that the <file> tag can be used in Oozie in any workflow, but
currently Sahara does not implement it at all. It is necessary for the
shell action, which is why it's mentioned here. Whether or not to add
general support for <file> tags in Sahara is a different discussion)
* **<env-var>NAME=VALUE</env-var>**
The env-var tag sets a variable in the shell's environment. Most likely
we can use the existing `params` dictionary field in `job_configs` to
pass env-var values even if we want to label them as "environment
variables" in the UI.
Alternatives
------------
Do nothing.
Data model impact
-----------------
This change adds a new job type, but since job types are stored as strings
it should not have any data model impact.
REST API impact
---------------
Only a change in validation code for job type
Other end user impact
---------------------
None
Deployer impact
---------------
There may be security considerations related to `Shell Action Caveats`,
bullet number 2, in the URL listed in the reference section.
It is unclear whether or not in Sahara EDP the user who started the TaskTracker
is different than the user who submits a workflow. This needs investigation --
how is Sahara authenticating to the Oozie client? What user is the Oozie
server using to deploy jobs?
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
We would need a new form for a Shell job type submission. The form should allow
specification of a main script, supporting libs, configuration values,
arguments, and environment variables (which are 100% analogous to params from
the perspective of the UI)
Implementation
==============
Assignee(s)
-----------
Primary assignee:
egafford
Other contributors:
tmckay
Work Items
----------
* Investigate user issue mentioned above (who is the user that runs
shell actions in Sahara and what are the implications?)
* Add a Shell job type and an implementation in the Oozie EDP engine
components under the `workflow_creator` directory
* Update job validation routines to handle the Shell job type
* Add an integration test for Shell jobs
* Update the EDP documentation to describe the Shell job type
* Add a UI form for Shell job submission
Dependencies
============
None
Testing
=======
* Unit tests to cover creation of the Shell job
* Integration tests to cover running of a simple shell job
Documentation Impact
====================
The EDP sections of the documentation need updating
References
==========
http://blog.cloudera.com/blog/2013/03/how-to-use-oozie-shell-and-java-actions/
A simple Shell action workflow looks like this::
<workflow-app xmlns='uri:oozie:workflow:0.3' name='shell-wf'>
<start to='shell1' />
<action name='shell1'>
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>default</value>
</property>
</configuration>
<exec>doit.sh</exec>
<argument>now</argument>
<env-var>VERSION=3</env-var>
<file>HBaseTest.jar</file>
<file>doit.sh</file>
</shell>
<ok to="end" />
<error to="fail" />
</action>
<kill name="fail">
<message>oops!</message>
</kill>
<end name='end' />
</workflow-app>

143
specs/kilo/edp-api-json-samples.rst
View File

@ -1,143 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=================================
JSON sample files for the EDP API
=================================
https://blueprints.launchpad.net/sahara/+spec/edp-api-json-samples
Provide sample JSON files for all EDP Sahara APIs to facilitate ease of
use by command-line users.
Problem description
===================
As an End User who prefers the Sahara CLI to its UI, I want a set of
pre-constructed example JSON payloads for the Sahara EDP API so that I can
easily learn the expected API signatures and modify them for my use.
Proposed change
===============
Example JSON payloads will be added to the directory
``sahara/etc/edp-examples/json-api-examples/v1.1``, with a subpath for each
relevant manager (data_source, job, job_binary, job_binary_internals, and
job_execution.) It is intended that examples for future API versions will
follow this path structure.
Each file will be named after the pattern: ``method_name.[variety.]json``,
where variety is optional and will be used to describe independently useful
variations in payloads for one method (as in varying engines underlying
data sources or processing jobs.)
Alternatives
------------
A tool could conceivably be created to generate template payloads from the
jsonschemata themselves. However, as the core use of this change is to
provide immediately available, semantically valid payloads for ease of
adoption, it is proposed that providing raw examples will better meet the
perceived user need.
It would also be possible to package these examples directly with the
python-saharaclient repository, an option which has much to recommend it.
However, as these examples are globally useful to any non-UI interface,
as they are reliant on the jsonschemata in the core repository for testing,
and as the extant etc/edp-examples path is a natural home for them,
placing them in the sahara repository itself seems indicated.
Data model impact
-----------------
None.
REST API impact
---------------
None.
Other end user impact
---------------------
None, though it is intended that the payloads may be used via the
python-saharaclient.
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
egafford
Other contributors:
tmckay (primary review)
Work Items
----------
* Payload generation: data_source
* Payload generation: job
* Payload generation: job_binary
* Payload generation: job_binary_internals
* Payload generation: job_execution
* Addition of schema validation unit tests for all the above.
Dependencies
============
None.
Testing
=======
After discussion with croberts and tmckay, it is proposed that integration
testing is in this case unnecessary; these examples constitute documentation.
While exhaustive testing is possible in this case, the resultant bloat of
the CI build would be disproportionate to the utility of the change.
Unit testing will validate that these resources pass schema validation for
their intended APIs.
Documentation Impact
====================
This task is itself a documentation effort. A README.rst will be provided
in place, in keeping with pre-existent etc/edp-examples.
References
==========
* https://etherpad.openstack.org/p/kilo-summit-sahara-edp

172
specs/kilo/edp-data-sources-in-job-configs.rst
View File

@ -1,172 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==================================================================
[EDP] Add options supporting DataSource identifiers in job_configs
==================================================================
https://blueprints.launchpad.net/sahara/+spec/edp-data-sources-in-job-configs
In some cases it would be convenient if users had a way to pass a DataSource
object as a job configuration value to take advantage of the data codified in
the object instead of copying and specifying that information manually. This
specification describes options that allow users to reference DataSource
objects in configuration values, parameters, and arguments in a JobExecution
record.
Problem description
===================
With the exception of Java and Spark all job types in Sahara take a single
input DataSource object and a single output DataSource object. The DataSource
object ids are passed as part of a job execution request and Sahara configures
the job based on the information in the objects. Sahara assumes that jobs at
runtime will consume the path information using a fixed set of parameters and
configuration values. This works well in the general case for the constrained
job types supported by Oozie/hadoop (Hive, Pig, MapReduce) but there are cases
where DataSource objects currently may not be used:
* Java and Spark job types do not require DataSources since they have
no fixed arg list. Currently input and output paths must be specified as URLs
in the ``args`` list inside of ``job_configs`` and authentication configs
must be manually specified.
* Hive, Pig, and MapReduce jobs which use multiple input or output paths or
consume paths through custom parameters require manual configuration.
Additional paths or special configuration parameters (ie anything outside
of Sahara's assumptions) require manual specification in the ``args``,
``params``, or ``configs`` elements inside of ``job_configs``.
Allowing DataSources to be referenced in ``job_configs`` is an incremental
improvement that gives users the option of easily using DataSource objects in
the above cases to specify IO.
Proposed change
===============
Add optional boolean config values on a JobExecution that cause Sahara to
to treat values in ``job_configs`` as potential names or uuids of DataSource
objects. This applies to configuration values, parameters, and arguments
for all job types for maximum flexibility -- Hive and Pig jobs use parameters
to pass values, MapReduce uses configuration values, and Java and Spark use
arguments.
If Sahara resolves a value to the name or uuid of a DataSource it will
substitute the path information from the DataSource for the value and update
the job configuration as necessary to support authentication. If a value does
not resolve to a DataSource name or uuid value, the original value will be
used.
Note that the substitution will occur during submission of the job to the
cluster but will *not* alter the original JobExecution. This means that if
a user relaunches a JobExecution or examines it, the original values will be
present.
The following non mutually exclusive configuration values will control this
feature:
* edp.substitue_data_source_for_name (bool, default False)
Any value in the ``args`` list, ``configs`` dict, or ``params`` dict in
``job_configs`` may be the name of a DataSource object.
* edp.substitute_data_source_for_uuid (bool, default False)
Any value in the ``args`` list, ``configs`` dict or ``params`` dict in
``job_configs`` may be the uuid of a DataSource object.
This change will be usable from all interfaces: client, CLI, and UI. The UI may
choose to wrap the feature in some way but it is not required. A user could
simply specify the config options and the DataSource identifiers on the job
execution configuration panel.
Alternatives
------------
A slightly different approach could be taken in which DataSource names or uuids
are prepended with a prefix to identify them. This would eliminate the need for
config values to turn the feature on and would allow individual values to be
looked up rather than all values. It would be unambiguous but may hurt
readability or be unclear to new users.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None required. However, I can imagine that the UI might benefit from some
simple tooling around this feature, like checkboxes to enable the feature
on a Spark or Java job submission panel.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<tmckay>
Other contributors:
<croberts>
Work Items
----------
* Support in Sahara
* Document
* Support in the UI (optional, it will work without additional work)
Dependencies
============
Testing
=======
Unit tests
Documentation Impact
====================
We will need to document this in the sections covering submission of jobs
to Sahara
References
==========

154
specs/kilo/edp-hive-vanilla-swift.rst
View File

@ -1,154 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=================================================================
Enable Swift resident Hive tables for EDP with the vanilla plugin
=================================================================
https://blueprints.launchpad.net/sahara/+spec/edp-hive-vanilla-swift
vanilla1 plugin supports Hive but Hive can't access Swift.
vanilla2 plugin not supports Hive.
This proposal aims that Hive can process the table that stored in Swift
and add hive support to vanilla2 plugin.
Problem description
===================
When Hive processes table that stored in Swift, `hiveserver` has to
access Swift. But `hiveserver` cannot read SwiftFS configuration from
Hive query. `hiveserver` reads configurations from xml files only
(core-site.xml/hive-site.xml).
Therefore `hiveserver` doesn't know the authentication info and can't access
Swift.
In vanilla2 plugin, doesn't implemented hive support code.
Proposed change
===============
`hiveserver` reads configuration at startup time only and doesn't read
configuration by hive query. Therefore sahara have to pass the swift
authentication info to `hiveserver` through hive-site.xml before
launching `hiveserver`.
When Hive enabled cluster created, Sahara creates keystone TRUST and Swift
proxy user (cluster-scope). And Sahara writes swift proxy user and TRUST to
hive-site.xml before `hiveserver` started.
This will enable `hiveserver` to read a authentication info at startup-time.
And when hive query arrived, `hiveserver` can access the swift with
cluster-scoped TRUST.
When cluster terminated, Sahara removes TRUST and proxy user before cluster's
database entry removed. If error on removing a TRUST, cluster goes error status
and not removed. Therefore there is no orphaned proxy user.
In vanilla2 plugin, implement hive support code in reference to vanilla1.
Alternatives
------------
1. Adds auth config field (`fs.swift.service.sahara.username`,
`fs.swift.service.sahara.password`) to hive-default.xml.
And end-user inputs username/password for cluster configurations.
I think this alternative is very inconvenience.
2. Fix hive-server to read a configuration from hive query.
Data model impact
-----------------
Database schema: No change
Cluster config: Change. But no affects others
cluster-config is dict. Adds internal key and stores swift proxy user info
and TRUST-id. This key doesn't send to client.
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
k.oikw (Kazuki OIKAWA)
Other contributors:
None
Work Items
----------
* Implement proxy user / TRUST creation feature when cluster creates
* Implementation of sanitizing cluster-config
* Implementation of writing proxy user and TRUST to hive-site.xml
* Implement hive support code in vanilla2
Dependencies
============
* https://blueprints.launchpad.net/sahara/+spec/edp-swift-trust-authentication
Testing
=======
We will add a integration test. This test checks whether Hive can process the
table that stored in the swift executes successfully.
Documentation Impact
====================
None
References
==========
None

140
specs/kilo/edp-improve-compatibility.rst
View File

@ -1,140 +0,0 @@
==========================================
[EDP] Improve Java type compatibility
==========================================
https://blueprints.launchpad.net/sahara/+spec/edp-improve-compatibility
Currently, EDP MapReduce (Java type) examples must add modifications
to be able to use from a java action in an Oozie workflow.
This bp aims that users can migrate from other Hadoop cluster to Sahara
without any modifications into their applications.
Problem description
===================
Users need to modify their MapReduce programs as below:
* Add `conf.addResource` in order to read configuration values from
the `<configuration>` tag specified in the Oozie workflow::
// This will add properties from the <configuration> tag specified
// in the Oozie workflow. For java actions, Oozie writes the
// configuration values to a file pointed to by ooze.action.conf.xml
conf.addResource(new Path("file:///",
System.getProperty("oozie.action.conf.xml")));
* Eliminate `System.exit` for following restrictions of Oozie's Java
action.
e.g. `hadoop-examples.jar` bundled with Apache Hadoop has been used
`System.exit`.
First, users would try to launch jobs using examples and/or
some applications executed on other Hadoop clusters (e.g. Amazon EMR).
We should support the above users.
Proposed change
===============
We will provide a new job type, called Java EDP Action, which overrides
the Main class specified by `main_class`.
The overriding class adds property and calls the original main method.
The class also catches an exception that is caused by `System.exit`.
Alternatives
------------
According to Oozie docs, Oozie 4.0 or later provides the way of overriding
an action's Main class (3.2.7.1).
The proposing implementation is more simple than using the Oozie feature.
(We will implement this without any dependencies of Oozie library.)
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
Users will no longer need to modify their applications to use EDP.
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
sahara-dashboard / horizon needs to add this new job type.
Implementation
==============
Assignee(s)
-----------
Primary assignee: Kazuki Oikawa (k.oikw)
Other contributors: Yuji Yamada (yamada-yuji)
Work Items
----------
* Add new job type (`Java.EDP`)
* `Java.EDP` will be subtype of `Java`
* Implement of uploading jar file of overriding class to HDFS
* Implement of creating the `workflow.xml`
* Implement the overriding class
Dependencies
============
None
Testing
=======
We will add a integration test.
This test checks whether WordCount example bundled with Apache Hadoop
executes successfully.
Documentation Impact
====================
If EDP examples use this feature, the docs need to update.
References
==========
* Java action in Oozie http://oozie.apache.org/docs/4.0.0/WorkflowFunctionalSpec.html#a3.2.7_Java_Action

501
specs/kilo/edp-job-types-endpoint.rst
View File

@ -1,501 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==================================
[EDP] Add a new job-types endpoint
==================================
https://blueprints.launchpad.net/sahara/+spec/edp-job-types-endpoint
Add a new job-types endpoint that can report all supported job types for
a Sahara instance and which plugins support them.
Problem description
===================
There are currently two problems around job types in Sahara that impact
user experience:
* The current */jobs/config-hints/<job_type>* endpoint is not adequate for
providing configuration hints because it does not take into account plugin
type or the framework version. This endpoint was meant to give users a list
of likely configuration values for a job that they may want to modify
(at one point the UI incorporated the hints as well). Although some config is
common across multiple plugins, hints need to be specific to the plugin and
the framework version in order to be useful. The endpoint does not take
a cluster or plugin argument and so hints must be general.
* A user currently has no indicator of the job types that are actually
available from a Sahara instance (the UI lists them all). The set of
valid job types is based on the plugins loaded for the current instance.
Furthermore, not all job types will be available to run on all
clusters launched by the user because they are plugin dependent.
These problems should be solved without breaking backward compatibility in
the REST API.
Proposed change
===============
Add a new endpoint that will indicate for the running Sahara instance
which job types are supported by which versions of which plugins.
Optionally, plugin-and-version-specific config hints will be included
for each supported job type.
Because config hints can be very long, they will not be included in a
response by default. A query string parameter will be used to indicate
that they should be included.
The endpoint will support the following optional query strings for filtering.
Each may be used more than once to query over a list of values, for example
`type=Pig&type=Java`:
* **type**
A job type to consider. Default is all job types.
* **plugin**
A plugin to consider. Default is all plugins.
* **version**
A plugin version to consider. Default is all versions.
The REST API method is specified in detail below under *REST API impact*.
We will need two new optional methods in the `Plugin SPI`. This information
ultimately comes from the EDP engine(s) used by a plugin but we do
not want to actually allocate an EDP engine object for this so the
existing **get_edp_engine()** will not suffice (and besides, it requires
a cluster object)::
@abc.abstractmethod
def get_edp_job_types(self, versions=[]):
return []
@abc.abstractmethod
def get_edp_config_hints(self, job_type, version):
return {}
These specific methods are mentioned here because they represent a
change to the public `Plugin SPI`.
Alternatives
------------
Fix the existing */jobs/config-hints* endpoint to take a cluster id or a
plugin-version pair and return appropriate config hints. However, this
would break backward compatibility.
Still add an additional endpoint to retrieve the supported job types
for the Sahara instance separate from config hints.
However, it makes more sense to deprecate the current config-hints interface
and add the new endpoint which serves both purposes.
Data model impact
-----------------
None
REST API impact
---------------
Backward compatibility will be maintained since this is a new endpoint.
**GET /v1.1/{tenant_id}/job-types**
Normal Response Code: 200 (OK)
Errors: none
Indicate which job types are supported by which versions
of which plugins in the current instance.
**Example**
**request**
.. sourcecode:: text
GET http://sahara/v1.1/775181/job-types
**response**
.. sourcecode:: http
HTTP/1.1 200 OK
Content-Type: application/json
.. sourcecode:: json
{
"job_types": [
{
"name": "Hive",
"plugins": [
{
"description": "The Apache Vanilla plugin.",
"name": "vanilla",
"title": "Vanilla Apache Hadoop",
"versions": {
"1.2.1": {}
}
},
{
"description": "The Hortonworks Sahara plugin.",
"name": "hdp",
"title": "Hortonworks Data Platform",
"versions": {
"1.3.2": {},
"2.0.6": {}
}
}
]
},
{
"name": "Java",
"plugins": [
{
"description": "The Apache Vanilla plugin.",
"name": "vanilla",
"title": "Vanilla Apache Hadoop",
"versions": {
"1.2.1": {}
}
},
{
"description": "The Hortonworks Sahara plugin.",
"name": "hdp",
"title": "Hortonworks Data Platform",
"versions": {
"1.3.2": {},
"2.0.6": {}
}
}
]
},
{
"name": "MapReduce",
"plugins": [
{
"description": "The Apache Vanilla plugin.",
"name": "vanilla",
"title": "Vanilla Apache Hadoop",
"versions": {
"1.2.1": {}
}
},
{
"description": "The Hortonworks Sahara plugin.",
"name": "hdp",
"title": "Hortonworks Data Platform",
"versions": {
"1.3.2": {},
"2.0.6": {}
}
}
]
},
{
"name": "MapReduce.Streaming",
"plugins": [
{
"description": "The Apache Vanilla plugin.",
"name": "vanilla",
"title": "Vanilla Apache Hadoop",
"versions": {
"1.2.1": {}
}
},
{
"description": "The Hortonworks Sahara plugin.",
"name": "hdp",
"title": "Hortonworks Data Platform",
"versions": {
"1.3.2": {},
"2.0.6": {}
}
}
]
},
{
"name": "Pig",
"plugins": [
{
"description": "The Apache Vanilla plugin.",
"name": "vanilla",
"title": "Vanilla Apache Hadoop",
"versions": {
"1.2.1": {}
}
},
{
"description": "The Hortonworks Sahara plugin.",
"name": "hdp",
"title": "Hortonworks Data Platform",
"versions": {
"1.3.2": {},
"2.0.6": {}
}
}
]
}
]
}
The job-types endpoint returns a list. Each item in the list is
a dictionary describing a job type that is supported by the
running Sahara. Notice for example that the *Spark* job type is missing.
Each job type dictionary contains the name of the job type and
a list of plugins that support it.
For each plugin, we include the basic identifying information and then
a `versions` dictionary. Each entry in the versions dictionary has
the name of the version as the key and the corresponding config hints
as the value. Since this example did not request config hints, the
dictionaries are empty.
Here is an example of a request that uses the plugin and version filters:
**Example**
**request**
.. sourcecode:: text
GET http://sahara/v1.1/775181/job-types?plugin=hdp&version=2.0.6
**response**
.. sourcecode:: http
HTTP/1.1 200 OK
Content-Type: application/json
.. sourcecode:: json
{
"job_types": [
{
"name": "Hive",
"plugins": [
{
"description": "The Hortonworks Sahara plugin.",
"name": "hdp",
"title": "Hortonworks Data Platform",
"versions": {
"2.0.6": {}
}
}
]
},
{
"name": "Java",
"plugins": [
{
"description": "The Hortonworks Sahara plugin.",
"name": "hdp",
"title": "Hortonworks Data Platform",
"versions": {
"2.0.6": {}
}
}
]
},
{
"name": "MapReduce",
"plugins": [
{
"description": "The Hortonworks Sahara plugin.",
"name": "hdp",
"title": "Hortonworks Data Platform",
"versions": {
"2.0.6": {}
}
}
]
},
{
"name": "MapReduce.Streaming",
"plugins": [
{
"description": "The Hortonworks Sahara plugin.",
"name": "hdp",
"title": "Hortonworks Data Platform",
"versions": {
"2.0.6": {}
}
}
]
},
{
"name": "Pig",
"plugins": [
{
"description": "The Hortonworks Sahara plugin.",
"name": "hdp",
"title": "Hortonworks Data Platform",
"versions": {
"2.0.6": {}
}
}
]
}
]
}
Here is another example that enables config hints and also filters by plugin,
version, and job type.
**Example**
**request**
.. sourcecode:: text
GET http://sahara/v1.1/775181/job-types?hints=true&plugin=hdp&version=1.3.2&type=Hive
**response**
.. sourcecode:: http
HTTP/1.1 200 OK
Content-Type: application/json
.. sourcecode:: json
{
"job_types": [
{
"name": "Hive",
"plugins": [
{
"description": "The Hortonworks Sahara plugin.",
"name": "hdp",
"title": "Hortonworks Data Platform",
"versions": {
"1.3.2": {
"job_config": {
"args": {},
"configs": [
{
"description": "Reduce tasks.",
"name": "mapred.reduce.tasks",
"value": "-1"
}
],
"params": {}
}
}
}
}
]
}
]
}
This is an abbreviated example that shows imaginary config hints.
Other end user impact
---------------------
The python-saharaclient should be extended to support this as well:
.. code::
$ sahara job-types-list [--type] [--plugin [--plugin-version]]
Output should look like this (not sure where else to specify this):
.. code::
+---------------------+-----------------------------------+
| name | plugin(versions) |
+---------------------+-----------------------------------+
| Hive | vanilla(1.2.1), hdp(1.3.2, 2.0.6) |
| Java | vanilla(1.2.1), hdp(1.3.2, 2.0.6) |
| MapReduce | vanilla(1.2.1), hdp(1.3.2, 2.0.6) |
| MapReduce.Streaming | vanilla(1.2.1), hdp(1.3.2, 2.0.6) |
| Pig | vanilla(1.2.1), hdp(1.3.2, 2.0.6) |
+---------------------+-----------------------------------+
Since config hints can return so much information, and description
fields for instance can contain so much text, how to support
config hints through the python-saharaclient is TBD.
As noted above, the `Plugin SPI` will be extended with optional
methods. Existing plugins that support EDP will be modified as
part of this change.
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
The UI will be able to take advantage of this information
and filter the job types available to the user on the forms.
It will also be able to make use of config hints.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
tmckay
Other contributors:
none
Work Items
----------
* Add basic endpoint support with optional methods in the plugin SPI
* Implement the methods for each plugin that supports EDP
This can be done as a series of separate small CRs
* Add support to python-saharaclient
* Update documentation
Dependencies
============
None
Testing
=======
* Unit tests
* Tempest tests for API
Documentation Impact
====================
It should be added to the REST API doc.
References
==========

196
specs/kilo/edp-spark-swift-integration.rst
View File

@ -1,196 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=====================================
Enable Spark jobs to access Swift URL
=====================================
https://blueprints.launchpad.net/sahara/+spec/edp-spark-swift-integration
Spark uses Hadoop filesystem libraries to dereference input and output URLs.
Consequently, Spark can access Swift filesystem URLs if the Hadoop Swift JAR
is included in the image and a Spark job’s Hadoop configuration includes the
necessary Swift credentials. This specification describes a method of
transparently adding Swift credentials to the Hadoop configuration of a Spark
job so that the job source code does not need to be altered and recompiled to
access Swift URLs.
Problem description
===================
Spark jobs may access Swift URLs assuming the cluster image includes the
Hadoop Swift JAR. To do this, the job’s Hadoop configuration must include the
necessary Swift credentials (`fs.swift.service.sahara.username` and
`fs.swift.service.sahara.password`, for example).
As with Oozie Java actions, job source code may be modified and recompiled to
add the necessary configuration values to the job’s Hadoop configuration.
However, this means that a Spark job which runs successfully with HDFS input
and output sources cannot be used "as is" with Swift input and output sources.
Sahara should allow users to run Spark jobs with Swift input and output
sources without altering job source code.
Proposed change
===============
This change follows the approach developed by Kazuki Oikawa in
https://blueprints.launchpad.net/sahara/+spec/edp-improve-compatibility for
Java compatibility.
A new configuration value will be added to Sahara, `edp.spark.adapt_for_swift`.
If this configuration value is set to True on a Spark job, Sahara will run a
wrapper class (SparkWrapper) instead of the original class indicated by the
job. The default for this configuration value will be False.
Sahara will generate a `spark.xml` file containing the necessary Swift
credentials as Hadoop configuration values. This XML file will be uploaded to
the master node with the JAR containing the SparkWrapper class, along with the
other files normally needed to execute a Spark job.
Sahara’s Spark launcher script will run the SparkWrapper class instead of the
job’s designated main class. The launcher will pass the name of the XML
configuration file to SparkWrapper at runtime, followed by the name of the
original class and any job arguments. SparkWrapper will add this XML file to
the default Hadoop resource list in the job's configuration before invoking the
original class with any arguments.
When the job’s main class is run, it’s default Hadoop configuration will
contain the specified Swift credentials.
The permissions of the job dir on the master node will be set to 0x700. This
will prevent users other than the job dir owner from reading the Swift values
from the configuration file.
The sources for the SparkWrapper class will be located in the
sahara-extra repository under the `edp-adapt-for-spark` directory.
This directory will contain a `pom.xml` so that the JAR may be built
with maven. Maintenance should be light since the SparkWrapper class is
so simple; it is not expected to change unless the Hadoop Configuration class
itself changes.
Currently, the plan is to build the JAR as needed and release it with
Sahara in `service/edp/resources/edp-spark-wrapper.jar`. Alternatively, the
JAR could be hosted publically and added to Sahara images as an element.
Alternatives
------------
There is no known way to supply Swift credentials to the Hadoop filesystem
libraries currently other than through configuration values. Additionally,
there is no way to add values to the Hadoop configuration transparently other
than through configuration files.
This does present some security risk, but it is no greater than the risk
already presented by Oozie jobs that include Swift credentials. In fact, this
is probably safer since a user must have direct access to the job directory to
read the credentials written by Sahara.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
The DIB image for Spark will need to change to include the Hadoop Swift JAR.
There is an existing element for this, this is trivial.
Additionally, we still need a solution to a compatibility issue with the
`jackson-mapper-asl.jar.`
This can be patched by including an additional JAR on the cluster nodes, so we
can conceivably bundle the extra jar with the Spark image as a patch.
Alternatively, the issue might be resolved by upgrading the CDH version (or
Spark assembly version) used in the image.
Sahara-dashboard / Horizon impact
---------------------------------
The Spark job submission form should have an input (checkbox?) which allows a
user to set `edp.spark.adapt_for_swift` to True. Default should be False.
We don’t want to assume that just because Swift paths are passed as arguments
to a Spark job that Sahara should run the wrapper. The job itself may handle
the Swift paths in its own way.
Implementation
==============
Assignee(s)
-----------
Primary assignee: tmckay
Other contributors: croberts
Work Items
----------
* Add a simple SparkWrapper class.
This is different than the wrapper class developed for Oozie Java actions.
* Update Spark image to include the Hadoop Openstack JAR
* Find a solution to the jackson issue
* Update the UI
* Implement handling of the `edp.spark.adapt_for_swift` option in Sahara.
This includes generation and upload of the extra XML file, upload of the
additional utility jar, and alteration of the command generated to invoke
spark-submit
* Updated node configuration
All nodes in the cluster should include the Hadoop `core-site.xml` with
general Swift filesystem configuration. Additionally, spark-env.sh should
point to the Hadoop `core-site.xml` so that Spark picks up the Swift configs
and `spark-defaults.conf` needs to set up the executor classpath. These
changes will allow a user to run Spark jobs with Swift paths manually using
`spark-submit` from any node in the cluster should they so choose.
Dependencies
============
https://blueprints.launchpad.net/sahara/+spec/edp-improve-compatibility
Testing
=======
Unit tests and integration tests for Spark jobs will identify any regressions
introduced by this change.
Once we have Spark images with all necessary elements included, we can
add an integration test for Spark with Swift URLs.
Documentation Impact
====================
Any user docs describing job submission should be updated to cover the new
option for Spark jobs.
References
==========

348
specs/kilo/event-log.rst
View File

@ -1,348 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==============================================
Storage of recently logged events for clusters
==============================================
https://blueprints.launchpad.net/sahara/+spec/event-log
This specification proposes to add event logs assigned to cluster.
Problem description
===================
It will be more user friendly have event log assigned to cluster.
In this case users will have the ability to see the steps performed to deploy
a cluster. If there is an issue with their cluster, users will be able to see
the reasons for the issues in the UI and won't be required to read the Sahara
logs.
Proposed change
===============
The new feature will provide the following ability:
* For each cluster there will be an event log assigned to it. The deployer
will have the ability to see it in Horizon. In that case the user will have
the ability to see all the steps for the cluster deploying.
* The deployer will have the ability to see the current progress of cluster
provisioning.
Alternatives
------------
None
Data model impact
-----------------
This change will require to write event log messages to the database.
It's good idea to store events messages in database in similar
manner, in which we store cluster data, node groups data and so on.
Plugins should provide a list of provisioning steps and be able to report
status of the current step. All steps should be performed in linear series,
and we will store events only for the current step. All completed steps should
have the duration time stored in the database. There are no reasons to store
events for successfully completed steps, so they will be dropped periodically.
If an error occurs while provisioning cluster we will have error events saved
for the current step. Also we should store for each error event error_id,
which would help for admins to determine reasons of failures in sahara logs.
We should have new a database object called ClusterEvent, which will have the
following fields:
* node_group_id;
* instance_id;
* instance_name;
* event_info;
* successful;
* provision_step_id;
* id;
* created at;
* updated at;
Also we should have a new database object called ClusterProvisionStep,
which will have the following fields:
* id;
* cluster_id;
* step_name;
* step_type;
* completed;
* total;
* successful;
* started_at;
* completed_at;
* created_at;
* updated_at;
Fields ``step_name`` and ``step_type`` will contain detail info about step.
``step_name`` will contain description of the step, for example
``Waiting for ssh`` or ``Launching instances``. ``step_type`` will
contain info about related process of this step. For example, if we
creating new cluster this field will contain ``creating`` and for
scaling some cluster this field will contain ``scaling``.
So, possible values of this field will be ``creating``, ``scaling``,
``deleting``. Also we should add ability to get main provisioning steps from
Plugin SPI for each ``step_type`` as dictionary. For example, expected
return value:
.. sourcecode:: console
{
"creating": [
"Launching instances",
"Waiting for ssh",
....
]
"scaling": [
....
]
"deleting": [
....
]
}
..
Cluster should have new field:
* provisioning_progress
This field will contain list with provisioning steps, which should provide
ability to get info about provisioning steps from cluster. We should update
this list with new steps every time we start new process with cluster
(creating/scaling/deleting). Provision steps should updated both from plugin
and infra, because some steps are same for all clusters.
REST API impact
---------------
Existing GET request for a cluster should be updated with completed steps
info, and short info for the current step. For example, we will have
following response: "Launching instances completed 1000 out of 1000 in 10
minutes", "Trying ssh completed: 59 out of 1000". Also response should be
sorted by increasing of value ``created_at``.
.. sourcecode:: console
{
"cluster": {
"status": "Waiting",
....
"provisioning_progress": [
{
"id": "1",
"cluster_id": "1111",
"step_name": "Launching instances",
"step_type": "creating",
"completed": 1000,
"total": 1000,
"successful": "True",
"created_at": "2013-10-09 12:37:19.295701",
"started_at": 36000000,
"completed_at": 18000000,
"updated_at": "2013-10-09 12:37:19.295701",
},
{
"id": "2",
"cluster_id": "1111",
"step_name": "Waiting for ssh",
"step_type": "creating",
"completed": 59,
"total": 1000,
"successful": None,
"created_at": "2013-10-09 12:37:19.295701",
"started_at": 18000000,
"completed_at": None,
"updated_at": "2013-10-09 12:37:19.295701",
}
]
....
}
}
..
In case of errors:
.. sourcecode:: console
{
"cluster": {
"status": "Waiting",
....
"provisioning_progress": [
{
"id": "1",
"cluster_id": "1111",
"step_name": "Launching instances",
"step_type": "creating",
"completed": 1000,
"total": 1000,
"successful": "True",
"created_at": "2013-10-09 12:37:19.295701",
"started_at": 36000000,
"completed_at": 18000000,
"updated_at": "2013-10-09 12:37:19.295701",
},
{
"id": "2",
"cluster_id": "1111",
"step_name": "Waiting for ssh",
"step_type": "creating",
"completed": 59,
"total": 1000,
"successful": False,
"created_at": "2013-10-09 12:37:19.295701",
"started_at": 18000000,
"completed_at": None,
"updated_at": "2013-10-09 12:37:19.295701",
}
]
....
}
}
..
Also in these cases we will have events stored in database from which we can
debug cluster problems.
Because first steps of cluster provision are same, then for these steps
infra should update ``provisioning_progress`` field. Also for all
plugin-related steps plugin should update ``provisioning_progress`` field.
So, new cluster field should be updated both from infra and plugin.
New endpoint should be added to get details of the current provisioning step:
GET /v1.1/<tenant_id>/clusters/<cluster_id>/progress
The expected response should looks like:
.. sourcecode:: console
{
"events": [
{
'node_group_id': "ee258cbf-4589-484a-a814-81436c18beb3",
'instance_id': "ss678cbf-4589-484a-a814-81436c18beb3",
'instance_name': "cluster-namenode-001",
'provisioning_step_id': '1',
'event_info': None,
'successful': True,
'id': "ss678cbf-4589-484a-a814-81436c18eeee",
'created_at': "2014-10-29 12:36:59.329034",
},
{
'cluster_id': "d2498cbf-4589-484a-a814-81436c18beb3",
'node_group_id': "ee258www-4589-484a-a814-81436c18beb3",
'instance_id': "ss678www-4589-484a-a814-81436c18beb3",
'instance_name': "cluster-datanode-001",
'provisioning_step_id': '1',
'event_info': None,
'successful': True,
'id': "ss678cbf-4589-484a-a814-81436c18eeee",
'created_at': "2014-10-29 12:36:59.329034",
},
{
'cluster_id': "d2498cbf-4589-484a-a814-81436c18beb3",
'node_group_id': "ee258www-4589-484a-a814-81436c18beb3",
'instance_id': "ss678www-4589-484a-a814-81436c18beb3",
'instance_name': "cluster-datanode-001",
'provisioning_step_id': '2',
'event_info': "Trying to access failed: reason in sahara logs
by id ss678www-4589-484a-a814-81436c18beb3",
'successful': False,
'id': "ss678cbf-4589-484a-a814-81436c18eeee",
'created_at': "2014-10-29 12:36:59.329034",
},
]
}
..
Event info for the failed step will contain the traceback of an error.
Other end user impact
---------------------
None
Deployer impact
---------------
This change will takes immediate effect after it is merged.
Also it is a good idea to have ability to disable event log from
configuration.
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
This change will add section in Horizon at page with event logs
/data_processing/clusters/cluster_id.
At this page it will be possible to see main provisioning steps,
and current progress of all of it.
Also we would have an ability to see events of current provisioning
step. In case of errors we will be able to see all events of the current
step and main reasons of failures.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
vgridnev
Other contributors:
slukjanov, alazarev, nkonovalov
Work Items
----------
This feature require following modifications:
* Add ability to get info about main steps of provisioning cluster from
plugin;
* Add ability to view progress of current provisioning step;
* Add ability to specify events to current cluster and current step;
* Add periodic task to erase redundant events from previous step;
* Add ability to view events about current step of cluster provisioning;
* Sahara docs should be updated with some use cases of this feature;
* Saharaclient should be modified with new REST API feature;
* New cluster tab with events in Horizon should be implemented;
* Add unit test to test new features of events.
Dependencies
============
Depends on OpenStack requirements
Testing
=======
As written in Work Items section this feature will require unit tests
Documentation Impact
====================
As written in Work Items section this feature will require docs updating
with some use cases of feature
References
==========

122
specs/kilo/exceptions-improvement.rst
View File

@ -1,122 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Exceptions improvement
==========================================
https://blueprints.launchpad.net/sahara/+spec/exceptions-improvement
This specification proposes to add identifiers for every raised Sahara
exception, so that they can be easily found in logs.
Problem description
===================
Now it's hard to find an error in logs, especially if there are a lot of
errors of the same type which occurs when a lot of operations are executed
simultaneously. They can produce a bunch of similar exceptions (error code
doesn't help in this kind of situation).
It would be nice to have an opportunity to find exceptions by unique
identifiers. This identifiers will be found in Horizon tab with events that
will be implemented in this spec: https://review.openstack.org/#/c/119052/.
Proposed change
===============
Support Features:
* Every error that has been raised during the workflow will have, besides of
error message, uuid property, whereby error can be found in logs easily.
For example, NotFoundException will leave in logs:
.. sourcecode:: console
NotFoundException: Error ID: 7a229eda-f630-4153-be03-d71d6467f2f4
Object 'object' is not found
..
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
apavlov-n
Other contributors:
sreshetniak
Work Items
----------
* Adding ability to generate unique identifiers for SaharaException class
* Change messages of Sahara exceptions so that all of them contain
identifier.
Dependencies
============
None
Testing
=======
None
Documentation Impact
====================
None
References
==========
None

146
specs/kilo/first-run-api-usage.rst
View File

@ -1,146 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Use first_run to One-step Start Cluster
==========================================
https://blueprints.launchpad.net/sahara/+spec/first-run-api-usage
This specification proposes to use cm_api method first_run to start cluster
for CDH plugin in Sahara, instead of current a batch of methods.
Problem description
===================
Now in CDH plugin method start_cluster, a lot of methods defined in
cloudera_utils.py is used to configure/prepare services to be started. Those
methods include cm_api methods in their body, and check the return status.
E.g., cu.format_namenode will call cm_api ApiService.format_hdfs.
However, this way is not preferred by Cloudera. The much easier way is using
a single method first_run to most of those work. In fact, in Cloudera Manager
first_run is also used to do the final step of deploying a cluster.
Changing current start_cluster codes into using first_run method will benefit
by:
* Leave work up to Cloudera Manager to itself, instead of manually doing it.
* Simplify the work of adding more service support.
* Avoid possible errors generated by future CM changes.
Proposed change
===============
The implementation will change start_cluster to call first_run, and remove the
other part of work can be done by first_run from the method body.
For detail, it will be like following:
In deploy.py, possible start_cluster method may be like::
def start_cluster(cluster):
cm_cluster = cu.get_cloudera_cluster(cluster)
""" some pre codes """
cu.first_run(cluster)
""" some post codes """
Current methods used to configure CDH components, like _configure_spark,
_install_extjs, and most part of start_cluster body can be removed.
In cloudera_utils.py, first_run can be defined as (just for example)::
@cloudera_cmd
def first_run(cluster):
cm_cluster = get_cloudera_cluster(cluster)
yield cm_cluster.first_run()
Methods for configuring CDH components, like create_yarn_job_history_dir,
create_oozie_db, install_oozie_sharelib, create_hive_metastore_db,
create_hive_dirs can be removed.
Alternatives
------------
Current way works at this stage, but it increases complexity of coding work to
add more services support to CDH plugin. And, when CM is upgraded in the
future, the correctness of current codes cannot be assured. At the end, the
first_run method to start services is recommended by Cloudera.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
It will be easier for developers to add more CDH services support.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
ken chen
Other contributors:
ken chen
Work Items
----------
The work items will be:
* Change current deploy.py and cloudera_utils.py in the way above.
* Test and evaluate the change.
Dependencies
============
None
Testing
=======
Take an integration test to create a cluster.
Documentation Impact
====================
None
References
==========
None

172
specs/kilo/hdp-plugin-enable-hdfs-ha.rst
View File

@ -1,172 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
============================================================
Enable HDFS NameNode High Availability with HDP 2.0.6 plugin
============================================================
https://blueprints.launchpad.net/sahara/+spec/hdp-plugin-enable-hdfs-ha
Extend HDP 2.0.6 plugin to include the setup and configuration of the HDFS
NameNode High Availability after creating, configuring and starting the
cluster.
Problem description
===================
Hadoop clusters are created with a single NameNode which represents a SPOF
(Single Point Of Failure). If the NameNode becomes unavailable, the whole
cluster becomes unavailable and we have to wait for the NameNode to come back
up before using the cluster again. NameNode High Availability was introduced in
Hadoop 2.0.0 and integrated into HDP as of the 2.0.0 release. The High
Availability is achieved by having 2 NameNodes, an active NameNode and a stand
by one. When the active fails the standby steps in and act as the cluster's
NameNode. NameNode's High Availability can be configured manually on clusters
deployed with HDP 2.0.6 plugin in Sahara through Ambari, but the process is
long, tedious and error prone.
End users might not have the necessary skills required to setup the High
Availability and deployers might prefer to deploy highly available clusters
without manually configuring each one.
HDFS NameNode High Availability (Using Quorum Journal Manager (QJM)) uses
Journal nodes to share HDFS edits between the active and the standby
namenodes. The journal nodes ensure that the two namenodes have the same set of
HDFS edits, but do not ensure the automatic failover. The automatic failover
requires Zookeeper servers and Zookeeper Failover Controllers (ZKFC). A
typical cluster with HDFS NameNode High Availability uses at least three (or a
an odd number greater than three) journal nodes and a zookeeper cluster with
three or five zookeeper servers. The Zookeeper Failover Controllers are
installed on the servers acting as active and standby namenodes. The setup
removes the secondary namenode (which is usually installed on a different
server than the one hosting the namenode) and installs a second namenode
process. The journal nodes and zookeeper servers can be installed on the same
servers running the active and standby (old secondary namenode) namenodes. This
leaves us with 2 journal nodes and 2 zookeeper servers. An additional server is
all what's needed with journal node and zookeeper server to setup a minimally
viable Hadoop cluster with HDFS NameNode High Availability. (For more info :
http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html)
Proposed change
===============
The 'Create Node Group Template' wizard will introduce a new process
'JOURNALNODE' to the list of available processes for HDP 2.0.6 plugin. Other
processes necessary for HDFS HA are either already included in the list
(NAMENODE, ZOOKEEPER_SERVER and SECONDARY_NAMENODE) or will be automatically
setup (Zookeeper Failover Controllers).
The 'Launch Cluster' wizard for HDP 2.0.6 plugin will include a checkbox
'Enable HDFS HA'. This option will default to False and will be added to the
cluster object.
The verification code will verify the necessary requirements for a Hadoop
cluster and a set of additional requirements in the case where 'Enable HDFS HA'
is set to True. The requirements include : NAMENODE and SECONDARY_NAMENODE on
different servers At least three journal nodes on different servers At least
three zookeeper servers on different servers
Upon successful validation the cluster will be created and once it's in the
'Active' state Availability' and if 'Enable HDFS HA' is True the service will
instruct the plugin to start configuring the NameNode High Availability. The
cluster will be set in 'Configuring HDFS HA' state and the plugin will start
the configuration procedure. The procedure starts by the plugin stopping all
the services and executing some preparation commands on the server with the
namenode process (through the hadoopserver objects). Then the plugin installs
and starts the journal nodes using Ambari REST API (POST, PUT, WAIT ASYNC).
Next the configuration is updated using Ambari REST API (PUT), other services
including Hive, Oozie and Hue might require configuration update if they are
installed. Finally some more remote commands are executed on the namenodes and
the Zookeeper Failover Controllers are installed and started and the
SECONDARY_NAMENODE process is deleted. The plugin will return and the cluster
will be set back in the 'Active' state.
Alternatives
------------
Manual setup through Ambari web interface
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
Developers of subsequent versions of the HDP plugin should take into account
this option and the added functionality. The procedure is not likely to change
in newer versions of HDP as it uses Ambari's API which stays intact with newer
versions.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
A checkbox 'Enable HDFS HA' in the 'Launch Cluster' wizard when the user
chooses HDP 2.0.6 plugin.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
abbass-marouni
Work Items
----------
* Add a new attribute to cluster-level configs to indicate whether HA is
enabled or not.
* Add new service classes to HDP 2.0.6 for Journal nodes and Zookeeper Failover
Controllers
* Add new remote methods to hdp hadoopserver.py for remote commands
* Add new methods to generate new configurations according to cluster
configuration
Dependencies
============
None
Testing
=======
Unit Test service classes
Unit Test new cluster specs
Integration Test cluster creation with HA
Integration Test cluster creation without HA
Documentation Impact
====================
Update documentation to reflect new changes and to explain new options.
References
==========
None

223
specs/kilo/indirect-vm-access.rst
View File

@ -1,223 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
======================
Indirect access to VMs
======================
https://blueprints.launchpad.net/sahara/+spec/indirect-vm-access
This blueprint proposes one more way for Sahara to manage VMs. Management
could be done via VM that works as proxy node. In this case access
from controller should be given for one VM only.
Problem description
===================
Currently there are several ways to give Sahara access to VMs:
1. flat private network
* not secure
* doesn't work with neutron
2. floating IPs
* all nodes need to have floating IPs
- floating IPs are limited resource
- floating IPs are usually for external world, not for access from
controller
* all nodes need to be accessible from controller nodes
- it is more complicated in HA mode
* access to data nodes should be secure
3. net_ns
* hard to configure
* can be inappropriate
* doesn't work in HA mode
5. tenant-specific proxy node (https://review.openstack.org/#/c/131142/)
* proxy setting is for the whole system (template based)
* proxy can't be configured for a specific cluster
* proxy node needs to be spawned and configured manually
4. agents
* not implemented yet
* require external message queue accessible from VMs and controllers
* require maintenance of agents
So, there can be cases when none of listed approaches work.
Proposed change
===============
This blueprint proposes one more way to access VMs by Sahara.
Sahara will use one of spawned VMs as proxy node and gain access to all other
nodes through it. Access to VM that has proxy node role could be gained using
any of methods above.
Sahara will understand which node to use as a proxy node by "is_proxy" field
of nodegroup. If this nodegroup contains several instances the first one will
be used as proxy (this leaves space for load balancing).
So, proposed workflow:
1. Nodegoup object is extended with "is_proxy" field, horizon is changed
accordingly.
2. User selects "is_proxy" checkbox for one of node groups (manager, master or
separate one). If proxy is used for separate node group it could be with
really small flavor.
3. Sahara spawns all infrastructure
4. Sahara communicates with all instances via node with proxy role. Internal
IPs are used for communication. This removes restriction that all nodes
must have floating IP in case if floating network used for management. In
case with proxy node this restriction will be applied to proxy node only.
Pros:
1. we need external access to only one VM.
2. data nodes could be isolated
Cons:
1. Indirect access could be slower.
2. Loss of proxy means loss of access to entire cluster. Intellectual
selection is possible, but not planned for the first implementation.
Implementation will extend global proxy implemented at
https://review.openstack.org/#/c/131142/. For indirect access Paramiko-based
analogy of "ssh proxy nc host port" command will be used. Paramiko
implementation will allow to use private keys from memory.
Note, proxy command can still be used to access proxy instance.
Implementation details
----------------------
Sahara uses two ways of access to instances:
1. SSH
2. HTTP
SSH access
++++++++++
For ssh access one more layer of ssh will be added. Old code:
.. sourcecode:: python
_ssh.connect(host, username=username, pkey=private_key, sock=proxy)
New code:
.. sourcecode:: python
_proxy_ssh = paramiko.SSHClient()
_proxy_ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
_proxy_ssh.connect(proxy_host, username=proxy_username,
pkey=proxy_private_key, sock=proxy)
chan = _proxy_ssh.get_transport().open_session()
chan.exec_command("nc {0} {1}".format(host, SSH_PORT))
_ssh.connect(host, username=username, pkey=private_key, sock=chan)
HTTP access
+++++++++++
Http access will be implemented in a similar way with ProxiedHTTPAdapter.
SshProxySocket class will be implemented that corresponds to netcat socket
running on remote host.
Note, if proxycommand present, it will be passed to paramiko directly without
involving NetcatSocket class.
Alternatives
------------
This blueprint offers one more way to access VMs. All existing ways will remain
unchanged.
Data model impact
-----------------
None
REST API impact
---------------
New boolean field "is_proxy" in nodegroup and nodegroup template objects.
Other end user impact
---------------------
None
Deployer impact
---------------
One more deployment option to consider.
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
Checkbox in nodegroup template edit form.
Implementation
==============
Assignee(s)
-----------
Primary Assignee:
Andrew Lazarev (alazarev)
Work Items
----------
* Sahara core changes
* Python client changes
* Horizon changes
* Doc changes
Dependencies
============
* Global proxy implementation (https://review.openstack.org/#/c/131142/)
Testing
=======
Manually
Documentation Impact
====================
The feature needs to be documented.
References
==========
None

153
specs/kilo/mapr-plugin.rst
View File

@ -1,153 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===========================
Plugin for Sahara with MapR
===========================
https://blueprints.launchpad.net/sahara/+spec/mapr-plugin
https://blueprints.launchpad.net/sahara/+spec/mapr-image-elements
This specification proposes to add MapR plugin with MapR Distribution of
Hadoop in Sahara.
Problem description
===================
The MapR Distribution for Apache Hadoop provides organizations with an
enterprise-grade distributed data platform to reliably store and process
big data. MapR packages a broad set of Apache open source ecosystem
projects enabling batch, interactive, or real-time applications. The
data platform and the projects are all tied together through an advanced
management console to monitor and manage the entire system.
MapR is one of the largest distributions for Hadoop supporting more than
20 open source projects. MapR also supports multiple versions of the
various individual projects thereby allowing users to migrate to the latest
versions at their own pace. The table below shows all of the projects
actively supported in the current GA version of MapR Distribution for
Hadoop as well as in the next Beta release.[1]
Proposed change
===============
MapR plugin implementation will support Hadoop 0.20.2 and Hadoop 2.4.1.
Plugin will support key Sahara features:
* Cinder integration
* Cluster scaling/decommission
* EDP
* Cluster topology validation
* Integration with Swift
Plugin will be able to install following services:
* MapR-FS
* YARN
* Oozie (two versions are supported)
* HBase
* Hive (two versions are supported)
* Pig
* Mahout
* Webserver
MapR plugin will support the following OS: Ubuntu 14.04 and CentOS 6.5.
MapR plugin will support the following node types:
* Nodes Running ZooKeeper and CLDB
* Nodes for Data Storage and Processing
* Edge Nodes
In a production MapR cluster, some nodes are typically dedicated to cluster
coordination and management, and other nodes are tasked with data storage
and processing duties. An edge node provides user access to the cluster,
concentrating open user privileges on a single host.
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
MapR plugin uses specific pre-installed images with
MapR local repository files.
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
aosadchiy
Other contributors:
ssvinarchuck
Work Items
----------
* Add implementation of plugin for bare images.
Dependencies
============
Depends on OpenStack requirements.
Testing
=======
* Add unit tests to Sahara to cover basic functionality of plugin
* Add integration tests to Sahara
Documentation Impact
====================
MapR plugin documentation should be added to the plugin section of Sahara docs.
References
==========
* [1] https://www.mapr.com/products/apache-hadoop
* [2] http://doc.mapr.com/display/MapR/Home
* [3] https://www.mapr.com/
* [4] https://github.com/mapr

124
specs/kilo/mapr-refactor.rst
View File

@ -1,124 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=========================
Refactor MapR plugin code
=========================
https://blueprints.launchpad.net/sahara/+spec/mapr-refactor
MapR plugin's code should be refactored to support easy addition of new
services and releases
Problem description
===================
Current plugin implementation has several weaknesses:
* Declarative nature of Service
* Code complexity caused by usage of plugin-spec.json
* Almost all actions are implemented as sequence of utility functions calls
* Not clear separation of behaviour to modules and classes
* Some code is redundant
Proposed change
===============
* Extract Service entity to separate class with customizable behaviour
* Move provisioning logic from utility modules to specialized classes
* Remove redundant code
MapR plugin implementation delegates all operations to it's counterparts in
VersionHandler interface. VersionHandler interface mimics plugin SPI with
additional methods get_context(cluster) and get_services(). ClusterContext
object returned by get_context wraps cluster object passed as argument and
provides additional information about cluster as well as utility methods
related to wrapped cluster.
Service definitions resides in 'sahara.plugins.mapr.services' package instead
of plugin-spec.json which is completely removed now. Each service definition
represents particular version of service.
Alternatives
------------
Leave code "as-is"
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
Developers of subsequent versions of the MapR plugin should take into account
this changes.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
aosadchiy
Other contributors:
ssvinarchuk
Work Items
----------
1) Extract Service entity to separate class with customizable behaviour
2) Move provisioning logic from utility modules to specialized classes
3) Remove redundant code
Dependencies
============
None
Testing
=======
Existing integration tests are sufficient
Documentation Impact
====================
None
References
==========
None

125
specs/kilo/periodic-cleanup.rst
View File

@ -1,125 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=============================================================
Clean up clusters that are in non-final state for a long time
=============================================================
https://blueprints.launchpad.net/sahara/+spec/periodic-cleanup
This spec is to introduce periodic task to clean up old clusters in
non-final state.
Problem description
===================
For now it is possible that sahara cluster becomes stuck because of different
reasons (e.g. if sahara service was restarted during provisioning or neutron
failed to assign floating IP). This could lead to clusters holding resources
for a long time. This could happen in different tenants and it is hard to
check such conditions manually.
Related bug: https://bugs.launchpad.net/sahara/+bug/1185909
Proposed change
===============
Add "cleanup_time_for_nonfinal_clusters" parameter in "periodic" section of
configuration.
Based on this configuration periodic task will search clusters that are in
non-final state and weren't updated for a given time.
Term "non-final" includes all cluster states except "Active" and "Error".
"cleanup_time_for_nonfinal_clusters" parameter will be in hours. Non-positive
value will indicate that clean up option is disabled.
Default value will be 0 to keep backward compatibility (users don't expect
that after upgrade all their non-final cluster will be deleted).
'updated_at' column of 'clusters' column will be used to determine last
change. This is not 100% accurate, but good enough. This field is changed
each time cluster status is changed.
Alternatives
------------
Add such functionality to external service (e.g. Blazar).
Data model impact
-----------------
None.
REST API impact
---------------
None.
Other end user impact
---------------------
None.
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
alazarev (Andrew Lazarev)
Other contributors:
None
Work Items
----------
* Implement feature
* Document feature
Dependencies
============
None.
Testing
=======
Manually.
Documentation Impact
====================
Need to be documented.
References
==========
* https://bugs.launchpad.net/sahara/+bug/1185909

151
specs/kilo/sahara-api-workers.rst
View File

@ -1,151 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Support multi-worker Sahara API deployment
==========================================
https://blueprints.launchpad.net/sahara/+spec/sahara-api-workers
Add support of multi-worker deployment of Sahara API.
Problem description
===================
Currently Sahara API uses one thread with wsgi application. This means that
API can service only one request at a time. Some requests (e.g. cluster scale)
require synchronous requests to Sahara engine (using message queue). This
means that Sahara API will not be able to service other requests until full
round trip finished.
Also, multi-threaded solution gives much more options for performance tuning.
It could allow to utilize more server CPU power.
Most of OpenStack services support several workers for API.
Proposed change
===============
The ideal solution for that would be migration to Pecan and WSME (
https://etherpad.openstack.org/havana-common-wsgi) with multi-threading
support. Although this will require a lot of work and there is no much
pressure to do that.
This spec suggests simple solution of specific problem without much
refactoring of existing code.
So, the solution is:
1. Leave current wsgi implementation
2. Leave current socket handling
3. Run wsgi server in several threads/processes
4. Implement only children processes management, leave all existing code as is.
Children processes management will include:
1. Handling of children processes, restart of dead processes
2. Proper signals handling (see https://bugs.launchpad.net/sahara/+bug/1276694)
3. Graceful shutdown
4. Support of debug mode (with green threads instead of real threads)
Things that will NOT be included:
1. Config reload / API restart
Alternatives
------------
Migrate to Pecan and WSME first.
Implementation details
----------------------
Most OpenStack services use deprecated oslo wsgi module. It has tight
connections with oslo services module.
So, there are three options here:
1. Use deprecated oslo wsgi module. (Bad, since module is deprecated)
2. Use oslo services module, but write all wsgi stuff ourselves (or copy-paste
from other project).
3. Write minimum code to make server start multi-worker (e.g. see how it is
implemented in Heat).
I propose going with the option 3. There is no much sense spending resources
for code, that will be replaced anyway (we will definitely migrate to Pecan
some day).
Data model impact
-----------------
None.
REST API impact
---------------
None.
Other end user impact
---------------------
None.
Deployer impact
---------------
One more configuration parameter.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
alazarev (Andrew Lazarev)
Other contributors:
None
Work Items
----------
* Implement feature
* Document feature
Dependencies
============
None
Testing
=======
Manually. Probably CI could be changed to run different tests in
different modes.
Documentation Impact
====================
Need to be documented.
References
==========
None.

174
specs/kilo/sahara-log-guidelines.rst
View File

@ -1,174 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
New style logging
==========================================
https://blueprints.launchpad.net/sahara/+spec/new-style-logging
Rewrite Sahara logging in unified OpenStack style proposed by
https://blueprints.launchpad.net/nova/+spec/log-guidelines
Problem description
===================
Now log levels and messages in Sahara are mixed and don't match the OpenStack
logging guidelines.
Proposed change
===============
The good way to unify our log system would be to follow the major guidelines.
Here is a brief description of log levels:
* Debug: Shows everything and is likely not suitable for normal production
operation due to the sheer size of logs generated (e.g. scripts executions,
process execution, etc.).
* Info: Usually indicates successful service start/stop, versions and such
non-error related data. This should include largely positive units of work
that are accomplished (e.g. service setup, cluster start, successful job
execution).
* Warning: Indicates that there might be a systemic issue;
potential predictive failure notice (e.g. job execution failed).
* Error: An error has occurred and an administrator should research the event
(e.g. cluster failed to start, plugin violations of operation).
* Critical: An error has occurred and the system might be unstable, anything
that eliminates part of Sahara's intended functionality; immediately get
administrator assistance (e.g. failed to access keystone/database, plugin
load failed).
Here are examples of LOG levels depending on cluster execution:
* Script execution:
.. code:: python
LOG.debug("running configure.sh script")
..
* Cluster startup:
.. code:: python
LOG.info(_LI("Hadoop stack installed successfully."))
..
* Job failed to execute:
.. code:: python
LOG.warning(_LW("Can't run job execution {job} (reason: {reason}")).format(
job = job_execution_id, reason = ex)
..
* HDFS can't be configured:
.. code:: python
LOG.error(_LE('Configuring HDFS HA failed. {reason}')).format(
reason = result.text)
..
* Cluster failed to start:
.. code:: python
LOG.error(_LE('Install command failed. {reason}')).format(
reason = result.text)
..
Additional step for our logging system should be usage of pep3101 as unified
format for all our logging messages. As soon as we try to make our code more
readable please use {<smthg>} instead of {0} in log messages.
Alternatives
------------
We need to follow OpenStack guidelines, but if needed we can move plugin logs
to DEBUG level instead of INFO. It should be discussed separately in each case.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
starodubcevna
Work Items
----------
* Unify existing logging system
* Unify logging messages
* Add additional logs if needed
Dependencies
============
None
Testing
=======
None
Documentation Impact
====================
None
References
==========
https://blueprints.launchpad.net/nova/+spec/log-guidelines
https://www.python.org/dev/peps/pep-3101/

125
specs/kilo/sahara-support-https.rst
View File

@ -1,125 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
============================
HTTPS support for sahara-api
============================
https://blueprints.launchpad.net/sahara/+spec/sahara-support-https
Most OpenStack services support running server supporting HTTPS connections.
Sahara must support such way too.
Problem description
===================
There are two common ways to enable HTTPS for the OpenStack service:
1. TLS proxy. Proxy communicates with user via HTTPS and redirects all
requests to service via unsecured HTTP. Keystone is configured to point
on HTTPS port. Internal port is usually closed for outside using firewall.
No additional work to support HTTPS required on service side.
2. Native support. Service can be configured to expect HTTPS connections. In
this case service handles all security aspects by itself.
Most OpenStack services support both types of enabling SSL. Sahara currently
can be secured only using TLS proxy.
Proposed change
===============
Add ability to Sahara API to listen on HTTPS port.
Currently there is no unified way for OpenStack services to work with HTTPS.
Process of unification started with sslutils module in oslo-incubator. Sahara
could use this module to be on the same page with other services.
Alternatives
------------
Copy-paste SSL-related code from other OpenStack project.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
* python-saharaclient should support SSL-related options
Deployer impact
---------------
One more option to consider.
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
Add SSL-related parameters to pass to python client.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
alazarev (Andrew Lazarev)
Other contributors:
None
Work Items
----------
* Implement feature in Sahara
* Import sslutils
* Configure WSGI server to HTTPS
* Add support to python-saharaclient
* Add support to devstack
* Add documentation
Dependencies
============
* sslutils module from oslo-incubator
Testing
=======
Devstack doesn't have HTTPS testing for now. It looks like manual testing is
the only option.
Documentation Impact
====================
Need to be documented.
References
==========
None

449
specs/kilo/scenario-integration-tests.rst
View File

@ -1,449 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=====================================
Scenario integration tests for Sahara
=====================================
https://blueprints.launchpad.net/sahara/+spec/scenario-integration-tests
For now the Sahara project has not functional integration tests.
We need to create new integration tests that will be more flexible.
Problem description
===================
Current integration tests allow to test only limited number of test scenarios
and cluster configurations that are hardcoded in code of tests.
In many cases we should test various configurations of Sahara clusters but
current integration tests don't have this functionality. Also we have many
copy-pasted code in test files and this code requires large work
for its refactoring.
Proposed change
===============
It is proposed to create new integration tests that will be more flexible.
Test scenarios will be defined in YAML files and it is supposed this approach
will provide more flexibility in testing. The usual scenario will have the
following look:
.. sourcecode:: yaml
credentials:
os_username: dev-user
os_password: swordfish
os_tenant: devs
os_auth_url: http://os_host:5000/v2.0
sahara_url: http://sahara_host:8336/v1.1 # optional
network:
type: neutron # or nova-network
private_network: private # for neutron
auto_assignment_floating_ip: false # for nova-network
public_network: public # or floating_ip_pool for nova-network
clusters:
- plugin_name: vanilla
plugin_version: 2.6.0
image: some_id
node_group_templates: # optional
- name: master
node_processes:
- namenode
- resourcemanager
flavor_id: '3'
- name: worker
node_processes:
- datanode
- nodemanager
flavor_id: '3'
cluster_template: # optional
name: vanilla
node_group_templates:
master: 1
worker: 3
scenario: # optional
- run_jobs
- scale
- run_jobs
- plugin_name: hdp
plugin_version: 2.0.6
image: some_id
Minimal scenario will look the following way:
.. sourcecode:: yaml
clusters:
- plugin_name: vanilla
plugin_version: 2.6.0
image: some_id
Full scenario will look the following way:
.. sourcecode:: yaml
concurrency: 3
credentials:
os_username: dev-user
os_password: swordfish
os_tenant: devs
os_auth_url: http://os_host:5000/v2.0
sahara_url: http://sahara_host:8336/v1.1 # optional
network:
type: neutron # or nova-network
private_network: private # for neutron
auto_assignment_floating_ip: false # for nova-network
public_network: public # or floating_ip_pool for nova-network
clusters:
# required
- plugin_name: vanilla
# required
plugin_version: 2.6.0
# required (id or name)
image: some_id
node_group_templates: # optional
- name: master
node_processes:
- namenode
- resourcemanager
flavor_id: '3'
description: >-
Some description
volumes_per_node: 2
volumes_size: 2
node_configs:
HDFS:
dfs.datanode.du.reserved: 10
security_groups: ~
auto_security_group: true
availability_zone: nova
volumes_availability_zone: nova
volume_type: lvm
- name: worker
node_processes:
- datanode
- nodemanager
flavor_id: 3
cluster_template: # optional
name: vanilla
description: >-
Some description
cluster_configs:
HDFS:
dfs.replication: 1
node_group_templates:
master: 1
worker: 3
anti_affinity: true
cluster:
name: test-cluster
is_transient: true
description: >-
Cluster description
scaling:
- operation: resize
node_group: worker
size: 4
- operation: add
node_group: worker
size: 2
scenario: # optional
- run_jobs
- scale
- run_jobs
edp_jobs_flow: example
retain_resource: true # optional
edp_jobs_flow:
example:
- type: Pig
main_lib:
source: swift
path: path_to_pig_script.pig
input_datasource:
type: swift
source: etc/edp-examples/edp-pig/top-todoers/data/input
output_datasource:
type: hdfs
destination: /user/hadoop/edp-output
configs:
dfs.replication: 1
- type: Java
additional_libs:
- type: database
source: |
etc/edp-examples/.../hadoop-mapreduce-examples-2.4.1.jar
configs:
edp.java.main_class: |
org.apache.hadoop.examples.QuasiMonteCarlo
args:
- 10
- 10
After we described test scenario in YAML file we run test as usual.
The python test code will be generated from these YAML files.
We will use the Mako library to generate the python code. The generated code
will look like:
.. sourcecode:: python
from sahara.tests.scenario import base
class vanilla2_4_1TestCase(base.BaseTestCase):
@classmethod
def setUpClass(cls):
super(vanilla2_4_1TestCase, cls).setUpClass()
cls.credentials = {
'os_username': 'dev-user',
'os_password': 'swordfish',
'os_tenant': 'devs',
'os_auth_url': 'http://172.18.168.5:5000/v2.0',
'sahara_url': None
}
cls.network = {
'type': 'neutron',
'public_network': 'net04_ext',
'auto_assignment_floating_ip': False,
'private_network': 'dev-network'
}
cls.testcase = {
'image': 'sahara-juno-vanilla-2.4.1-ubuntu-14.04',
'plugin_name': 'vanilla',
'retain_resources': False,
'class_name': 'vanilla2_4_1',
'edp_jobs_flow': [
{
'configs': {
'dfs.replication': 1
},
'output_datasource': {
'type': 'hdfs',
'destination': '/user/hadoop/edp-output'
},
'input_datasource': {
'type': 'swift',
'source':
'etc/edp-examples/edp-pig/top-todoers/data/input'
},
'main_lib': {
'type': 'swift',
'source':
'etc/edp-examples/edp-pig/top-todoers/example.pig'
},
'type': 'Pig'
},
{
'type': 'Java',
'args': [10, 10],
'additional_libs': [
{
'type': 'database',
'source':
'etc/edp-examples/hadoop2/edp-java/'
'hadoop-mapreduce-examples-2.4.1.jar'
}
],
'configs': {
'edp.java.main_class':
'org.apache.hadoop.examples.QuasiMonteCarlo'
}
}
],
'scenario': ['run_jobs', 'scale', 'run_jobs'],
'plugin_version': '2.4.1'
}
def test_plugin(self):
self.create_cluster()
self.check_run_jobs()
self.check_scale()
self.check_run_jobs()
class hdp2_0_6TestCase(base.BaseTestCase):
@classmethod
def setUpClass(cls):
super(hdp2_0_6TestCase, cls).setUpClass()
cls.credentials = {
'os_username': 'dev-user',
'os_password': 'swordfish',
'os_tenant': 'devs',
'os_auth_url': 'http://172.18.168.5:5000/v2.0',
'sahara_url': None
}
cls.network = {
'type': 'neutron',
'public_network': 'net04_ext',
'auto_assignment_floating_ip': False,
'private_network': 'dev-network'
}
cls.testcase = {
'image': 'f3c4a228-9ba4-41f1-b100-a0587689d4dd',
'plugin_name': 'hdp',
'retain_resources': False,
'class_name': 'hdp2_0_6',
'edp_jobs_flow': None,
'scenario': ['run_jobs', 'scale', 'run_jobs'],
'scaling': [
{
'operation': 'resize',
'size': 5,
'node_group': 'worker'
}
],
'plugin_version': '2.0.6'
}
def test_plugin(self):
self.create_cluster()
self.check_run_jobs()
self.check_scale()
self.check_run_jobs()
Mako template will look the following way:
.. sourcecode:: mako
from sahara.tests.scenario import base
% for testcase in testcases:
${make_testcase(testcase)}
% endfor
<%def name="make_testcase(testcase)">
class ${testcase['class_name']}TestCase(base.BaseTestCase):
@classmethod
def setUpClass(cls):
super(${testcase['class_name']}TestCase, cls).setUpClass()
cls.credentials = ${credentials}
cls.network = ${network}
cls.testcase = ${testcase}
def test_plugin(self):
self.create_cluster()
% for check in testcase['scenario']:
self.check_${check}()
% endfor
</%def>
By default concurrency will be equal to number of cpu cores. This value
can be changed in YAML file.
We are going to use new integration tests for CI as soon as they are completely
implemented.
Alternatives
------------
We can use current integration tests for further testing Sahara but they
don't have sufficient coverage of Sahara use cases.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
Developers will be able much better to test their changes in Sahara.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
sreshetniak
Other contributors:
ylobankov
slukjanov
Work Items
----------
The work items will be the following:
* Add python code to sahara/tests
* Add examples of test scenarios
* Add documentation for new integration tests
Dependencies
============
Mako
tempest-lib
Testing
=======
None
Documentation Impact
====================
We need to add a note about new tests in Sahara documentation.
References
==========
None

153
specs/kilo/security-guidelines-doc.rst
View File

@ -1,153 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=============================================
Creation of Security Guidelines Documentation
=============================================
https://blueprints.launchpad.net/sahara/+spec/security-guidelines-doc
As Sahara increases in functionality and complexity, the need for clear,
concise documentation grows. This is especially true for security related
topics as they are currently under-represented. This specification
proposes the creation of a document that will provide a source for security
related topics, guides, and instructions as they pertain to Sahara.
Problem description
===================
There is currently a distinct lack of centralized, security related,
documentation for Sahara. Several features have been implemented to address
security shortfalls and they could use broadened discussions of their
application and proper usage. Additionally there is no current documentation
which discusses the specific procedures for securing the individual plugin
technologies at use within Sahara.
Proposed change
===============
This specification proposes the creation of Sahara specific documentation
addressing the security concerns of users and operators. This documentation
will cover current features, best practices, and guidelines for installing and
operating Sahara.
The documentation generated by this effort will be included in the OpenStack
Security Guide[1]. Bug patches will be generated against the current OpenStack
manuals, and the OpenStack Security Group will be engaged with respect to
finalizing and including the documentation.
The process will be broken down into sub-chapter sections that will make up a
Sahara chapter for the OpenStack Security Guide. Initially these sub-chapters
will include; Sahara controller installs, current feature discussions, and
plugin specific topics.
The information provided is intended to be updated as new methodologies,
plugins, and features are implemented. It will also be open to patching
through the standard OpenStack workflows by the community at large.
Alternatives
------------
Creation of a separate document managed by the Sahara team outside the purview
of the OpenStack manuals, and distributed through the Sahara documentation.
This solution would be non-ideal as it creates an alternate path for OpenStack
manuals that is outside the expected locations for end users.
Creation of a separate document as above with the exception that it will be
maintained with the other OpenStack manuals. This option might be more
plausible than the previous, but it still suffers from the problem of creating
an alternate location for security related guidelines that is separate from
the official manual. It also bears the burden of creating a new project
within the manuals infrastructure.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
mimccune (Michael McCune)
Other contributors:
None
Work Items
----------
* Create a bug against the OpenStack manuals for Sahara chapter inclusion
* Create the documentation and change requests for the following sub-chapters:
* Sahara controller install guides, with security related focus
* Feature discussions and examples
* plugin specific topics
* Hadoop
* Spark
* Storm
Dependencies
============
None
Testing
=======
Format testing as provided by the security guide project.
Documentation Impact
====================
This specification proposes the creation of hypertext and PDF documentation.
References
==========
[1]: http://docs.openstack.org/security-guide/content/ch_preface.html

141
specs/kilo/spark-cleanup.rst
View File

@ -1,141 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==============================================
Spark Temporary Job Data Retention and Cleanup
==============================================
https://blueprints.launchpad.net/sahara/+spec/spark-cleanup
Creates a configurable cron job at cluster configuration time to clean up data
from Spark jobs, in order to ease maintenance of long-lived clusters.
Problem description
===================
The current Spark plugin stores data from any job run in the /tmp directory,
without an expiration policy. While this is acceptable for short-lived
clusters, it increases maintenance on long-running clusters, which are likely
to run out of space in time. A mechanism to automatically clear space is
needed.
Proposed change
===============
On the creation of any new Spark cluster, a script (from
sahara.plugins.spark/resources) will be templated with the following
variables (which will be defined in Spark's config_helper module and thus
defined per cluster):
* Minimum Cleanup Seconds
* Maximum Cleanup Seconds
* Minimum Cleanup Megabytes
That script will then be pushed to /etc/hadoop/tmp_cleanup.sh. In the
following cases, no script will be pushed:
1) Maximum Cleanup Seconds is 0 (or less)
2) Minimum Cleanup Seconds and Minimum Cleanup Megabytes are both 0 (or less)
Also at cluster configuration time, a cron job will be created to run this
script once per hour.
This script will iterate over each extant job directory on the cluster; if it
finds one older than Maximum Cleanup Seconds, it will delete that directory.
It will then check the size of the set of remaining directories. If there is
more data than Minimum Cleanup Megabytes, then it will delete directories
older than Minimum Cleanup Seconds, starting with the oldest, until the
remaining data is smaller than Minimum Cleanup Megabytes or no sufficiently
aged directories remain.
Alternatives
------------
Any number of more complex schemes could be developed to address this problem,
including per-job retention information, data priority assignment (to
effectively create a priority queue for deletion,) and others. The above plan,
however, while it does allow for individual cluster types to have individual
retention policies, does not demand excessive maintenance or interface with
that policy after cluster creation, which will likely be appropriate for most
users. A complex retention and archival strategy exceeds the intended scope of
this convenience feature, and could easily become an entire project.
Data model impact
-----------------
None; all new data will be stored as cluster configuration.
REST API impact
---------------
None; operative Spark cluster template configuration parameters will be
documented the current interface allows this change.
Other end user impact
---------------------
None.
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
None. (config_helper.py variables will be automatically represented in
Horizon.)
Implementation
==============
Assignee(s)
-----------
Primary assignee:
egafford
Work Items
----------
* Creation of periodic job
* Creation of deletion script
* Testing
Dependencies
============
None
Testing
=======
Because this feature is entirely Sahara-internal, and requires only a remote
shell connection to the Spark cluster (without which many, many other tests
would fail) I believe that Tempest tests of this feature are unnecessary. Unit
tests should be sufficient to cover this feature.
Documentation Impact
====================
The variables used to set retention policy will need to be documented.
References
==========
https://blueprints.launchpad.net/sahara/+spec/spark-cleanup

167
specs/kilo/spec-repo-backlog-refactor.rst
View File

@ -1,167 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=========================================
Specification Repository Backlog Refactor
=========================================
https://blueprints.launchpad.net/sahara/+spec/add-backlog-to-specs
This specification proposes the refactoring of the `sahara-specs` repository
to inform a new workflow for backlogged features. This new workflow will
increase the visibility of specifications that have been approved but which
require implementation.
Problem description
===================
Currently the `sahara-specs` repository suggests a layout that creates
"implemented" and "approved" subdirectories for each release. This pattern
could create duplicates and/or empty directories as plans that have been
approved for previous releases but have not been completed are moved and
copied around as necessary.
Additionally, this pattern can increase the barrier for new contributors
looking for features to work on and the future direction of the project. By
having a single location for all backlogged specifications it will increase
visibility of items which require attention. And the release directories
can be devoted entirely to specifications implemented during those cycles.
Proposed change
===============
This change will add a backlog directory to the specs directory in the
repository, and refactor the juno directory to contain only the specs
implemented in juno.
It will also update the documentation to clearly indicate the usage of the
backlog directory and the status of the release directories. This
documentation will be part of the root readme file, a new document in the
backlog directory, and a reference in the main documentation.
This change is also proposing a workflow that goes along with the
repository changes. Namely that any work within a release that is either
predicted to not be staffed, or that is not started at the end of a
release should be moved to the backlog directory. This process should be
directed by the specification drafters as they will most likely be the
primary assignees for new work. In situations where the drafter of a
specification feels that there will be insufficient resources to create
an implementation then they should move an approved specification to the
backlog directory. This process should also be revisited at the end of a
release cycle to move all specifications that have not been assigned to the
backlog.
Alternatives
------------
Keep the directory structure the way it is currently. This does not improve
the status of the repository but is an option to consider. If the
directory structure is continued as currently configured then the release
directories will need to create additional structures for each cycle's
incomplete work.
Refactor each release directory to contain a backlog. This is very similar to
leaving things in their current state with the exception that it changes
the names in the directories. This change might add a small amount of
clarification but it is unlikely as the current names for "implemented" and
"approved" are relatively self explanatory.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
The main impact will be for crafters of specifications to be more aware of
the resource requirements to implement the proposed changes. And potentially
move their specifications based on those requirements.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
mimccune (Michael McCune)
Work Items
----------
* Create the backlog directory and documentation.
* Clean up the juno directory.
* Add references to backlog in the contributing documentation.
Dependencies
============
None
Testing
=======
None
Documentation Impact
====================
The "How to Participate" document should be updated to make reference to
the backlog directory as a place to look for new work.
References
==========
Keystone is an example of another project using this format for backlogged
work. Examples of the documentation for the backlog directory[1] and the
root documentation[2] will be used as reference in the creation of Sahara
specific versions.
[1]: https://github.com/openstack/keystone-specs/tree/master/specs/backlog
[2]: https://github.com/openstack/keystone-specs

170
specs/kilo/storm-integration.rst
View File

@ -1,170 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=================
Storm Integration
=================
https://blueprints.launchpad.net/sahara/+spec/storm-integration
This blueprint aims to implement Storm as a processing option in Sahara.
Storm is a real time processing framework that is highly used in the big
data processing community.
Problem description
===================
Sahara today is only able to process big data in batch. Storm provides a
easy setup stream processing. Having storm integrated with Sahara gives sahara
a new feature, so not only users will be able to process batch but also real
time data.
Proposed change
===============
* The implementation is divided in three steps:
* First we need to implement Storm Plugin.
* Identify Storm configuration files and manage their creation via sahara;
* Create the plugin itself to manage Storm deploy;
* Second, we need to create a new Job Manager for storm, following Trevor
McKay's refactoring
* Last we will create a new image that comes with storm installed.
* Node Groups:
* Storm has two basic components **Nimbus** and **Supervisor**, Nimbus is
the master node, hosts the UI and is the main node of the topology.
Supervisors are the worker nodes. Other than that, storm needs only
zookeeper to run, we need have it as well.
The basic Node Groups that we will have are:
* Nimbus;
* Supervisor;
* Nimbus and Supervisor;
* Zookeeper;
* Nimbus and Zookeeper;
* Supervisor and Zookeeper;
* Nimbus, Supervisor and Zookeeper
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
Sahara-image-elements may be the place to create storm image, it will need
deeper investigation but looks like the best place to create and publish
storm images.
Sahara-dashboard / Horizon impact
---------------------------------
The first changes needed here is to show the configuration options of Storm
when this type of job is selected.
There will be also necessary to have a deploy job where the user can submit
the topology jar and the command line to run it.
As for now, I don't see other changes in the UI, the jobs are very simple and
need no special configurations.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
* tellesmvn
Work Items
----------
The implementation is divided in three steps:
* First we need to implement Storm Plugin.
* Storm plugin is similar to Spark plugin. The implementation will be based
on Spark's but necessary changes will be made.
* Storm doesn't rely on many configuration files, there is only one needed
and it is used by all nodes. This configuration file is written in YAML
and it should be dynamically written in the plugin since it needs to have
the name or ip of the master node and also zookeeper node(s). We will need
PYYAML to parse this configuration to YAML. PYYAML is already a global
requirement of OpenStack and will be added to Sahara's requirement as well.
* The plugin will run the following processes:
* Storm Nimbus;
* Storm Supervisor;
* Zookeeper.
* Second, we need to create a new Job Manager for storm, following Trevor
McKay's refactoring
* This implementation is under review and the details can be seen here:
https://review.openstack.org/#/c/100678/
* Last we will create a new image that comes with storm installed.
* This part is yet not the final decision, but it seems to me that it is
better to have a prepared image with storm than having to install it
every time a new cluster is set up.
Dependencies
============
* https://review.openstack.org/#/c/100622/
Testing
=======
I will write Unit Tests, basically to test the write configuration file and
more tests can be added as needed.
Documentation Impact
====================
We will need to add Storm as a new plugin in the documentation and write how
to use the plugin.
Also a example in sahara-extra on how to run a storm job will be provided
References
==========
* `Wiki <https://wiki.openstack.org/wiki/HierarchicalMultitenancy>`
* `Etherpad <https://etherpad.openstack.org/p/juno-summit-sahara-edp>`
* `Storm Documentation <http://storm.incubator.apache.org/>`

142
specs/kilo/support-cinder-api-v2.rst
View File

@ -1,142 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
============================
Support Cinder API version 2
============================
https://blueprints.launchpad.net/sahara/+spec/support-cinder-api-v2
This specification proposes to add support for the second version of the Cinder
API, which brings useful improvements and will soon replace version one.
Problem description
===================
Currently Sahara uses only version 1 of Cinder API to create volumes. Version
two, however, brings useful features such as scheduler hints, more consistent
responses, caching, filtering, etc.
Also, Cinder is deprecating version 1 in favor of 2, so supporting both would
make switching easier for users.
Use cases:
* As a developer I want to be able to pass scheduler hints to Cinder when
creating clusters, in order to choose volumes more precisely and achieve
improved performance.
* As a developer I want filtering into my requests to Cinder to make queries
lighter.
* As a deployer I want to be able to choose between legacy Cinder API v1 and
newer v2 API.
Proposed change
===============
The implementation will add a configuration option, cinder_api_version, that
will be defaulted to:
``cinder_api_version=1``
but can be changed to
``cinder_api_version=2``
by modifying sahara.conf.
The client() method in sahara/utils/openstack/cinder.py will either return
clientv1() or clientv2() depending on the configuration.
Alternatives
------------
Wait for Cinder API v1 to be deprecated and switch abruptly to v2.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
* If the deployer wants to keep using Cinder API version 1, nothing has to be
done.
* If the deployer wants to upgrade to version 2, the cinder_api_version option
in sahara.conf should be overwritten.
Developer impact
----------------
Developers can read CONF.cinder_api_version to know what API version is being
used.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
adrien-verge
Work Items
----------
* Add a configuration option: cinder_api_version.
* Put some magic in sahara/utils/openstack/cinder.py to pick the correct Cinder
client depending on the configuration.
Dependencies
============
None
Testing
=======
Same as for v1 API.
Documentation Impact
====================
None
References
==========
* https://wiki.openstack.org/wiki/CinderAPIv2

153
specs/kilo/support-cinder-availability-zones.rst
View File

@ -1,153 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=================================
Support Cinder availability zones
=================================
https://blueprints.launchpad.net/sahara/+spec/support-cinder-availability-zones
Extend API to support specifying Cinder availability zones where to create
volumes.
Problem description
===================
It can be desirable to assign Cinder availability zones to node groups, in
order to have fine-grained cluster volumes topologies.
Use case:
* As a end user I want namenode volumes to be spawned in the regular AZ and
datanode volumes in a high-performance ``ssd-high-io`` AZ.
Proposed change
===============
It adds a new ``volumes_availability_zone`` property in NodeGroup and
NodeGroupTemplate objects. When set, it modifies the direct and Heat engines
to force creating of volumes into the right AZ.
Alternatives
------------
None
Data model impact
-----------------
This change will add ``volumes_availability_zone`` columns in sahara database,
next to ``volumes_per_node`` and ``volumes_size``. Impacted tables are
``node_groups``, ``node_group_templates`` and ``templates_relations``.
A database migration will accompany this change.
REST API impact
---------------
Each API method which deals with node groups and node groups templates will
have an additional (and optional) ``volumes_availability_zone`` parameter,
which will be taken into account if ``volumes_per_node`` is set and non-zero.
Example::
{
"name": "cluster1",
"node_groups": [
{
"name": "master",
"count": 1,
},
{
"name": "worker",
"count": 3,
"volumes_per_node": 1,
"volumes_size": 100,
"volumes_availability_zone": "ssd-high-io"
}
]
}
Other end user impact
---------------------
python-saharaclient should be modified to integrate this new feature.
Deployer impact
---------------
Needs to migrate DB version using:
``sahara-db-manage --config-file /etc/sahara/sahara.conf upgrade head``
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
sahara-dashboard should be modified to integrate this new feature.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
adrien-verge
Work Items
----------
* Add a ``volumes_availability_zone`` property in NodeGroup and
NodeGroupTemplate objects.
* When a user specifies the (optional) volumes_availability zone, check its
existence.
* In the direct engine, include the ``volumes_availability_zone`` argument in
the call to cinder.client().volumes.create().
* In the Heat engine, add the ``availability_zone`` property in
sahara/resources/volume.heat.
Dependencies
============
None
Testing
=======
* Test cluster creation with ``volumes_availability_zone`` specified.
* Test cluster creation with wrong ``volumes_availability_zone`` specified.
Documentation Impact
====================
Documentation will need to be updated, at sections related to node group
template creation and cluster creation.
References
==========
None

140
specs/kilo/support-nova-availability-zones.rst
View File

@ -1,140 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===============================
Support Nova availability zones
===============================
https://blueprints.launchpad.net/sahara/+spec/support-nova-availability-zones
Extend API to support specifying Nova availability zones where to spawn
instances.
Problem description
===================
It can be desirable to assign Nova availability zones to node groups, in order
to have fine-grained clusters topologies.
Use cases:
* As a end user I want namenode instances to be spawned in the regular ``nova``
AZ and datanode instances in my high-performance ``nova-highperf`` AZ.
* As a end user I want instances from node-group A to be all together in the
``nova-1`` AZ, separated from instances from node-group B in ``nova-2``.
Proposed change
===============
The proposed change is already implemented at [1].
It adds an ``availability_zone`` property in NodeGroup and NodeGroupTemplate
objects. When set, it modifies the direct and Heat engines to force spawning
of instances into the right AZ.
Alternatives
------------
None
Data model impact
-----------------
This change will add ``availability_zone`` columns in the sahara database
(``node_groups``, ``node_group_templates`` and ``templates_relations`` tables).
A database migration will accompany this change.
REST API impact
---------------
Each API method which deals with node groups and node groups templates will
have an additional (and optional) ``availability_zone`` parameter.
Other end user impact
---------------------
python-saharaclient should be modified to integrate this new feature.
Deployer impact
---------------
Needs to migrate DB version using:
``sahara-db-manage --config-file /etc/sahara/sahara.conf upgrade head``
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
sahara-dashboard should be modified to integrate this new feature.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
adrien-verge
Work Items
----------
The proposed change is already implemented at [1].
* Add an ``availability_zone`` property in NodeGroup and NodeGroupTemplate
objects.
* When a user specifies the (optional) availability zone, check its existence.
* In the direct engine, include the ``availability_zone`` argument in the call
to nova.client().servers.create().
* In the Heat engine, add the ``availability_zone`` property in
sahara/resources/instance.heat.
Dependencies
============
None
Testing
=======
* Test cluster creation without availability zone specified.
* Test cluster creation with availability zone specified.
* Test cluster creation with wrong availability zone specified.
Documentation Impact
====================
Documentation will need to be updated, at sections related to node group
template creation and cluster creation.
References
==========
* [1] https://review.openstack.org/#/c/120096

178
specs/kilo/support-query-filtering.rst
View File

@ -1,178 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Spec - Add support for filtering results
==========================================
Blueprints:
Sahara service: https://blueprints.launchpad.net/sahara/+spec/enable-result-filtering
Client library: https://blueprints.launchpad.net/python-saharaclient/+spec/enable-result-filtering
Horizon UI: https://blueprints.launchpad.net/horizon/+spec/data-processing-table-filtering
As result sets can be very large when dealing with nontrivial openstack
environments, it is desirable to be able to filter those result sets by
some set of field constraints. This spec lays out how Sahara should support
such queries. Changes will be required in the API, client library,
and in the Horizon UI.
Problem description
===================
The original use case for this change came up when we wanted to provide a
way to filter some of the tables in the UI. In order to make that filtering
possible, either the UI has to perform the filtering (inefficient since
large result sets would still be returned by the api) or the api/client
library need to provide support for such filtering. Upon review of other
api/client libraries, it looks like most, if not all,
provide filtering capabilities. Sahara should also make filtering possible.
Proposed change
===============
In order to make filtering possible, each of the "list()" methods in the
client library should be extended to take an optional (default=None)
parameter, "search_opts". "search_opts" should be a dict that will include
one or more {field:query} entries. Pagination can also be handled through
search_opts by allowing for the setting of page/page_size/max results.
In addition, the REST api needs to be extended for the list() operations to
support query syntax. This will require changes to how Sahara currently
does its database queries to include query options. Currently,
they are set to just return all entries for each call.
Alternatives
------------
Filtering could be done entirely in the UI, but that is a wasteful way to do
it since each library/api call will still return the full result set.
Also, any users of the REST api or the client library would not benefit
from this functionality.
It would also be possible to just implement filtering at the client library
level, but for the same reasons that doing it in the UI is undesirable,
this approach is also suboptimal.
Data model impact
-----------------
No changes to the model itself should be required. Only changes to how we
query the database are needed.
REST API impact
---------------
The rest api methods for the list operations will need to be changed to
support query-style parameters. The underlaying database access methods
will need to be changed from the basic "get all" functionality to support a
`**kwargs` parameter (this is already done in cluster_get_all(),
but needs to be done for the other methods).
There are a pair of special cases to be considered for filtering on the Job
Executions table. The "job" and "cluster" columns actually contain
information that are not part of the job execution object. For filters on
those fields, a field value of "job.name" or "cluster.name" should be passed
down. That will trigger the database query to be joined to a filter on the
name property of either the job or cluster table.
Additionally, there is special handling required to search on the Job
Executions status field. This is because status is not a proper field in
the Job Execution itself, but rather it is part of the "info" field. To
handle this, a bit of manual filtering will be done against the info.status
field.
Other end user impact
---------------------
Users of the REST API, client library and Horizon UI will be able to filter
their result sets via simple queries.
Deployer impact
---------------
These changes should not affect deployment of the Sahara service.
Developer impact
----------------
No developer impact.
Sahara-image-elements impact
----------------------------
No sahara-image-elements impact.
Sahara-dashboard / Horizon impact
---------------------------------
Once the REST API and client library changes are in place,
the Horizon UI will be modified to allow for filtering on the following
tables: Clusters, Cluster Templates, Node Group Templates, Job Executions,
Jobs. It would also be possible to filter any of our tables in the future
without any other API/client library changes.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
croberts (Horizon UI)
Other contributors:
tmckay (REST API/sahara.db.api)
croberts (Client Library)
Work Items
----------
These may be able to be worked on in parallel, but they must be released in
the order they are given below.
1. REST API implementation
2. Client library implementation
3. Horizon UI implementation
Dependencies
============
None
Testing
=======
There should be tests added against the REST interface,
client library and Horizon UI to test the filtering capabilities being added.
Documentation Impact
====================
Docs for the REST API will need to be updated to reflect the querying
functionality.
Docs for the client library will need to be updated to reflect the querying
functionality.
The dashboard user guide should be updated to note the table filtering
functionality that will be added.
References
==========
Sahara service: https://blueprints.launchpad.net/sahara/+spec/enable-result-filtering
Client library: https://blueprints.launchpad.net/python-saharaclient/+spec/enable-result-filtering
Horizon UI: https://blueprints.launchpad.net/horizon/+spec/data-processing-table-filtering

174
specs/kilo/support-template-editing.rst
View File

@ -1,174 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Spec - Add support for editing templates
==========================================
Blueprints:
Sahara Service: https://blueprints.launchpad.net/sahara/+spec/support-template-editing
Horizon UI: https://blueprints.launchpad.net/horizon/+spec/data-processing-edit-templates
Currently, once a node group template or a cluster template has been
created, the only operations available are "copy" or "delete". That has
unfortunate consequences on a user's workflow when they want to change a
template. For example, in order to make even a small change to a node group
template that is used by a cluster template, the user must make a copy of
the node group template, change it, save it, and then create a new (or copy)
cluster template that uses the new node group template. Ideally,
a user could just edit the template in place and move on.
Problem description
===================
Sahara currently lacks implementation for the update operation for both node
group and cluster templates. In order to provide an implementation for
these operations, the following issues must be addressed.
1. What to do with existing clusters that have been built by using a
template that is being edited. Would we automatically scale a cluster if a
cluster template was changed in a way that added/removed nodes? Would we
shut down or start up processes if a node group template changed to
add/remove processes?
I propose that for this iteration, a user may not change any templates that
were used to build a currently running cluster. This is consistent with the
rule currently in place to not allow deletion of templates that were are
used by a currently active cluster. A future iteration could remove that
restriction for both delete and edit.
A user could still make a copy of a template and edit that if they wanted to
start a new cluster with the altered version of the template.
2. Make sure that all cluster templates that are dependant upon an edited
node group template will pick-up the changes to the node group template.
This is only a problem if we go the route of allowing templates used by an
active cluster to be edited. In that case, we would need to change the
existing templates to reference the newly created template.
Proposed change
===============
The put (update) method will be implemented for both node group and cluster
templates. Currently, the stubs are there, but they just return "Not
Implemented".
I think that the simplest method of sending an updated template is to send
the entire new template rather than just a diff. It could be slightly
inefficient to do it this way, but avoids the need to build-in the diff
logic in the UI. The current client library methods for update() seem to be
anticipating that sort of implementation. If we went to sending a diff,
we may need to adjust the client library.
Alternatives
------------
Updates could be achieved by doing a delete/add combination,
but that is not a very good solution to this problem since any given node
group or cluster template could be referenced by another object. Deleting
and adding would require updating each referencing object.
If we need to be able to edit templates that are used by an active cluster,
the edited version of the template will retain the ID and name of the
original template. Prior to overwriting the original template,
it will be saved with a new ID and some form of "dotted" name to
indicate the version ("origname.1"). All running clusters would be changed
to reference the original template with the new ID.
Data model impact
-----------------
N/A
REST API impact
---------------
N/A.
The update operation are already defined in the API, but they are not yet
implemented.
Other end user impact
---------------------
The Horizon UI will be updated to include "Edit" as an operation for each
node group and cluster template. The edit button will appear in the list of
operations in each row of the table.
The python-saharaclient library already defines the update() methods that
will be used for implementing this feature in the Horizon UI. No changes to
python-saharaclient are anticipated to support this feature.
Deployer impact
---------------
N/A
Developer impact
----------------
N/A
Sahara-image-elements impact
----------------------------
N/A
Sahara-dashboard / Horizon impact
---------------------------------
Horizon will be updated to include an "edit" button in each row of both the
node group and cluster templates tables. That edit button will bring up the
edit form (essentially the "create" form, but with all the values
pre-populated from the existing template). Clicking on "Save" from the edit
form will result in a call to the node group/cluster template update()
method in the python-saharaclient library.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
croberts
Other contributors:
tmckay
Work Items
----------
Horizon UI: croberts
Sahara service: tmckay
Dependencies
============
N/A
Testing
=======
There will be unit and integration tests added in Sahara.
Horizon will have a test added to the node group and cluster template panels
to verify that the appropriate forms are generated when edit is chosen.
Documentation Impact
====================
The Sahara UI user guide should be updated to note the availability of edit
functionality for templates.
References
==========
N/A

126
specs/kilo/volume-instance-locality.rst
View File

@ -1,126 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=============================================
Cinder volume instance locality functionality
=============================================
https://blueprints.launchpad.net/sahara/+spec/volume-instance-locality
This specification proposes to add an opportunity to have an instance and an
attached volumes on the same physical host.
Problem description
===================
Currently there is no way to specify that volumes are attached to the same
physical host as the instance. It would be nice to have this opportunity.
Proposed change
===============
This feature could be done with Cinder InstanceLocalityFilter which allows to
request creation of volumes local to instance. It would increase performance
of I/O operations.
There will be several changes:
* Boolean field ``volume_local_to_instance`` will be added to every node group
template, node group and templates relation. This field will be optional and
``False`` by default.
* If ``volume_local_to_instance`` is set to ``True``, Cinder volumes will be
created on the same host.
* If ``volume_local_to_instance`` is ``True``, all instances of the node group
should be created on hosts with free disk space >= ``volumes_per_node`` *
``volumes_size``. If it cannot be done, error should be occurred.
Alternatives
------------
None
Data model impact
-----------------
``volume_local_to_instance`` field should be added to ``node_groups``,
``node_group_templates``, ``templates_relations``.
REST API impact
---------------
* API will be extended to support ``volume_local_to_instance`` option.
``volume_local_to_instance`` is optional argument with ``False`` value by
default, so this change will be backward compatible.
* python client will be updated
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
New field to select ``volume_local_to_instance`` option during node group
template creation will be added.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
apavlov-n
Work Items
----------
* Adding ability to create instance and volumes on the same host;
* Adding ability to create instance on appropriate host;
* Updating documentation;
* Updating UI;
* Updating python client;
* Adding unit tests.
Dependencies
============
None
Testing
=======
Unit tests will be added.
Documentation Impact
====================
Documentation will be updated. Will be documented when this feature can be
used and when it cannot. Also will be noted how to enable it on Cinder side.
References
==========
* http://docs.openstack.org/developer/cinder/api/cinder.scheduler.filters.instance_locality_filter.html

8
specs/kilo_idx.rst
View File

@ -1,8 +0,0 @@
Kilo specs
^^^^^^^^^^
.. toctree::
:glob:
:maxdepth: 1
kilo/*

120
specs/liberty/adding-custom-scenario-tests.rst
View File

@ -1,120 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
========================================
Adding custom scenario to scenario tests
========================================
https://blueprints.launchpad.net/sahara/+spec/custom-checks
This specification proposes to add custom tests to scenario tests for more
exhaustive testing of Sahara.
Problem description
===================
Now, scenario tests testing of basic functionality and user can not add
custom personal tests for check of other functionality in Sahara.
Extra tests should be added:
* checks for mount and available cinder volumes;
* checks for started services on cluster;
* checks for other processes that now not testing
Proposed change
===============
Custom test need add to sahara/tests/scenario/custom_checks and need implement
support of this scenarios in scenario tests.
For implementation this spec, need change field parameters for field "scenario"
in scenario tests. Now is type "enum", need change to "string" for adding
ability set custom tests.
Additionally, should be rewrite sahara/tests/scenario/testcase.py.mako
template. Custom tests will be called from module with name in format
`check_{name of check}` with method `check()` inside.
All auxiliary methods for current custom check will be written in module with
this tests. Methods, for global using in several custom scenario can be
implemented in sahara/tests/scenario/base.py.
Alternatives
------------
Tests can be added manually to scenario tests in Base class.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
esikachev
Work Items
----------
* Adding ability to run custom scenario tests;
* Move scripts from old integration tests to scenario tests as custom checks;
* Adding new custom checks.
Dependencies
============
None
Testing
=======
None
Documentation Impact
====================
None
References
==========
None

167
specs/liberty/adding-ids-to-logs.rst
View File

@ -1,167 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=========================================================
Adding cluster/instance/job_execution ids to log messages
=========================================================
https://blueprints.launchpad.net/sahara/+spec/logs-improvement
This specification proposes to add more information to Sahara logs.
Problem description
===================
Looking at some Sahara logs it is difficult to determine what
cluster/instance/job_execution to which they refer.
Extra information should be added:
* logs associated with cluster creation/scaling/deletion should contain
cluster id;
* logs associated with job execution/canceling/deletion should contain
job execution id;
* logs associated with operations executed on specific instance should
contain id of this instance.
Proposed change
===============
Information can be added to context resource_uuid field and then can be
used by ContextAdapter in openstack.common.log for a group of logs.
This change requires additional saving of context in
openstack.common.local.store to access context from openstack.common.log
We need to set cluster id and job execution id only once. So, it could be done
with 2 methods that will be added to sahara.context:
set_current_cluster_id(cluster_id) and set_current_job_execution_id(je_id)
Additionally, instances and their ids are changing during the thread. So,
instance id should be set only when operation executes on this instance.
It will be provided by class SetCurrentInstanceId and will be used with
wrapping function set_current_instance_id(instance_id) this way:
.. sourcecode:: python
with set_current_instance_id(instance_id):
..
Code inside "with" statement will be executed with new context (which
includes instance id in resource_uuid field) but outside of it context will
stay the same.
If instance and cluster specified, log message will looks like:
.. sourcecode:: console
2014-12-22 13:54:19.574 23128 ERROR sahara.service.volumes [-] [instance:
3bd63e83-ed73-4c7f-a72f-ce52f823b080, cluster: 546c15a4-ab12-4b22-9987-4e
38dc1724bd] message
..
If only cluster specified:
.. sourcecode:: console
2014-12-22 13:54:19.574 23128 ERROR sahara.service.volumes [-] [instance:
none, cluster: 546c15a4-ab12-4b22-9987-4e38dc1724bd] message
..
If job execution specified:
.. sourcecode:: console
2014-12-22 13:54:19.574 23128 ERROR sahara.service.edp.api [-] [instance:
none, job_execution: 9de0de12-ec56-46f9-80ed-96356567a196] message
..
Field "instance:" is presented in every message (even if it's not necessary)
because of default value of instance_format='[instance: %(uuid)s] '
that cannot be fixed without config changing.
After implementation of this changes, Sahara log messages should be checed and
fixed to avoid information duplication.
Alternatives
------------
Information can be added manually to every log message.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
apavlov-n
Work Items
----------
* Adding ability to access context from openstack.common.log;
* Adding information about cluster/instance/job execution ids to context;
* Fixing log messages to avoid information duplication.
Dependencies
============
None
Testing
=======
None
Documentation Impact
====================
None
References
==========
None

147
specs/liberty/allow-creation-of-multiple-cluster-simultaneously.rst
View File

@ -1,147 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
======================================================
Allow the creation of multiple clusters simultaneously
======================================================
https://blueprints.launchpad.net/sahara/+spec/simultaneously-creating-multiple-clusters
We want to improve user experience when creating new clusters by allowing the
user to create multiple clusters at the same time.
Problem description
===================
When creating new clusters, the user has only the option to start a single
cluster. In some cases, for example when dealing with researches, the user
needs more than a single cluster with a given template. In order to reduce the
work of creating a cluster then going back to create a second one and so on, we
want to include the option of creating multiple clusters simultaneously by
adding an option of number of clusters.
Proposed change
===============
We want to introduce an option for the user to select how many clusters will be
spawned.
When creating multiple clusters we will add a sequential number to the given
cluster name (hadoop-cluster1, hadoop-cluster2, ...).
The creation workflow would go as follows:
* 1) The users will request the creation of multiple clusters
POST v1.1/<project_id>/clusters/multiple using a body as described below.
* 2) The return of this call will be a list of clusters id
* 3) Finally the user will be able to track the cluster state using the ids.
Alternatives
------------
The user keeps creating a cluster at a time.
Data model impact
-----------------
None.
REST API impact
---------------
We need to create a new API call (create_multiple_clusters()) to allow the
creation of multiple clusters by passing a new parameter specifying the number
of clusters that will be created.
POST v1.1/<project_id>/clusters/multiple
Other end user impact
---------------------
We will also need to change the python-saharaclient to allow the creation of
multiple clusters.
* Request
{
"plugin_name": "vanilla",
"hadoop_version": "2.4.1",
"cluster_template_id": "1beae95b-fd20-47c0-a745-5125dccbd560",
"default_image_id": "be23ce84-68cb-490a-b50e-e4f3e340d5d7",
"user_keypair_id": "doc-keypair",
"name": "doc-cluster",
"count": 2,
"cluster_configs": {}
}
* Response
{clusters: ["c8c3fee5-075a-4969-875b-9a00bb9c7c6c",
"d393kjj2-973b-3811-846c-9g33qq4c9a9f"]
}
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
We need to add a box to allow the user to insert the number of clusters that
will be created (default set to 1).
Implementation
==============
Assignee(s)
-----------
Primary assignee:
tellesmvn
Work Items
----------
* Implement the backend for creating multiple clusters
* Implement the API change
* Implement Unit tests
* Implement changes to the python-saharaclient
* Implement changes to the UI
* Update WADL file in the api-site repo
Dependencies
============
None.
Testing
=======
We will implement unit tests.
Documentation Impact
====================
The documentation needs to be updated datailing the new option.
References
==========
None.

124
specs/liberty/api-for-objects-update.rst
View File

@ -1,124 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
====================================
Objects update support in Sahara API
====================================
https://blueprints.launchpad.net/sahara/+spec/api-for-objects-update
This specification proposes to add api calls that allows to update objects that
currently cant be updated this way.
Problem description
===================
Current Sahara API doesn't support update of some objects, which can be
required by some other features (for example it's needed for shared and
protected resources implementation that will be proposed in ACL spec later).
Updates are already implemented for node group templates, cluster templates,
job binaries, data sources and should be done for clusters,
jobs, job executions and job binary internals.
Proposed change
===============
Update operation will be added for cluster, job, job execution and job binary
internal objects.
For clusters and jobs only description and name update will be allowed for now.
For job binary internals only name will be allowed to update.
There is nothing will be allowed to update for job executions, only
corresponding methods will be added.
Also will be added support of PATCH HTTP method to modify existing resources.
It will be implemented the same way as current PUT method.
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
Following API calls will be added:
**PATCH /v1.1/{tenant_id}/clusters/{cluster_id}**
**PATCH /v1.1/{tenant_id}/jobs/{job_id}**
**PATCH /v1.1/{tenant_id}/job-executions/{job_execution_id}**
**PATCH /v1.1/{tenant_id}/job-binary-internals/{job_binary_internal_id}**
Other end user impact
---------------------
This update methods will be added to saharaclient API.
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
This update methods will not be added to Horizon yet, but will be added later
as part of ACL spec.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
apavlov-n
Work Items
----------
* Adding PATCH method;
* Adding new API calls;
* Adding operations to saharaclient;
* Documentation update in api-ref.
Dependencies
============
None
Testing
=======
Unit tests and API tests in tempest will be added.
Documentation Impact
====================
Sahara REST API documentation in api-ref will be updated.
References
==========
None

156
specs/liberty/clients-calls-retry.rst
View File

@ -1,156 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
====================================
Retry of all OpenStack clients calls
====================================
https://blueprints.launchpad.net/sahara/+spec/clients-calls-retry
This specification proposes to add ability of retrying OpenStack clients calls
in case of occasional errors occurrence.
Problem description
===================
Sahara uses a bunch of OpenStack clients to communicate with other OpenStack
services. Sometimes during this clients calls can be occurred occasional
errors that lead to Sahara errors as well. If you make a lot of calls, it may
not be surprising if one of them doesn't respond as it should - especially for
a service under heavy load.
You make a valid call and it returns a 4xx or 5xx error. You make the same
call again a moment later, and it succeeds. To prevent such kind of failures,
all clients calls should be retried. But retries should be done only for
certain error codes, because not all of the errors can be avoided just with
call repetition.
Proposed change
===============
Swift client provides the ability of calls retry by its own. So, only number of
retries and retry_on_ratelimit flag should be set during client initialisation.
Neutron client provides retry ability too, but repeats call only if
``ConnectionError`` occurred.
Nova, Cinder, Heat, Keystone clients don't offer such functionality at all.
To retry calls ``execute_with_retries(method, *args, **kwargs)`` method will be
implemented. If after execution of given method (that will be passed with first
param), error occurred, its ``http_status`` will be compared with http statuses
in the list of the errors, that can be retried. According to that, client call
will get another chance or not.
There is a list of errors that can be retried:
* ``REQUEST_TIMEOUT (408)``
* ``OVERLIMIT (413)``
* ``RATELIMIT (429)``
* ``INTERNAL_SERVER_ERROR (500)``
* ``BAD_GATEWAY (502)``
* ``SERVICE_UNAVAILABLE (503)``
* ``GATEWAY_TIMEOUT (504)``
Number of times to retry the request to clients before failing will be taken
from ``retries_number`` config value (5 by default).
Time between retries will be configurable (``retry_after`` option in
config) and equal to 10 seconds by default. Additionally, Nova client provides
``retry_after`` field in ``OverLimit`` and ``RateLimit`` error classes, that
can be used instead of config value in this case.
These two config options will be under ``timeouts`` config group.
All clients calls will be replaced with ``execute_with_retries`` wrapper.
For example, instead of the following method call
.. sourcecode:: python
nova.client().images.get_registered_image(id)
it will be
.. sourcecode:: python
execute_with_retries(nova.client().images.get_registered_image, id)
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
apavlov-n
Work Items
----------
* Adding new options to Sahara config;
* ``execute_with_retries`` method implementation;
* Replacing OpenStack clients call with ``execute_with_retries`` method.
Dependencies
============
None
Testing
=======
Unit tests will be added. They will check that only specified errors will
be retried
Documentation Impact
====================
None
References
==========
None

145
specs/liberty/cluster-creation-with-trust.rst
View File

@ -1,145 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===========================================
Use trusts for cluster creation and scaling
===========================================
https://blueprints.launchpad.net/sahara/+spec/cluster-creation-with-trust
Creation of cluster can be a pretty long operation. Currently we use sessions
that usually expire in less than one hour. So, currently it is impossible to
create a cluster that requires more than one hour for spawn.
Sahara could get trust from user and use it whenever it is needed for cluster
creation or scaling.
Problem description
===================
Sahara communicates with OpenStack services using session provided by user.
Session is created by keystone for comparably small amount of time. Creation
of large cluster could require more than session life time. That's why Sahara
will not be able to operate cluster at the end of the process.
Proposed change
===============
Sahara could perform all operations with cluster using trust obtained from
user. Now trusts are used for termination of transient cluster. The same
logic could be used for all operations during cluster creation and scaling.
Since we still support keystone v2, the option should be configurable. I
suggest making it enabled by default.
The proposed workflow:
1. User requests cluster creation or scaling.
2. Sahara creates trust to be used for OpenStack operation. This trust is
stored in the DB in the cluster's trust_id field.
3. Sahara finishes cluster provisioning or the periodic cluster cleanup task
recognizes that cluster activation has timed out and uses the trust to
delete the cluster.
4. Sahara deletes the trust.
For safety reasons created trusts should be limited by time, but their life
time should be sufficient for cluster creation. Parameter in config file with
1 day default should work well.
Alternatives
------------
The trust id could be stored in memory rather than in the database. However,
this implementation would not allow the periodic cluster cleanup task (which
is not run in the context of the tenant cluster provisioning request) to
successfully delete stale clusters.
It is notable that storing the trust_id in the database will also be of use
to us in improving the HA capabilities of cluster creation if we move to a
multi-stage, DAG-based provisioning flow.
While potential concerns about storing trust ids in the DB exist, these ids
require a valid auth token for either the admin or tenant user to utilize,
adding some degree of security in depth in case of a control plane database
breach. This mechanism may be further secured by storing all trust ids (for
both transient and long-running clusters) via a secret storage module in the
future. This change, however, falls outside the scope of this specification.
Data model impact
-----------------
None.
REST API impact
---------------
None.
Other end user impact
---------------------
None.
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
alazarev (Andrew Lazarev)
Other contributors:
None
Work Items
----------
* Add option to Sahara config
* Implement new authentication strategy
* Document new behavior
Dependencies
============
None.
Testing
=======
Manually. CI will cover the feature.
Documentation Impact
====================
Because trusts are now being used to clean up clusters, we will need to
document that the periodic cluster cleanup task should be run on a schedule
that fits within the expiration period of a trust.
References
==========
None.

125
specs/liberty/deprecate-direct-engine.rst
View File

@ -1,125 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
============================
Deprecation of Direct Engine
============================
https://blueprints.launchpad.net/sahara/+spec/deprecate-direct-engine
Currently Sahara has two types of infrastructure engines. First is Direct
Infrastructure Engine and second is Heat Infrastructure Engine. This spec
proposes deprecating the Direct Engine.
Problem description
===================
Each time when Sahara start support new feature, Sahara should support it in
both engines. So, it became much harder to support both versions of engines,
because in case of Direct Engine it would need to have duplication of work,
which is already done in Heat.
Proposed change
===============
It's proposed to deprecate Direct Engine in Liberty release, but it will be
available to use. After merging this spec Direct Engine should be ``freezed``
for new feautures which will be added in Liberty. It will be opened for
fixes of ``High`` and ``Critical`` bugs. We should make Heat Engine used by
default in Sahara.
This change will allow to switch most testing jobs in Sahara CI
to use Heat Engine instead of Direct Engine.
In M release we should remove all operations from direct engine.
After that the only operation which can be done with direct-engine-created
cluster is the cluster deletion. We should rewrite cluster deletion behavior
to support deletion direct-engine-created cluster via Heat Engine. Now heat
engine removes cluster from database but doesn't remove all cluster elements
(for example, instances).
Alternatives
------------
Sahara can continue support of both engines.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
Deployers should switch to use Heat Engine instead of Direct Engine.
Developer impact
----------------
New features, which impacts infrastructure part of Sahara, should be supported
only in Heat Engine.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
vgridnev
Other contributors:
None
Work Items
----------
This change will require following changes:
* Add deprecation warnings at sahara startup.
* Mark Heat Engine as default in Sahara.
* Document deprecation of Direct Engine.
Dependencies
============
None
Testing
=======
This change require manual testing of deletion direct-engine-created cluster
after switch to heat engine.
Documentation Impact
====================
Need to document that Direct Engine became deprecated.
References
==========
None

119
specs/liberty/drop-hadoop-1-support.rst
View File

@ -1,119 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==============================================
Drop Hadoop v1 support in provisioning plugins
==============================================
https://blueprints.launchpad.net/sahara/+spec/drop-hadoop-1
This specification proposes to drop old Hadoop versions
Problem description
===================
Support for Hadoop 1 in provisioning plugins has become an unused feature. The
Hadoop development for v1 is almost frozen and Hadoop vendors are also dropping
its support.
As the sahara-plugins and sahara main repository split is going to happen it
will be a lot easier move less plugins to the new repo. Also the number of
unit and scenario tests will reduce.
The list of versions to be dropped is:
* Vanilla 1.2.1
* HDP 1.3.2
This spec does not suppose to drop the deprecated versions of plugins. That
should be a regular part of the release cycle.
Proposed change
===============
Drop the plugins code for Hadoop v1 along with:
* xml/json resources
* unit and scenario tests
* sample files
* image elements
Alternatives
------------
TBD
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
Elements for building Hadoop v1 images should be dropped as well
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
nkonovalov
Work Items
----------
* Disable Hadoop v1 tests in sahara-ci
* Drop related code and resource from sahara main repo
* Drop image elements
Dependencies
============
None
Testing
=======
Sahara-ci should not test Hadoop v1 anymore
Documentation Impact
====================
Add a note that Hadoop v1 is not supported in new releases.
References
==========
None

171
specs/liberty/edp-add-spark-shell-action.rst
View File

@ -1,171 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=====================================
[EDP] Add Spark Shell Action job type
=====================================
https://blueprints.launchpad.net/sahara/+spec/edp-add-spark-shell-action
The EDP Shell action job type allows users to run arbitrary
shell scripts on their cluster, providing a great deal of flexibility
to extend EDP functionality without engine changes or direct cluster
interface. This specification proposes the addition of this job type
for the Spark engine.
A fuller explication of the benefits of this feature can be found in
the edp-add-oozie-shell-action_ specification, which need not be
repeated here.
.. _edp-add-oozie-shell-action: http://specs.openstack.org/openstack/sahara-specs/specs/kilo/edp-add-oozie-shell-action.html
Problem description
===================
While the Oozie engine now supports Shell actions, Spark users do not
presently have access to this job type. Its addition would allow the
creation of cluster maintenance tools, pre- or post-processing jobs
which might be cumbersome to implement in Spark itself, the retrieval
of data from filesystems not supported as Sahara data sources, or any
other use case possible from a shell command.
Proposed change
===============
From an interface standpoint, the Spark Shell action implementation will
follow the Oozie Shell action implementation almost precisely:
* Shell jobs will require a single script binary in ``mains``, which will be
pushed to the cluster's master node and executed.
* Shell jobs will optionally permit any number of file binaries to be
passed as ``libs``, which will be placed in the script's working directory
and may be used by the script as it executes.
* ``configs`` will be permitted to allow Sahara EDP-internal features
(``substitute_data_source_for_uuid`` and ``subsitute_data_source_for_name``
will be implemented for this job type, as they are for Oozie Shell actions.)
* ``params`` key-value pairs will be passed to the script as environment
variables (whether these are passed into a remote ssh client or injected
into the script itself is left to the discretion of the implementer.)
* ``args`` will be passed to the script as positional command-line arguments.
* The Shell engine for Spark will store files as the main Spark engine does,
creating a directory under /tmp/spark-edp/``job_name``/``job_execution_id``,
which will contain all required files and the output of the execution.
* The Spark Shell engine will reuse the ``launch_command.py`` script (as used
by the main Spark engine at this time,) which will record childpid, stdout,
and stderr from the subprocess for record-keeping purposes.
Spark Shell actions will differ in implementation from Oozie Shell actions
in the following ways:
* As Spark jobs and Shell actions which happen to be running on a Spark
cluster differ quite entirely, the Spark plugin will be modified to contain
two separate engines (provided via an extensible strategy pattern based on
job type.) Sensible abstraction of these engines is left to the discretion
of the implementer.
* ``configs`` values which are not EDP-internal will not be passed to the
script by any means (as there is no intermediary engine to act on them.)
* Spark Shell actions will be run as the image's registered user, as Spark
jobs are themselves. As cluster and VM maintenance tasks are part of the
intended use case of this feature, allowing sudo access to the VMs is
desirable.
Alternatives
------------
Do nothing.
Data model impact
-----------------
None.
REST API impact
---------------
No additional changes after merge of the Oozie Shell action job type
implementation.
Other end user impact
---------------------
None.
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
No additional changes after merge of the Shell action job type UI
implementation.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
sgotliv
Other contributors:
egafford
Work Items
----------
* Add a Shell engine to the Spark plugin, and refactor this plugin to provide
an appropriate engine branching on job type.
* Add an integration test for Spark Shell jobs (as per previous plugin-
specific Shell job tests).
* Update the EDP documentation to specify that the Spark plugin supports the
Shell job type.
* Verify that the UI changes made for Oozie Shell jobs are sufficient to
support the Shell job type in the Spark case (as is anticipated).
Dependencies
============
This change builds on the change `[EDP] Add Oozie Shell Job Type`_.
.. _[EDP] Add Oozie Shell Job Type: https://review.openstack.org/#/c/159920/
Testing
=======
* Unit tests to cover the Spark Shell engine and appropriate engine selection
within the plugin.
* One integration test to cover running of a simple shell job through the
Spark plugin.
Documentation Impact
====================
The EDP sections of the documentation need to be updated.
References
==========
None.

143
specs/liberty/edp-datasource-placeholders.rst
View File

@ -1,143 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=====================================
Allow placeholders in datasource URLs
=====================================
https://blueprints.launchpad.net/sahara/+spec/edp-datasource-placeholders
This spec is to allow using placeholders in EDP data source URL.
Problem description
===================
Common use case: user wants to run EDP job two times. Now the only way to do
that with the same data sources is to erase result of the first run before
running job the second time. Allowing to have random part in URL will allow
to use output with random suffix.
Proposed change
===============
Introduce special strings that could be used in EDP data source URL and will
be replaced with appropriate value.
The proposed syntax for placeholder is %FUNC(ARGS)%.
As a first step I suggest to implement two functions only:
* %RANDSTR(len)% - will be replaced with random string of lowercase letters of
length ``len``.
* %JOB_EXEC_ID% - will be replaced with the job execution ID.
Placeholders will not be allowed in protocol prefix. So, there will be no
validation impact.
List of functions could be extended later (e.g. to have %JOB_ID%, etc.).
URLs after placeholders replacing will be stored in ``job_execution.info``
field during job_execution creation. This will allow to use them later to find
objects created by a particular job run.
Example of create request for data source with placeholder:
.. sourcecode:: json
{
"name": "demo-pig-output",
"description": "A data source for Pig output, stored in Swift",
"type": "swift",
"url": "swift://edp-examples.sahara/pig-job/data/output.%JOB_EXEC_ID%",
"credentials": {
"user": "demo",
"password": "password"
}
}
Alternatives
------------
Do not allow placeholders.
Data model impact
-----------------
``job_execution.info`` field (json dict) will also store constructed URLs.
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
Horizon need to be updated to display actual URLs for job execution. Input
Data Source and Output Data Source sections of job execution details page will
be extended to include information about URLs used.
REST will not be changed since new information is stored in the existing
'info' field.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
alazarev (Andrew Lazarev)
Other contributors:
None
Work Items
----------
* Implement feature
* Document feature
Dependencies
============
None.
Testing
=======
Manually.
Documentation Impact
====================
Need to be documented.
References
==========
None

197
specs/liberty/edp-edit-data-sources.rst
View File

@ -1,197 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
======================================
[EDP] Allow editing datasource objects
======================================
https://blueprints.launchpad.net/sahara/+spec/edp-edit-data-sources
Currently there is no way to edit a datasource object. If a path needs
to be changed, for example, the datasource must be deleted and a new
one created. The most common use case is a situation where a user
creates a datasource, runs a job, and receives an error from the job
because the path does not exist. Although it is not strictly necessary,
editable datasource objects would be a convenience when a user needs to
correct a path or credentials.
Problem description
===================
There are no API methods for updating a datasource object in the
REST API or at the conductor level.
The only way to correct a datasource is to delete an existing one and
create a new one with corrected information. Although it is possible to
use the same name, the object id will be different.
If editing is allowed, a user only needs to do a single operation to
make corrections. Additionally, the id is preserved so that objects which
reference it will reference the corrected path.
In the general case, editing a datasource should not be a problem for
a job execution which references it. Once a job execution enters the "RUNNING"
state, any information in datasource objects it references has been extracted
and passed to the process running the job. Consequently, editing a datasource
referenced by running or completed jobs will cause no errors. On relaunch,
a job execution will extract the current information from the datasource.
There is only a small window where perhaps editing should not be allowed.
This is when a datasource object is referenced by a job execution in the
"PENDING" state. At this point, information has not yet been extracted
from the datasource object, and a change during this window would
cause the job to run with paths other than the ones that existed at submission
time.
Proposed change
===============
Add an update operation to the REST API for datasource objects. Do not
allow updates for datasource objects that are referenced by job executions
in the "PENDING" state (this can be checked during validation).
Datasource objects referenced by job executions that are not in the PENDING
state may be changed. In an existing blueprint and related CR (listed in the
reference section) the URLs used by a job execution will be recorded in
the job execution when the job enters the RUNNING state. This means that for
any running or completed job execution, the list of exact datasource URLs
used in the execution will be available from the job execution itself even
if the referenced datasource has been edited.
Allow any fields in a datasource object to be updated except for id.
The object id should be preserved.
Add the corresponding update operation to the python-saharaclient.
Alternatives
------------
Do nothing
Data model impact
-----------------
None
REST API impact
---------------
Backward compatiblity will be maintained since this is a new endpoint.
**PUT /v1.1/{tenant_id}/data-sources/{data_source_id}**
Normal Response Code: 202 (ACCEPTED)
Errors: 400 (BAD REQUEST), 404 (NOT FOUND)
Update the indicated datasource object
**Example**
**request**
.. sourcecode:: text
PUT http://sahara/v1.1/{tenant_id}/data-sources/{data_source_id}
.. sourcecode:: json
{
"description": "some description",
"name": "my_input",
"url": "swift://container/correct_path"
}
**response**
.. sourcecode:: http
HTTP/1.1 202 ACCEPTED
Content-Type: application/json
.. sourcecode:: json
{
"created_at": "2015-04-08 20:27:13",
"description": "some_description",
"id": "7b25fc64-5913-4bc3-aaf4-f82ad03ea2bc",
"name": "my_input",
"tenant_id": "33724d3bf3114ae9b8ab1c170e22926f",
"type": "swift",
"updated_at": "2015-04-09 10:27:13",
"url": "swift://container_correct_path"
}
Other end user impact
---------------------
This operation should be added to the python-saharaclient API as well
$ sahara data-source-update [--name NAME] [--id ID] [--json]
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
To take advantage of this from the Horizon UI, we would need a selectable
"Edit" action for each datasource on the datasources page
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Trevor McKay
Other contributors:
Chad Roberts
Work Items
----------
Add REST and support methods to Sahara
Add operation to python-saharaclient
Add operation to datasource screens in Horizon
Add to WADL in api-ref
Dependencies
============
None
Testing
=======
Unit tests in Sahara and python-saharaclient
Documentation Impact
====================
Potentially any user documentation that talks about relaunch, or
editing of other objects like templates
References
==========
https://blueprints.launchpad.net/sahara/+spec/edp-datasource-placeholders
https://review.openstack.org/#/c/158909/

201
specs/liberty/edp-edit-job-binaries.rst
View File

@ -1,201 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
================================
[EDP] Allow editing job binaries
================================
https://blueprints.launchpad.net/sahara/+spec/edp-edit-job-binaries
Currently there is no way to edit a job binary. If a path needs
to be changed, for example, the job binary must be deleted and a new
one created. The most common use case is a situation where a user
creates a job binary, runs a job, and receives an error from the job
because the path does not exist. Although it is not strictly necessary,
editable job binary objects would be a convenience when a user needs to
correct a path or credentials.
Problem description
===================
There are no API methods for updating a job binary object in the
REST API or at the conductor level.
The only way to correct a job binary is to delete an existing one and
create a new one with corrected information. Although it is possible to
use the same name, the object id will be different.
If editing is allowed, a user only needs to do a single operation to
make corrections. Additionally, the id is preserved so that objects which
reference it will reference the corrected path.
In the general case, editing a job binary should not be a problem for
a job object that references it. Once a job execution enters the "RUNNING"
state, any job binary objects it references indirectly through the job object
have been uploaded to the cluster for execution. Consequently, editing a
job binary object will cause no errors.
There is only a small window where editing should not be allowed.
This is when a job binary object is referenced by a job execution in the
"PENDING" state. At this point, binaries have not yet been uploaded to the
cluster and a change during this window would cause the job to run with
paths other than the ones that existed at submission time.
Note, the paths of binaries used by a job execution should be recorded in
the job execution. This will remove a restriction on editing of paths in
a job binary that is referenced by an existing job execution. This will be
done in a separate blueprint listed in the references section (similar
recording of data source paths used during an execution is supported in
another blueprint).
Proposed change
===============
Add an update operation to the REST API for job binary objects. Do not
allow updates for job binaries that are referenced by job executions
in the "PENDING" state (this can be checked during validation).
Allow the following fields in the job binary to be edited:
* name
* description
* url if the value is not an "internal-db://" path
For binaries stored in the Sahara database, the URL is generated by
Sahara and should not be editable.
Add the corresponding update operation to the python-saharaclient.
Alternatives
------------
Do nothing
Data model impact
-----------------
None
REST API impact
---------------
Backward compatiblity will be maintained since this is a new endpoint.
**PUT /v1.1/{tenant_id}/job-binaries/{job_binary_id}**
Normal Response Code: 202 (ACCEPTED)
Errors: 400 (BAD REQUEST), 404 (NOT FOUND)
Update the indicated job-binary object
**Example**
**request**
.. sourcecode:: text
PUT http://sahara/v1.1/{tenant_id}/job-binaries/{job_binary_id}
.. sourcecode:: json
{
"description": "some description",
"name": "my.jar",
"url": "swift://container/correct_path"
}
**response**
.. sourcecode:: http
HTTP/1.1 202 ACCEPTED
Content-Type: application/json
.. sourcecode:: json
{
"created_at": "2015-04-08 20:48:18",
"description": "",
"id": "640ca841-d4d9-48a1-a838-6aa86b12520f",
"name": "my.jar",
"tenant_id": "33724d3bf3114ae9b8ab1c170e22926f",
"updated_at": "2015-04-09 10:48:18",
"url": "swift://container/correct_path"
}
Other end user impact
---------------------
This operation should be added to the python-saharaclient API as well
$ sahara job-binary-update [--name NAME] [--id ID] [--json]
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
To take advantage of this from the Horizon UI, we would need a selectable
"Edit" action for each job binary on the job binaries page
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Trevor McKay
Other contributors:
Chad Roberts
Work Items
----------
Add REST and support methods to Sahara
Add operation to python-saharaclient
Add operation to job binary screens in Horizon
Add to WADL in api-ref
Dependencies
============
This is a blueprint to store the job binary paths in the job execution object.
Implementing this first will allow editing of job binaries as long as they are
not in the PENDING state. Otherwise, editing will have to be disallowed for job
binaries referenced by an existing job execution.
https://blueprints.launchpad.net/sahara/+spec/edp-store-binary-paths-in-job-executions
Testing
=======
Unit tests in Sahara and python-saharaclient
Documentation Impact
====================
Potentially any user documentation that talks about
editing of other objects like templates
References
==========
None

140
specs/liberty/enable-cdh-hdfs-ha.rst
View File

@ -1,140 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===================
CDH HDFS HA Support
===================
https://blueprints.launchpad.net/sahara/+spec/cdh-ha-support
This blueprint aims to implement HDFS High-Availability (HA) for Cloudra
plugin.
Problem description
===================
Currently Cloudera plugin does not support HA for services. We plan to
implement HDFS HA as the first step. HA for Yarn and other services will be
the later steps.
Proposed change
===============
The implementation of the HDFS HA will be done via CM API enable_nn_ha(). This
API will help us enable HDFS HA by giving several info augments.
CDH 5 only supports Quorum-based Storage as the only HA implementation, so we
will only implement this. To achieve this, we need to add a Standby NameNode,
and several JournalNodes. The JournalNode number should be odd and at least 3.
When HDFS HA is enabled, SecondaryNameNode will not be used. So we can reuse
the node for SecondaryNameNode for StandbyNameNode.
HDFS HA has several hardware constraints (see the reference link). However,
for all resources are virtual in Openstack, we will only require NameNode and
StandbyNameNode are on different physical hosts.
Overall, we will implement HDFS as below:
* Add a role JournalNode.
* If JournalNode was selected by user (cluster admin), then HA will be enabled.
* If HA is enabled, we will validate whether JournalNode number meet
requirements.
* JournalNode roles will not be really created during cluster creation. In fact
they will be used as parameters of CM API enable_nn_ha.
* If HA is enabled, we will use SecondaryNameNode as the StandbyNameNode.
* If HA is enabled, we will set Anti-affinity to make sure NameNode and
SecondaryNameNode will not be on the same physical host.
* If HA is enabled, Zookeeper service is required in the cluster.
* After the cluster was started, we will call enable_nn_ha to enable HDFS HA.
* If HA is enabled, in Oozie workflow xml file, we will give nameservice name
instead of the NameNode name in method get_name_node_uri. So that the cluster
can determine by itself which NameNode is active.
Alternatives
------------
None.
Data model impact
-----------------
None.
REST API impact
---------------
None.
Other end user impact
---------------------
None.
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Ken Chen
Work Items
----------
Changes will be only in sahara/plugins/cdh directory. We will only do this
based on CDH 5.4.0 at this stage. CDH 5.0.0 and CDH 5.3.0 plugins will not be
supported. Changes were described in the Proposed change section.
Dependencies
============
None.
Testing
=======
We will only do primitive checks: create a Cloudera cluster with HDFS HA, and
see whether it is active.
Documentation Impact
====================
The documentation needs to be updated with information about enabling CDH HDFS
HA.
References
==========
* `NameNode HA with QJM <http://www.edureka.co/blog/namenode-high-availability-with-quorum-journal-manager-qjm/>`
* `Introduction to HDFS HA <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_hag_hdfs_ha_intro.html>`
* `Enable HDFS HA Using Cloudera Manager <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_hag_hdfs_ha_enabling.html#cmug_topic_5_12_unique_1>`
* `Configuring Hardware for HDFS HA <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_hag_hdfs_ha_hardware_config.html>`

143
specs/liberty/enable-cdh-rm-ha.rst
View File

@ -1,143 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===================================
CDH YARN ResourceManager HA Support
===================================
https://blueprints.launchpad.net/sahara/+spec/cdh-ha-support
This blueprint aims to implement YARN ResourceManager (RM) High-Availability
(HA) for Cloudra plugin.
Problem description
===================
Currently Cloudera plugin does not support HA for YARN ResourceManager.
Therefore we plan to implement RM HA.
Proposed change
===============
The implementation of the RM HA will be done via CM API enable_rm_ha(). This
API will help us enable ResourceManager HA by giving several info augments.
To achieve RM HA, we need to add one Standby NodeManager in the node
of the cluster templates (Cloudera Plugin only support one Standby NodeManager,
while the other implementations might allow >1 Standby NodeManager).
When RM HA is enabled, we will start both the Primary and Standby NodeManagers,
let the Primary NodeManager be active, and leave the Standby NodManager
standby.
CM API enable_rm_ha accepts a new ResourceManager Host new_rm_host_id as
parameter. A Standby ResourceManager will be started on new_rm_host_id node.
ResourceManager HA requires Primary ResourceManager and Standby ResourceManager
roles deployed on different physical hosts. Zookeeper service is required too.
When ResourceManager HA is enabled, Oozie or applications depended on RM should
be able to check all available RMs and get the active RM by themselves. There
is no way to automatically switch between RMs smoothly.
Overall, we will implement RM HA as below:
* Add a role YARN_STANDBYRM.
* If YARN_STANDBYRM was selected by user (cluster admin), then YARN RM HA will
be enabled.
* If RM HA is enabled, we will check Anti-affinity to make sure YARN_STANDBYRM
and YARN_RESOURCEMANAGER will not be on the same physical host.
* If RM HA is enabled, Zookeeper service is required in the cluster.
* If RM HA is enabled, we will create cluster with ResourceManager on node
where YARN_RESOURCEMANAGER role is assigned.
* If RM HA is enabled, after the cluster is started, we will call enable_rm_ha
to enabled RM HA by using the YARN_STANDBYRM node as parameter.
It should be noted that, if HA is enabled, in Oozie workflow xml file, we need
to detect the active ResourceManager in method get_resource_manager_uri and
pass it to Oozie each time when we use it. I plan to include this part of codes
in later patches.
Alternatives
------------
None.
Data model impact
-----------------
None.
REST API impact
---------------
None.
Other end user impact
---------------------
None.
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Ken Chen
Work Items
----------
Changes will be only in sahara/plugins/cdh directory. We will only do this
based on CDH 5.4.0 at this stage. CDH 5.0.0 and CDH 5.3.0 plugins will not be
supported. Changes were described in the Proposed change section.
Dependencies
============
None.
Testing
=======
We will only do primitive checks: create a Cloudera cluster with RM HA, and
see whether it is active.
Documentation Impact
====================
The documentation needs to be updated with information about enabling CDH YARN
ResourceManager HA.
References
==========
* `Configuring High Availability for ResourceManager (MRv2/YARN) <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_hag_rm_ha_config.html/>`

215
specs/liberty/hdp-22-support.rst
View File

@ -1,215 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================
Add support HDP 2.2 plugin
==========================
https://blueprints.launchpad.net/sahara/+spec/hdp-22-support
This specification proposes to add new HDP plugin based on Ambari
Blueprints [1] with Ambari Management Console.
Problem description
===================
Currently we support old HDP plugin which contains old HDP distribution.
Also old HDP plugin looks like unsupported by HortonWorks team every year [2].
Many customers want new version of HDP. New HDP plugin will be based on Ambari
Blueprints. Ambari Blueprints are a declarative definition of a cluster.
With a Blueprint, you specify a Stack, the Component layout and
the Configurations to materialize a Hadoop cluster instance via REST API.
Proposed change
===============
New HDP plugin will support provisioning HDP stack via Ambari Blueprints.
Plugin will support key Sahara features:
* Cinder integration
* Swift integration
* EDP
* Scaling
* Event logs
New HDP plugin will support the following OS: Ubuntu 12.04 and CentOS 6. Aslo
new plugin will support mirrors with HDP packages.
New HDP plugin will support all services which supports Ambari. Also new plugin
will support HA for NameNode and ResourceManager. Client will be installed on
all nodes if selected our process. For example if selected Oozie then will be
installed Oozie client on all nodes.
Plugin wil be support the following services:
+-------------------+---------------------------+
| Service | Process |
+===================+===========================+
| Ambari | Ambari |
+-------------------+---------------------------+
| Falcon | Falcon Server |
+-------------------+---------------------------+
| Flume | Flume |
+-------------------+---------------------------+
| HBase | HBase Master |
| +---------------------------+
| | HBase RegionServer |
+-------------------+---------------------------+
| HDFS | NameNode |
| +---------------------------+
| | DataNode |
| +---------------------------+
| | SecondaryNameNode |
| +---------------------------+
| | JournalNode |
+-------------------+---------------------------+
| Hive | Hive Metastore |
| +---------------------------+
| | HiveServer |
+-------------------+---------------------------+
| Kafka | Kafka Broker |
+-------------------+---------------------------+
| Knox | Knox Gateway |
+-------------------+---------------------------+
| Oozie | Oozie |
+-------------------+---------------------------+
| Ranger | Ranger Admin |
| +---------------------------+
| | Ranger Usersync |
+-------------------+---------------------------+
| Slider | Slider |
+-------------------+---------------------------+
| Spark | Spark History Server |
+-------------------+---------------------------+
| Sqoop | Sqoop |
+-------------------+---------------------------+
| Storm | DRPC Server |
| +---------------------------+
| | Nimbus |
| +---------------------------+
| | Storm UI Server |
| +---------------------------+
| | Supervisor |
+-------------------+---------------------------+
| YARN | YARN Timeline Server |
| +---------------------------+
| | MapReduce History Server |
| +---------------------------+
| | NodeManager |
| +---------------------------+
| | ResourceManager |
+-------------------+---------------------------+
| ZooKeeper | ZooKeeper |
+-------------------+---------------------------+
Alternatives
------------
Add support of HDP 2.2 in old plugin, but it is very difficult to do without
Ambari Blueprints.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
Need to add elements for building images with pre-installed Ambari packages.
For installing HDP Stack plugin should use mirror with HDP packages. Also
should add elements for building local HDP mirror.
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
sreshetniak
Other contributors:
nkonovalov
Work Items
----------
* Add base implementation of plugin [3] [4]
* Add elements for building image with Ambari [5]
* Add EDP support [6]
* Add additional services support [7]
* Add scaling support [8]
* Add HA support [9]
* Add elements for building HDP mirror [10]
Dependencies
============
None
Testing
=======
* Add unit tests for plugin
* Add scenario tests and job on sahara-ci
Documentation Impact
====================
New plugin documentation should be added to Sahara docs.
References
==========
[1] https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
[2] http://stackalytics.com/?module=sahara-group&release=all&company=hortonworks&metric=commits
[3] https://review.openstack.org/#/c/184292/
[4] https://review.openstack.org/#/c/185100/
[5] https://review.openstack.org/#/c/181732/
[6] https://review.openstack.org/#/c/194580/
[7] https://review.openstack.org/#/c/195726/
[8] https://review.openstack.org/#/c/193081/
[9] https://review.openstack.org/#/c/197551/
[10] https://review.openstack.org/#/c/200570/

127
specs/liberty/heat-hot.rst
View File

@ -1,127 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
============================
Migrate to HEAT HOT language
============================
https://blueprints.launchpad.net/sahara/+spec/heat-hot
This blueprint suggests to rewrite cluster template for Heat from JSON to HOT
language.
Problem description
===================
Heat supports two different template languages: YAML-based
`HOT <http://docs.openstack.org/developer/heat/template_guide/hot_guide.html>`_ templates
and JSON-based CFN templates (no documentation about it, only a number of
examples).
HOT is the de-facto main markup language for Heat. `Template Guide
<http://docs.openstack.org/developer/heat/template_guide/index.html>`_
recommends to use HOT and contains examples for HOT only.
CFN templates are supported mostly for compatibility with AWS CloudFormation.
Sahara historically uses CFN templates. Given that Sahara is an integrated
OpenStack project it would be nice to switch to HOT.
Proposed change
===============
There is no urgent need in switching to HOT. But it would be nice to be
compliant with current tendencies in community.
This spec suggests to use HOT template language for sahara heat templates.
This will require changes mostly in .heat resources. Code that generate
template parts on the fly should be changed too.
Having templates written on HOT will simplify implementation of new
heat-related features like `template decomposition
<https://blueprints.launchpad.net/sahara/+spec/heat-template-decomposition>`_.
Alternatives
------------
Do not change anything.
Data model impact
-----------------
None.
REST API impact
---------------
None.
Other end user impact
---------------------
None.
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
alazarev (Andrew Lazarev)
Other contributors:
None
Work Items
----------
* Change all .heat files used by Sahara
* Update code that generates parts of template
* Update unit tests
* Make sure that sahara with heat engine still works in all supported
configurations
Dependencies
============
None
Testing
=======
Mostly manually. CI should also cover heat changes.
Documentation Impact
====================
None.
References
==========
* http://docs.openstack.org/developer/heat/template_guide/hot_guide.html

127
specs/liberty/heat-template-decomposition.rst
View File

@ -1,127 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===================================
Decompose cluster template for Heat
===================================
https://blueprints.launchpad.net/sahara/+spec/heat-template-decomposition
Currently Sahara creates one large template with a lot of copy-paste
resources. Heat features like `template composition
<http://docs.openstack.org/developer/heat/template_guide/composition.html>`_
could help to move all composition work from Sahara to Heat. This will also
allow sahara to have individual cluster parts as separate templates and insert
them as resources (in comparison to current text manipulations).
Problem description
===================
Currently Sahara serializes cluster resources as text to heat template. There
are several issues with this approach:
1. Code duplication. If a node group contains 10 instances the template will
contain all instance-dependent resources 10 times.
2. No code validation. There is no guarantee that resulting template will be
syntactically correct (Sahara treats it as text). Missing comma in one
resource could influence the other resource.
3. Not Sahara's work. Sahara micro-manages the process of infrastructure
creation. This is Heat's job.
Proposed change
===============
Use `OS::Heat::ResourceGroup <http://docs.openstack.org/hot-reference/content/OS__Heat__ResourceGroup.html>`_
for resources inside node group. Each node group will contain only one
resource group and specify number of instances needed. Each individual
instance of resource group will contain all resources needed for a
corresponding sahara instance (nova server, security group, volume,
floating ip, etc.).
This change will also prepare ground for node group auto-scaling feature.
Alternatives
------------
None.
Data model impact
-----------------
None.
REST API impact
---------------
None.
Other end user impact
---------------------
Resulting Sahara stack in Heat will contain nested stacks.
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
alazarev (Andrew Lazarev)
Other contributors:
None
Work Items
----------
* Add ability to generate separate ResourceGroup for a single instance of node
group
* Switch template for node groups to ResourceGroup with specified count of
instances
* Update unit tests
* Make sure that Sahara with heat engine still works for all supported
configurations
Dependencies
============
None.
Testing
=======
Manually. CI will also cover changes in heat.
Documentation Impact
====================
None.
References
==========
None.

208
specs/liberty/keystone-sessions.rst
View File

@ -1,208 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
================================================
Updating authentication to use keystone sessions
================================================
https://blueprints.launchpad.net/sahara/+spec/keystone-sessions
Sahara currently uses per access authentication when creating OpenStack
clients. This style of authentication requires keystone connections on every
client creation. The keystone project has created a mechanism to streamline
and improve this process in the form of Session objects. These objects
encapsulate mechanisms for updating authentication tokens, caching of
connections, and a single point for security improvements. Sahara should
migrate its OpenStack client objects to use session objects for all clients.
Problem description
===================
For all OpenStack client instances, sahara uses authentication on a
per-client creation basis. For each client object that is requested, a set of
credentials are acquired from the context, or the configuration file in the
case of admin accounts, which are used to initialize the client object.
During this initialization a request is made to the Identity service to
determine the user's privileges with respect to the new client.
Sahara must be aware of any changes to the authentication methods for each
client as well as any potential security vulnerabilities resulting from the
usage of those methods.
This method of authentication does not allow sahara to share any information
between clients, aside from the raw credentials. In turn, this introduces
brittleness to the sahara/client interface as each authentication
relationship must be maintained separately or, worse yet, with a partial
shared model.
Having separate interfaces for each client also makes applying security
updates more difficult as each client instance must be visited, researched,
and ultimately fixed according the specific details for that client.
Although this methodology has served sahara well thus far, the keystone
project has introduced new layers of abstraction to aid in sharing common
authentication between clients. This shared methodology, the keystoneclient
Session object, provides a unified point of authentication for all clients.
It serves as a single point to contain security updates, on-demand
authentication token updating, common authentication methods, and
standardized service discovery.
Proposed change
===============
Sahara should standardize its client authentication code by utilizing
keystoneclient Session objects. This change will entail creating a new
module, modifying the OpenStack client utility functions, and adding
an authentication plugin object to the context.
A new module, ``sahara.service.sessions``, will be created to contain utility
functions and classes to aid in the creation and storage of session objects.
This module will also contain a global singleton for the sessions cache.
``sahara.service.sesssions`` will provide a class named ``SessionCache`` as
well as a function to gain the global singleton instance of that class. The
``SessionCache`` will contain cached session objects that can be reused in
the creation of individual OpenStack clients. It will also contain functions
for generating the session objects required by specific clients. Some clients
may require unique versions to be cached, for example if a client requires a
specific certificate file then it may have a unique session. For all other
clients that do not require a unique session, a common session will be used.
Authentication for session objects will be provided by one of a few methods
depending on the type of session needed. For user based sessions,
authentication will be obtained from the keystonemiddleware authentication
plugin that is generated with each request. For admin based sessions, the
credentials found in the sahara configuration file will be used to generate
the authentication plugin. Trust based authentication will be handled by
generating an authentication plugin based on the information available in
each case, either a token for a user or a password for an admin or proxy
user.
The ``sahara.context.Context`` object will be changed to incorporate an
authentication plugin object. When created through REST calls the
authentication plugin will be obtained from the keystonemiddleware. When
copying a context, the authentication plugin will be copied as well. For
other cases the authentication plugin may be set programmatically, for
example if an admin authentication plugin is required it can be generated
from values in the configuration file, or if a trust based authentication
is required it can be generated.
The individual OpenStack client utility modules will be changed to use
session and authentication plugin objects for their creation. The
sessions will be obtained from the global singleton and the authentication
plugin objects can be obtained from the context, or created in circumstances
that require more specific authentication, for example when using the
admin user or trust based authentication.
The clients for heat and swift do not yet enable session based
authentication. These clients should be monitored for addition of this
feature and migrated when available.
Alternatives
------------
An alternative to this approach would be to create our own methodology for
storing common authentication credentials, but this would be an exercise in
futility as we would merely be replicating the work of keystoneclient.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Michael McCune (elmiko)
Other contributors:
Andrey Pavlov (apavlov)
Work Items
----------
* create sahara.service.sessions
* modify context to accept authentication plugin
* modify sahara.utils.openstack clients to use sessions
* cinder
* keystone
* neutron
* nova
* modify admin authentications to use plugin objects
* modify trust authentications to use plugin objects
* create tests for session cache
* create developer documentation for client usage
Dependencies
============
None
Testing
=======
The tests created for this feature will be unit based, to exercise the code
paths and logic points. Functional testing should not be necessary as these
authentication methods will be exercised in the course of the standard
functional testing.
Documentation Impact
====================
This change will only create documentation within the sahara project.
Currently there exists no documentation about client usage within the sahara
codebase. This change will add a small section describing how to instantiate
clients using the ``sahara.utils.openstack`` package, with a note about
common session authentication.
References
==========
`Keystoneclient documentation about using Sessions <http://docs.openstack.org/developer/python-keystoneclient/using-sessions.html>`_
`How to Use Keystoneclient Sessions (article by Jamie Lennox) <http://www.jamielennox.net/blog/2014/09/15/how-to-use-keystoneclient-sessions/>`_

189
specs/liberty/manila-as-a-data-source.rst
View File

@ -1,189 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===============================
Manila as a runtime data source
===============================
https://blueprints.launchpad.net/sahara/+spec/manila-as-a-data-source
Using manila nfs shares and the mounting features developed for use with
job binaries, it should be feasible to use nfs shares to host input
and output data. This will allow data to be referenced through local
filesystem paths as a another simple alternative to hdfs or swift storage.
Problem description
===================
The work has already been done to support mounting of manila nfs shares at
cluster provisioning time or automounting shares for EDP job binaries with a
url of the form *manila://<share-id>/path*. Additionally, the Hadoop filesystem
APIs already support *file:///path* urls for referencing the local filesystem.
Sahara can build on these existing features by allowing
*manila://<share-id>/path* urls for data sources, automounting shares
referenced by data sources when necessary, and generating the correct
local filesystem urls for EDP jobs at runtime.
Some of the benefits of this approach are:
* parity for job binaries and data sources in regards to manila shares
* the ability to store job binaries and data sources in the same share
* the flexibility to add a new data share to a cluster at any time
* the ability to operate on data from a cluster node using native OS tools
* works on any node where *mount -t nfs ...* is supported
* lays the groundwork for other manila share types in the future
The problem can be divided into three high-level items:
* Add a *manila* data source type with validation of *manila://* urls
* Translate *manila://* urls to *file://* urls for use at job runtime
* Call the existing automounting methods when a data source references
a new share
Note, automounting and url translation will only work for manila shares
referenced by data source objects. A *manila://* url embedded as a literal
in a job config, param, or arg will be ignored. It will not be translated
to a *file://* url by Sahara and it will not cause automounting. However,
there is a precedent for this -- Sahara currently has other features that
are only supported on data source objects, not on literal urls. (It may
be possible to remove these limitations in the future through greater
use of the unified job mapping interface recently introduced).
Proposed change
===============
A *manila* data source type will be added to the JSON schema for data sources,
with appropriate validation of *manilla://* urls.
The existing code in *sahara/service/edp/binary_retrievers/manila_share.py*
that supports path name generation and automounting of manila nfs shares for
job binaries will be refactored and broken up between
*sahara/service/edp/job_utils.py* and *sahara/service/shares.py*. The essential
implementation is complete, but this logic needs to be callable from multiple
places and in different combinations to support data sources.
Currently, all data source urls are returned to the EDP engines from
*get_data_sources()* and *resolve_data_source_references()* in *job_utils.py*.
The returned urls are recorded in the job_execution object and used by the
EDP engine to generate the job on the cluster. These two routines will be
extended to handle manila data sources in the following ways:
* Mount a referenced nfs share on the cluster when necessary. Since an
EDP job runs on multiple nodes, the share must be mounted to the whole
cluster instead of to an individual instance
* Translate the *manila://* url to a *file://* url and return both urls
Since the submission time url and the runtime url for these data
sources will be different, both must be returned. Sahara will record
the submission time url in the job_execution but use the runtime url
for job generation
Alternatives
------------
Do not support *manila://* urls for data sources but support data hosted on nfs
as described in
https://review.openstack.org/#/c/210839/
However, these features are complementary, not mutually exclusive, and most of
the appartus necessary to make this proposal work already exists.
Data model impact
-----------------
None
REST API impact
---------------
None (only "manila" as a valid data source type in the JSON schema)
Other end user impact
---------------------
None
Deployer impact
---------------
Obviously, if this feature is desired then the manila service should be running
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None (nfs-utils element is already underway)
Sahara-dashboard / Horizon impact
---------------------------------
Sahara needs a manila data source type on the data source creation form
Implementation
==============
Assignee(s)
-----------
Primary assignee:
tmckay
Other contributors:
croberts, egafford
Work Items
----------
* Add manila data source type to the JSON schema
* Allow submission time and runtime urls to differ
(https://review.openstack.org/#/c/209634/3)
* Refactor binary_retrievers/manila_share.py
* Extend job_utils get_data_sources() and resolve_data_source_references()
to handle manila:// urls
* Add manila data source creation to Horizon
* Modify/extend unit tests
* Documentation
Dependencies
============
https://blueprints.launchpad.net/sahara/+spec/manila-as-binary-store
Testing
=======
Unit tests.
Eventually, as with job binaries, this can be tested with integration
tests if/when we have manila support in the gate
Documentation Impact
====================
Discussion of the manila data source type should be added to any
sections we currently have that talk about data being hosted in swift of hdfs.
Additionally, we should consider adding information to the Sahara section
of the security guide on the implications of using manila data shares.
If the security guide or the manila documentation contains a section on
security, this probably can be a short discussion from a Sahara perspective
with a link to the security info. If there isn't such a section currently, then
probably there should be a separate CR against the security guide to create a
section for Manila.
References
==========

165
specs/liberty/manila-as-binary-store.rst
View File

@ -1,165 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
====================================
Addition of Manila as a Binary Store
====================================
https://blueprints.launchpad.net/sahara/+spec/manila-as-binary-store
Network and distributed filesystems are a useful means of sharing files among
a distributed architecture. This core use case makes them an excellent
candidate for storage of job binaries.
Problem description
===================
While internal database storage and Swift storage are sensible options for
binary storage, the addition of Manila integration allows it as a third option
for job binary storage and retrieval. This specification details how this
will be implemented.
Proposed change
===============
The scheme ``manila`` will be added as an option in creation of job binaries.
URLs will take the form:
``manila://{share_id}/absolute_path_to_file``
URLs stored as binary locations can be passed through the following
validations:
1) The manila share id exists.
2) The share is of a type Sahara recognizes as a valid binary store.
Because manila shares are not mounted to the control plane and should not be,
we will not be able to assess the existence or nonexistence of files intended
to be job binaries.
We can, however, assess the share type through Manila. For the initial
reference implementation, only NFS shares will be permitted for this
use; other share types may be verified and added in later changes.
For this binary type, Sahara will not retrieve binary files and copy them into
the relevant nodes; they are expected to be reachable through the nodes' own
filesystems. Instead, Sahara will:
1) Ensure that the share is mounted to the appropriate cluster nodes (in the
Oozie case, the node group containing Oozie server; in the Spark case, the
node group containing Spark Master, etc.) If the share is not already
mounted, Sahara will mount the share to the appropriate node groups using
the mechanism described by blueprint mount-share-api (at the default path)
and update the cluster's DB definition to note the filesystem mount.
2) Replace ``manila://{share_id}`` with the local filesystem mount point, and
use these local filesystem paths to build the workflow document or job
execution command, as indicated by engine. Job execution can then take
place normally.
It is notable that this specification does not cover support of Manila
security providers. Such support can be added in future changes, and should
not affect this mechanism.
Alternatives
------------
While verification of paths on binary creation would be ideal, mounting tenant
filesystems (in the abstract, without a cluster necessarily available) is a
prohibitive security concern that outweighs this convenience feature (even
if we assume that networking is not an issue.)
We could also create a more general ``file://`` job binary scheme, either in
addition to ``manila://`` or as a replacement for it. However, this would not
particularly facilitate reuse among clusters (without a number of manual steps
on the user's part) or allow auto-mounting when necessary.
We could also opt to simply raise an exception if the share has not already
been mounted to the cluster by the user. However, as the path to automatic
mounting is clear and will be reasonably simple once the mount-share-api
feature is complete, automatic mounting seems sensible for the initial
implementation.
Data model impact
-----------------
None.
REST API impact
---------------
None.
Other end user impact
---------------------
None.
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None.
Sahara-dashboard / Horizon impact
---------------------------------
Manila will be added as a visible job binary storage option; no other changes.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
egafford
Secondary assignee/reviewer:
croberts
Work Items
----------
* Validation changes (URL scheme).
* Integration with mounting feature.
* Creation of "job binary retriever" strategy for Manila (which will mostly
no-op, given the strategy above).
* Modification of workflow and execution command code to facilitate this flow.
* Horizon changes (in separate spec).
* Documentation.
Dependencies
============
None.
Testing
=======
Unit testing is assumed; beyond this, full integration testing will depend on
the feasibility of adding a manila endpoint to our CI environment. If this is
feasible, then our testing path becomes clear; if it is not, then gated
integration testing will not be possible.
Documentation Impact
====================
This feature will require documentation in edp.rst.
References
==========
See https://wiki.openstack.org/wiki/Manila/API if unfamiliar with manila
operations.

254
specs/liberty/mount-share-api.rst
View File

@ -1,254 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=========================================================
API to Mount and Unmount Manila Shares to Sahara Clusters
=========================================================
https://blueprints.launchpad.net/sahara/+spec/mount-share-api
As OpenStack's shared file provisioning service, manila offers great
integration potential with sahara, both for shared binary storage and as a
data source. While it seems unnecessary to wrap manila's share provisioning
APIs in sahara, allowing users to easily mount shares to all nodes of a Sahara
cluster in a predictable way will be a critical convenience feature for this
integration.
Problem description
===================
Manually mounting shares to every node in a large cluster would be a tedious
and error-prone process. Auto-mounting shares that are requested for use in
either the data source or binary storage case might be feasible for some use
cases. However, outside of our (optional) EDP interface this functionality
would never be usable. As such, it is best to provide the user an API for
mounting of shares onto Sahara clusters.
Proposed change
===============
This change proposes to expand the node group template, node group, cluster
template, and cluster API resources and database objects to contain a "shares"
field. As per other node group fields, the cluster template and cluster APIs
will allow overwrite of this field (which is particularly critical for
composition, given that manila shares represent concrete data rather than
abstract resource pools.) At the resource level (for both resource types),
this field will be defined by the following jsonschema:
::
"shares": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {
"type": "string",
"format": "uuid"
},
"path": {
"type": ["string", "null"]
},
"access_level": {
"type": ["string", "null"],
"enum": ["rw", "ro"],
"default": "rw"
}
},
"additionalProperties": False,
"required": [
"id"
]
}
}
"id" above refers to the UUID of the manila share to be mounted. It is
required.
"path" refers to the local path on each cluster node on which to mount this
share, which should be universal across all nodes of the cluster for
simplicity. It will default to ``/mnt/{share_id}``.
"access_level" governs permissions set in manila for the cluster ips.
This defaults to 'rw'.
Because no part of this field requires indexing, it is proposed that the
above structure be directly serialized to the database as a TEXT field in
JSON format.
Any share specified at the node group level will be mounted to all instances
of that node group. Any share specified at the cluster level will be mounted
to all nodes of that cluster. At cluster creation, in the case that a specific
share id is specified at both the node group and cluster level, the cluster's
share configuration (path and access level) will entirely replace the node
group's configuration. Any merge of share configurations is seen as needlessly
complex and error-prone, and it is our longstanding pattern that cluster-level
configurations trump node group-level configurations.
Error cases in this API include:
1. The provided id is not a valid manila share id, as assessed via
manilaclient with the user's credentials.
2. The provided path is not a valid, absolute Linux path.
3. Path is not unique (within the set of shares specified for any one node
or the set of shares specified for any cluster.)
4. The provided id maps to a manila share type which sahara is not currently
equipped to mount.
5. No manila service endpoint exists within the user's service catalog.
On cluster creation (or update, if update becomes an available endpoint,)
just after the cluster becomes available and before delegating to the plugin
(such that any shares intended for HDFS integration will be in place for
the plugin configuration to act upon,) Sahara will execute a share mounting
step. For each share, Sahara will take the following steps:
1. Query manila for share information, including share type and defaults.
2. Query the cluster object to find internal ip addresses for all cluster
nodes of any node group for which the share should be mounted.
3. For each such node, call to manila to allow access for each ip according
to the permissions set on the share.
4. Make a remote call to each qualifying node and mount the share via its
mount address as returned from manila.
Steps 1-3 above will be handled via common code in an abstract ShareHandler
class. The last step will be delegated to a concrete instance of this class,
based on share type as reported by manila, which will execute appropriate
command-line operations over a remote socket to mount the share.
The reference and test implementation for the first revision of this feature
will only provide an NFS mounter. An HDFS mounter is the next logical step,
but this feature set is already being worked on in parallel to this change and
falls outside of the scope of this specification.
Unmounting is a natural extension of this class, but is not covered in this
specification.
Alternatives
------------
A more-seamless approach to manila share storage and data sourcing could be
attempted, in which no API is exposed to the user, and shares are automatically
mounted and unmounted when resources on the share in question are needed (as
referenced in a data source URL or binary storage path). However, giving the
user the ability to mount and unmount shares at will may allow use cases which
we do not anticipate, and particularly in the context of usage of a sahara-
provisioned cluster without use of the EDP API, the new API is critical.
It would also be possible to attempt to wrap manila share creation (or even
share network creation or network router configuration) in sahara. It seems
reasonable, however, to assert that this would be an overstep of our charter,
and that asking users to create shares directly through manila will allow them
much fuller and up-to-date access to manila's feature set.
On the sahara implementation side, it would be possible to create a new
'share' resource and table, for ease of update and compositional modelling.
However, shares will likely never be a top-level noun in sahara; it seems that
a field is a better fit for the degree of share management we intend to
undertake than an entire resource.
It should be noted that this specification does not attempt to deal with the
question of filesystem driver installation across n distributions of Linux and
m filesystem types; such an effort is better suited to many specifications and
change sets than one. For the first stage of this effort, NFS will be used as
the test reference filesystem type.
Note that both binary storage and data source integration are intentionally
not handled here. A binary storage specification will build on this spec, but
this spec is being posted independently such that the engineers working on data
source integration can propose revisions to only the changes relevant to their
needs.
Data model impact
-----------------
A new 'shares' TEXT field will be added to both node groups and node group
templates.
REST API impact
---------------
A new 'shares' field will be added to the resource for both node groups and
node group templates. This field will only allow create functionality in the
initial change, as cluster update is currently a sticking point in our API.
Other end user impact
---------------------
Python-saharaclient will need to be made aware of the new shares field on all
supported resources.
Deployer impact
---------------
None.
Developer impact
----------------
None.
Sahara-image-elements impact
----------------------------
None for the initial change; addition of specialized fs drivers in the
future may require image changes.
Sahara-dashboard / Horizon impact
---------------------------------
The share mounting feature in Horizon will likely require a separate tab on
all affected resources, and is left for a separate spec.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
egafford
Secondary assignee/reviewer:
croberts
Work Items
----------
* API Resource modification and call validation.
* DB model modification and testing.
* Manila client integration with Sahara.
* Logical glue code on cluster provisioning.
* ShareMounter abstraction and NFS impl.
* Unit testing.
* Integration testing as feasible (will require manila in CI env for full CI.)
* Update of API WADL site.
* Horizon changes (in separate spec).
* Documentation.
Dependencies
============
This feature introduces a new dependency on python-manilaclient.
Testing
=======
Unit testing is assumed; beyond this, full integration testing will depend on
the feasibility of adding a manila endpoint to our CI environment. If this is
feasible, then our testing path becomes clear; if it is not, then gated
integration testing will not be possible.
Documentation Impact
====================
This feature will require documentation in features.rst, and will drive changes
to the api documentation.
References
==========
See https://wiki.openstack.org/wiki/Manila/API if unfamiliar with manila
operations.

153
specs/liberty/recommend-configuration.rst
View File

@ -1,153 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=================================================================
Provide ability to configure most important configs automatically
=================================================================
https://blueprints.launchpad.net/sahara/+spec/recommend-configuration
Now users manually should configure most important hadoop configurations.
It would be friendly to provide advices about cluster configurations for
users.
Problem description
===================
Now users manually should configure most important hadoop configs, but it's
required to have advanced knowledge in Hadoop. Most configs are complicated
and not all users know them. We can
provide advices about cluster configuration and automatically configure
few basic configs, that will improve user experience. Created workaround
can extended in future with new confiuguration and advices.
Proposed change
===============
It's proposed to add calculator, which would automatically configure
most important configurations in dependency cluster specification:
available disk space, ram, cpu, and so on. Such calculator already
implemented in Ambari (see [1] and [2]), and we can use it as well. We should
have ability to switch off autoconfiguration and if user also manually
configured some hadoop config, autoconfiguration also will not be applied.
The following list of configs will be configured, using formulas from [1] and
[2]:
* yarn.nodemanager.resource.memory-mb
* yarn.scheduler.minimum-allocation-mb
* yarn.scheduler.maximum-allocation-mb
* yarn.app.mapreduce.am.resource.mb
* yarn.app.mapreduce.am.command-opts
* mapreduce.map.memory.mb
* mapreduce.reduce.memory.mb
* mapreduce.map.java.opts
* mapreduce.reduce.java.opts
* mapreduce.task.io.sort.mb
Also as a simple example we can autoconfigure before cluster validation
``dfs.replication`` if amout of ``datanodes`` less than default value.
Also it's required to add new plugin SPI method ``recommend_configs`` which
will autoconfigure cluster configs.
Alternatives
------------
None
Data model impact
-----------------
It's required to add new column ``use_autoconfig`` to cluster,
cluster_template, node_group, node_group_template, templates_relations
objects in DB. By default ``use_autoconfig`` will be ``True``. If
``use_autoconfig`` is ``False``, then we will not use autoconfiguration
during cluster creation. If none of the configs from the list above are
configured manually and ``use_autoconfig`` is ``True``, then we will
autoconfigure configs from list above. Same behaviour will be used for
node_groups configs autoconfiguration.
REST API impact
---------------
Need to support of switch off autoconfiguration.
Other end user impact
---------------------
Need to support of switch off autoconfiguration via python-saharaclient.
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
Need to add new checkbox which will allow to swith off autoconfiguration from
Horizon during cluster creation/scaling. If plugin doesn't support autoconfig
this checkbox will not be displayed. We can use ``_info`` field at [3] for
field.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
vgridnev
Other contributors:
sreshetniak
Work Items
----------
Proposed change will consists with following steps:
* Implement new plugin SPI method which will provide configuration advices;
* Add support of this method in following plugins: CDH, Vanilla 2.6.0,
Spark (``dfs.replication`` only);
* Provide ability to switch on autoconfiguration via UI;
* Provide ability to switch on autoconfiguration via saharaclient;
* Update WADL docs about new feilds objects.
Dependencies
============
Depends on Openstack requirements
Testing
=======
Unit tests will be implemented for this feature. Sahara CI also can start use
autoconfiguration as well.
Documentation Impact
====================
Need to document feature and all rules, which will be used for
autoconfiguration.
References
==========
[1] https://apache.googlesource.com/ambari/+/a940986517cbfeb2ef889f0d8a45579b27adad1c/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py
[2] https://apache.googlesource.com/ambari/+/a940986517cbfeb2ef889f0d8a45579b27adad1c/ambari-server/src/main/resources/stacks/HDP/2.1/services/stack_advisor.py
[3] https://github.com/openstack/sahara/blob/master/sahara/service/api.py#L188

109
specs/liberty/sahara-heat-wait-conditions.rst
View File

@ -1,109 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===========================
Heat WaitConditions support
===========================
https://blueprints.launchpad.net/sahara/+spec/sahara-heat-wait-conditions
Before Heat engine in Sahara Nova was continuously asked for fixed and
assigned floating IP and for active SSH connections to VMs. To get rid of
such polling mechanism suggested to use Heat WaitConditions feature.
Problem description
===================
Now Sahara checks instances availability via SSH. Wait Condition resource
supports reporting signals to Heat. We should report signal to Heat about
booting instance.
Proposed change
===============
Add WaitCondition resource to Sahara Heat template.
Alternatives
------------
Using SSH for polling instance accessible.
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
sreshetniak
Work Items
----------
Add Wait Condition support to Sahara
Dependencies
============
WaitCondition requires pre-installed cloud-init.
Testing
=======
Need to add unit tests for this feature.
Integration tests will cover this feature.
Documentation Impact
====================
None
References
==========
http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::WaitCondition

163
specs/liberty/scenario-test-config-template.rst
View File

@ -1,163 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==============================================
Use Templates for Scenario Tests Configuration
==============================================
https://blueprints.launchpad.net/sahara/+spec/scenario-test-config-template
Users that want to run the scenario tests available in the Sahara repository
need to modify the provided YAML files, which are really template files even
if not marked as such. The usability of this process could be improved by
extending the test runner to read the templates and perform the substitution
of the required variables instead.
Problem description
===================
The scenario tests provided in the Sahara repository are template files
even if they are simply marked as YAML files; this is not clear from the
README.rst file.
Users that want to run them need to manually replace the required variables
(the variables are needed because they depend on the environment: host ip,
credentials, network type, flavors used, etc).
This step needs to be done both by developers and testers, and by wrapper
scripts used to run the script on a CI. The repository of the main Sahara CI,
sahara-ci-config, contains code which replaces the variables:
https://github.com/stackforge/sahara-ci-config/blob/master/slave-scripts/functions-common.sh#L148
Proposed change
===============
The current template files (mostly under etc/sahara-ci right now) need to be
properly identified as templates. The chosen format is Mako, because it is
already a dependency for the scenario test runner (runner.py).
The files will be marked using a special suffix (file.yaml.mako)
and the variables used will be converted in Mako format.
The usage of templating would be limited to variable replacement, which means
no logic in the templates.
The test runner will continue to handle normal YAML files as usual, in
addition to template files.
runner.py will also take a simply INI-style file with the values for
the variables used in the template. It will be used by the runner
to generate the real YAML files used for the input (in addition to
the normal YAML files, if specified).
A missing value for some variable (key not available in the INI file)
will raise an exception and lead to the runner termination.
This differs from the current behavior of sahara-ci-config code, where a
missing values just prints a warning, but given that this would likely bring
to a failure, enforcing an early termination limit the resource consumption.
The current sahara-ci-config code allows to specify more details for the
replacement variables, like specific end keys where to stop the match for
a certain key, but they are likely not needed with a proper choice of names
for the variables.
Finally, sahara/tests/scenario/README.rst should be changed to document the
currently used variables and the instruction on how to feed the key/value
configuration file.
The code in sahara-ci-config should be changed as well to create such
configuration file and to use the new names of the template files for the
respective tests.
Alternatives
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Other end user impact
---------------------
None
Deployer impact
---------------
CI systems which runs tox -e scenario should be updated to use the new
filenames.
Developer impact
----------------
Developers/QE running tests need to use the new template names.
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
ltoscano
Work Items
----------
The change will be implemented as follow:
1. allow runner.py to use mako template files as input;
2. copy the existing files to the new name and use non-ambiguous variable
names for templates;
3. change sahara-ci scripts to use to set the new variables and use the
renamed template files;
4. remove the old yaml files;
5. (optional) clean up unnedded code (insert_scenario_value, etc)
Repositories affected:
- sahara: 1, 2, 4
- sahara-ci-config: 3, 5
Dependencies
============
None
Testing
=======
Successful CI run will ensure that the new code did not regress the existing
scenarios.
Documentation Impact
====================
sahara/tests/scenario/README.rst needs to be updated.
References
==========
None

153
specs/liberty/shared-protected-resources.rst
View File

@ -1,153 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=========================================
Support of shared and protected resources
=========================================
https://blueprints.launchpad.net/sahara/+spec/shared-protected-resources
This specification proposes to add ability of creation and modification of
shared across tenants and protected from updates objects.
Problem description
===================
Currently all objects created by Sahara are visible only from the tenant in
which they were created and not insured from occasional modification or
deletion.
Proposed change
===============
This specification proposes to add ``is_public`` and ``is_protected`` boolean
fields to all Sahara objects that can be accessed through REST API. They will
be added to clusters, cluster templates, node group templates, data sources,
job executions, jobs, job binaries and job binary internals.
All this objects can be created with enabled ``is_public`` and
``is_protected`` parameters which can be updated after creation with
corresponding API call. Both of them will be False by default.
If some object has ``is_public`` field set to ``True``, it means that it's
visible not only from the tenant in which it was created, but from any other
tenants too.
If some object has ``is_protected`` field set to ``True``, it means that it
could not be modified (updated, scaled, canceled or deleted) unless this field
will be set to ``False``. If ``is_protected`` parameter is set to ``True``,
object can be modified only if ``is_protected=False`` will be supplied in
update request.
Public objects created in one tenant can be used by other tenants (for example,
cluster can be created from public cluster template which is created in another
tenant), but to prevent management of resources in different tenants,
operations like update, delete, cancel and scale will be possible only from
tenant in which object was created.
To control this restrictions, a couple of methods will be implemented in
``sahara.service.validation.acl``:
.. sourcecode:: python
def check_tenant_for_delete(context, object)
def check_tenant_for_update(context, object)
def check_protected_from_delete(object)
def check_protected_from_update(object, data)
``check_tenant_for_*`` will compare tenant_id in context with object tenant_id
and if they different, raise an error. But this check should be skipped for
periodics as there is no tenant_id in context in this case.
``check_protected_from_delete`` will check ``is_protected`` field and if it's
set to True, raise an error.
``check_protected_from_update`` will additionally check that ``is_protected``
field wasn't changed to ``False`` with update data.
This methods will be called mostly in ``sahara.db.sqlalchemy.api`` inside of
update and delete methods that make only db changes. But for cluster_create,
cluster_scale, job_execute, job_execution_cancel and job_execution_delete
operations they will be called during validation before api calls.
Alternatives
------------
None
Data model impact
-----------------
Two extra fields ``is_public`` and ``is_protected`` will be added to
objects listed above.
REST API impact
---------------
New API calls will not be added, but existing ones will be updated to support
new fields.
Other end user impact
---------------------
Saharaclient API will be updated to support new fields.
Deployer impact
---------------
None
Developer impact
----------------
None
Sahara-image-elements impact
----------------------------
None
Sahara-dashboard / Horizon impact
---------------------------------
``is_public`` and ``is_protected`` checkboxes will be added to ``Update`` and
``Create`` panels of each object.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
apavlov-n
Work Items
----------
* Adding new fields ``is_public`` and ``is_protected`` to objects listed above;
* Implementation of validations, described above;
* Updating saharaclient with corresponding changed;
* Documentation about new features will be added.
Dependencies
============
None
Testing
=======
Unit tests will be added and a lot of manual testing.
Documentation Impact
====================
All changes will be documented.
References
==========
None

Some files were not shown because too many files have changed in this diff Show More