openstack
/
sahara-plugin-storm
Code Issues Proposed changes

Retire Sahara: remove repo content 

Sahara project is retiring
- https://review.opendev.org/c/openstack/governance/+/919374

this commit remove the content of this project repo

Depends-On: https://review.opendev.org/c/openstack/project-config/+/919376
Change-Id: I7a1fdd10f0655b57ec0a9914f6ebffd0c179ae4c
This commit is contained in:
Ghanshyam Mann 2024-05-10 17:29:17 -07:00
parent 4085a75197
commit d90338ed0b
66 changed files with 8 additions and 2398 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

31
.gitignore vendored
View File

@ -1,31 +0,0 @@
*.egg-info
*.egg[s]
*.log
*.py[co]
.coverage
.testrepository
.tox
.stestr
.venv
.idea
.un~
AUTHORS
ChangeLog
build
cover
develop-eggs
dist
doc/build
doc/html
eggs
etc/sahara.conf
etc/sahara/*.conf
etc/sahara/*.topology
sdist
target
tools/lintstack.head.py
tools/pylint_exceptions
doc/source/sample.config
# Files created by releasenotes build
releasenotes/build

3
.stestr.conf
View File

@ -1,3 +0,0 @@
[DEFAULT]
test_path=./sahara_plugin_storm/tests/unit
top_dir=./

6
.zuul.yaml
View File

@ -1,6 +0,0 @@
- project:
templates:
- check-requirements
- openstack-python3-zed-jobs
- publish-openstack-docs-pti
- release-notes-jobs-python3

19
CONTRIBUTING.rst
View File

@ -1,19 +0,0 @@
The source repository for this project can be found at:
https://opendev.org/openstack/sahara-plugin-storm
Pull requests submitted through GitHub are not monitored.
To start contributing to OpenStack, follow the steps in the contribution guide
to set up and use Gerrit:
https://docs.openstack.org/contributors/code-and-documentation/quick-start.html
Bugs should be filed on Storyboard:
https://storyboard.openstack.org/#!/project/openstack/sahara-plugin-storm
For more specific information about contributing to this repository, see the
sahara-plugin-storm contributor guide:
https://docs.openstack.org/sahara-plugin-storm/latest/contributor/contributing.html

175
LICENSE
View File

@ -1,175 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

44
README.rst
View File

@ -1,38 +1,10 @@
========================
Team and repository tags
========================
This project is no longer maintained.
.. image:: https://governance.openstack.org/tc/badges/sahara.svg
:target: https://governance.openstack.org/tc/reference/tags/index.html
.. Change things from this point on
OpenStack Data Processing ("Sahara") Storm Plugin
==================================================
OpenStack Sahara Storm Plugin provides the users the option to
start Storm clusters on OpenStack Sahara.
Check out OpenStack Sahara documentation to see how to deploy the
Storm Plugin.
Sahara at wiki.openstack.org: https://wiki.openstack.org/wiki/Sahara
Storyboard project: https://storyboard.openstack.org/#!/project/openstack/sahara-plugin-storm
Sahara docs site: https://docs.openstack.org/sahara/latest/
Quickstart guide: https://docs.openstack.org/sahara/latest/user/quickstart.html
How to participate: https://docs.openstack.org/sahara/latest/contributor/how-to-participate.html
Source: https://opendev.org/openstack/sahara-plugin-storm
Bugs and feature requests: https://storyboard.openstack.org/#!/project/openstack/sahara-plugin-storm
Release notes: https://docs.openstack.org/releasenotes/sahara-plugin-storm/
License
-------
Apache License Version 2.0 http://www.apache.org/licenses/LICENSE-2.0
The contents of this repository are still available in the Git
source code management system. To see the contents of this
repository before it reached its end of life, please check out the
previous commit with "git checkout HEAD^1".
For any further questions, please email
openstack-discuss@lists.openstack.org or join #openstack-dev on
OFTC.

1
babel.cfg
View File

@ -1 +0,0 @@
[python: **.py]

9
doc/requirements.txt
View File

@ -1,9 +0,0 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
openstackdocstheme>=2.2.1 # Apache-2.0
os-api-ref>=1.4.0 # Apache-2.0
reno>=3.1.0 # Apache-2.0
sphinx>=2.0.0,!=2.1.0 # BSD
sphinxcontrib-httpdomain>=1.3.0 # BSD
whereto>=0.3.0 # Apache-2.0

214
doc/source/conf.py
View File

@ -1,214 +0,0 @@
# -*- coding: utf-8 -*-
#
# sahara-plugin-storm documentation build configuration file.
#
# -- General configuration -----------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'reno.sphinxext',
'openstackdocstheme',
]
# openstackdocstheme options
openstackdocs_repo_name = 'openstack/sahara-plugin-storm'
openstackdocs_pdf_link = True
openstackdocs_use_storyboard = True
openstackdocs_projects = [
'sahara'
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
copyright = u'2015, Sahara team'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = []
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'native'
# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []
# -- Options for HTML output ---------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'openstackdocs'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
#html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'saharastormplugin-testsdoc'
# -- Options for LaTeX output --------------------------------------------------
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass [howto/manual]).
latex_documents = [
('index', 'doc-sahara-plugin-storm.tex', u'Sahara Storm Plugin Documentation',
u'Sahara team', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# If true, show page references after internal links.
#latex_show_pagerefs = False
# If true, show URL addresses after external links.
#latex_show_urls = False
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_domain_indices = True
smartquotes_excludes = {'builders': ['latex']}
# -- Options for manual page output --------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'sahara-plugin-storm', u'sahara-plugin-storm Documentation',
[u'Sahara team'], 1)
]
# If true, show URL addresses after external links.
#man_show_urls = False
# -- Options for Texinfo output ------------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'sahara-plugin-storm', u'sahara-plugin-storm Documentation',
u'Sahara team', 'sahara-plugin-storm', 'One line description of project.',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
#texinfo_appendices = []
# If false, no module index is generated.
#texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'

14
doc/source/contributor/contributing.rst
View File

@ -1,14 +0,0 @@
============================
So You Want to Contribute...
============================
For general information on contributing to OpenStack, please check out the
`contributor guide <https://docs.openstack.org/contributors/>`_ to get started.
It covers all the basics that are common to all OpenStack projects: the
accounts you need, the basics of interacting with our Gerrit review system, how
we communicate as a community, etc.
sahara-plugin-storm is maintained by the OpenStack Sahara project.
To understand our development process and how you can contribute to it, please
look at the Sahara project's general contributor's page:
http://docs.openstack.org/sahara/latest/contributor/contributing.html

8
doc/source/contributor/index.rst
View File

@ -1,8 +0,0 @@
=================
Contributor Guide
=================
.. toctree::
:maxdepth: 2
contributing

8
doc/source/index.rst
View File

@ -1,8 +0,0 @@
Storm plugin for Sahara
=======================
.. toctree::
:maxdepth: 2
user/index
contributor/index

BIN
doc/source/user/.quickstart.rst.un~
View File

Binary file not shown.

8
doc/source/user/index.rst
View File

@ -1,8 +0,0 @@
==========
User Guide
==========
.. toctree::
:maxdepth: 2
storm-plugin

83
doc/source/user/storm-plugin.rst
View File

@ -1,83 +0,0 @@
Storm Plugin
============
The Storm plugin for sahara provides a way to provision Apache Storm clusters
on OpenStack in a single click and in an easily repeatable fashion.
Currently Storm is installed in standalone mode, with no YARN support.
Images
------
For cluster provisioning, prepared images should be used.
.. list-table:: Support matrix for the `storm` plugin
:widths: 15 15 20 15 35
:header-rows: 1
* - Version
(image tag)
- Distribution
- Build method
- Version
(build parameter)
- Notes
* - 1.2
- Ubuntu 16.04
- sahara-image-create
- 1.2.1, 1.2.0
- both versions are supported by the same image tag
* - 1.1.0
- Ubuntu 16.04
- sahara-image-create
- 1.1.1, 1.1.0
- both versions are supported by the same image tag
For more information about building image, refer to
:sahara-doc:`Sahara documentation <user/building-guest-images.html>`.
The Storm plugin requires an image to be tagged in the sahara image registry
with two tags: 'storm' and '<Storm version>' (e.g. '1.1.0').
The image requires a username. For more information, refer to the
:sahara-doc:`registering image <user/registering-image.html>` section
of the Sahara documentation.
Note that the Storm cluster is deployed using the scripts available in the
Storm distribution, which allow the user to start all services (nimbus,
supervisors and zookeepers), stop all services and so on. As such Storm is not
deployed as a standard Ubuntu service and if the virtual machines are rebooted,
Storm will not be restarted.
Storm configuration
-------------------
Storm needs few parameters to work and has sensible defaults. If needed they
can be changed when creating the sahara cluster template. No node group
options are available.
Once the cluster is ready, connect with ssh to the master using the `ubuntu`
user and the appropriate ssh key. Storm is installed in `/usr/local/storm` and
should be completely configured and ready to start executing jobs. At the
bottom of the cluster information page from the OpenStack dashboard, a link to
the Storm web interface is provided.
Cluster Validation
------------------
When a user creates a Storm cluster using the Storm plugin, the cluster
topology requested by user is verified for consistency.
Currently there are the following limitations in cluster topology for the
Storm plugin:
+ Cluster must contain exactly one Storm nimbus
+ Cluster must contain at least one Storm supervisor
+ Cluster must contain at least one Zookeeper node
The tested configuration has nimbus, supervisor, and Zookeeper processes each
running on their own nodes.
Another possible configuration is one node with nimbus alone, and additional
nodes each with supervisor and Zookeeper processes together.

BIN
releasenotes/notes/.ambari26-image-pack-88c9aad59bf635b2.yaml.un~
View File

Binary file not shown.

BIN
releasenotes/notes/.boot-from-volume-e7078452fac1a4a0.yaml.un~
View File

Binary file not shown.

BIN
releasenotes/notes/.cdh-513-bdce0d5d269d8f20.yaml.un~
View File

Binary file not shown.

BIN
releasenotes/notes/.enable-mutable-configuration-2dd6b7a0e0fe4437.yaml.un~
View File

Binary file not shown.

BIN
releasenotes/notes/.spark-2.3-0277fe9feae6668a.yaml.un~
View File

Binary file not shown.

BIN
releasenotes/notes/.spark-image-pack-abe0bf0ec50ff804.yaml.un~
View File

Binary file not shown.

BIN
releasenotes/notes/.storm-1.2-af75fedb413de56a.yaml.un~
View File

Binary file not shown.

BIN
releasenotes/notes/.updating-plugins-versions-b8d27764178c3cdd.yaml.un~
View File

Binary file not shown.

6
releasenotes/notes/drop-py2-7-91adeb48d903b52b.yaml
View File

@ -1,6 +0,0 @@
---
upgrade:
- |
Python 2.7 support has been dropped. Last release of sahara and its plugins
to support python 2.7 is OpenStack Train. The minimum version of Python now
supported by sahara and its plugins is Python 3.6.

0
releasenotes/source/_static/.placeholder
View File

0
releasenotes/source/_templates/.placeholder
View File

210
releasenotes/source/conf.py
View File

@ -1,210 +0,0 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Sahara Release Notes documentation build configuration file
extensions = [
'reno.sphinxext',
'openstackdocstheme'
]
# openstackdocstheme options
openstackdocs_repo_name = 'openstack/sahara-plugin-storm'
openstackdocs_use_storyboard = True
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# General information about the project.
copyright = u'2015, Sahara Developers'
# Release do not need a version number in the title, they
# cover multiple versions.
# The full version, including alpha/beta/rc tags.
release = ''
# The short X.Y version.
version = ''
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = []
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'native'
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'openstackdocs'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
# html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
# html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
# html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
# html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
# html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
# html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
# html_extra_path = []
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
# html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
# html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
# html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
# html_additional_pages = {}
# If false, no module index is generated.
# html_domain_indices = True
# If false, no index is generated.
# html_use_index = True
# If true, the index is split into individual pages for each letter.
# html_split_index = False
# If true, links to the reST sources are added to the pages.
# html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
# html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
# html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
# html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
# html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'SaharaStormReleaseNotesdoc'
# -- Options for LaTeX output ---------------------------------------------
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
('index', 'SaharaStormReleaseNotes.tex',
u'Sahara Storm Plugin Release Notes Documentation',
u'Sahara Developers', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
# latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
# latex_use_parts = False
# If true, show page references after internal links.
# latex_show_pagerefs = False
# If true, show URL addresses after external links.
# latex_show_urls = False
# Documents to append as an appendix to all manuals.
# latex_appendices = []
# If false, no module index is generated.
# latex_domain_indices = True
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'saharastormreleasenotes',
u'Sahara Storm Plugin Release Notes Documentation',
[u'Sahara Developers'], 1)
]
# If true, show URL addresses after external links.
# man_show_urls = False
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'SaharaStormReleaseNotes',
u'Sahara Storm Plugin Release Notes Documentation',
u'Sahara Developers', 'SaharaStormReleaseNotes',
'One line description of project.',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
# texinfo_appendices = []
# If false, no module index is generated.
# texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
# texinfo_show_urls = 'footnote'
# If true, do not generate a @detailmenu in the "Top" node's menu.
# texinfo_no_detailmenu = False
# -- Options for Internationalization output ------------------------------
locale_dirs = ['locale/']

15
releasenotes/source/index.rst
View File

@ -1,15 +0,0 @@
===================================
Sahara Storm Plugin Release Notes
===================================
.. toctree::
:maxdepth: 1
unreleased
yoga
xena
wallaby
victoria
ussuri
train
stein

44
releasenotes/source/locale/de/LC_MESSAGES/releasenotes.po
View File

@ -1,44 +0,0 @@
# Andreas Jaeger <jaegerandi@gmail.com>, 2019. #zanata
# Andreas Jaeger <jaegerandi@gmail.com>, 2020. #zanata
msgid ""
msgstr ""
"Project-Id-Version: sahara-plugin-storm\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2020-04-24 23:45+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2020-04-25 10:39+0000\n"
"Last-Translator: Andreas Jaeger <jaegerandi@gmail.com>\n"
"Language-Team: German\n"
"Language: de\n"
"X-Generator: Zanata 4.3.3\n"
"Plural-Forms: nplurals=2; plural=(n != 1)\n"
msgid "Current Series Release Notes"
msgstr "Aktuelle Serie Releasenotes"
msgid ""
"Python 2.7 support has been dropped. Last release of sahara and its plugins "
"to support python 2.7 is OpenStack Train. The minimum version of Python now "
"supported by sahara and its plugins is Python 3.6."
msgstr ""
"Python 2.7 Unterstützung wurde beendet. Der letzte Release von Sahara und "
"seinen Plugins der Python 2.7 unterstützt ist OpenStack Train. Die minimal "
"Python Version welche von Sahara und seinen Plugins unterstützt wird, ist "
"Python 3.6."
msgid "Sahara Storm Plugin Release Notes"
msgstr "Sahara Storm Plugin Releasenotes"
msgid "Stein Series Release Notes"
msgstr "Stein Serie Releasenotes"
msgid "Train Series Release Notes"
msgstr "Train Serie Releasenotes"
msgid "Upgrade Notes"
msgstr "Aktualisierungsnotizen"
msgid "Ussuri Series Release Notes"
msgstr "Ussuri Serie Releasenotes"

48
releasenotes/source/locale/en_GB/LC_MESSAGES/releasenotes.po
View File

@ -1,48 +0,0 @@
# Andi Chandler <andi@gowling.com>, 2020. #zanata
msgid ""
msgstr ""
"Project-Id-Version: sahara-plugin-storm\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2020-10-07 22:07+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2020-11-04 12:49+0000\n"
"Last-Translator: Andi Chandler <andi@gowling.com>\n"
"Language-Team: English (United Kingdom)\n"
"Language: en_GB\n"
"X-Generator: Zanata 4.3.3\n"
"Plural-Forms: nplurals=2; plural=(n != 1)\n"
msgid "3.0.0"
msgstr "3.0.0"
msgid "Current Series Release Notes"
msgstr "Current Series Release Notes"
msgid ""
"Python 2.7 support has been dropped. Last release of sahara and its plugins "
"to support python 2.7 is OpenStack Train. The minimum version of Python now "
"supported by sahara and its plugins is Python 3.6."
msgstr ""
"Python 2.7 support has been dropped. Last release of sahara and its plugins "
"to support python 2.7 is OpenStack Train. The minimum version of Python now "
"supported by sahara and its plugins is Python 3.6."
msgid "Sahara Storm Plugin Release Notes"
msgstr "Sahara Storm Plugin Release Notes"
msgid "Stein Series Release Notes"
msgstr "Stein Series Release Notes"
msgid "Train Series Release Notes"
msgstr "Train Series Release Notes"
msgid "Upgrade Notes"
msgstr "Upgrade Notes"
msgid "Ussuri Series Release Notes"
msgstr "Ussuri Series Release Notes"
msgid "Victoria Series Release Notes"
msgstr "Victoria Series Release Notes"

21
releasenotes/source/locale/id/LC_MESSAGES/releasenotes.po
View File

@ -1,21 +0,0 @@
# Andreas Jaeger <jaegerandi@gmail.com>, 2019. #zanata
msgid ""
msgstr ""
"Project-Id-Version: sahara-plugin-storm\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2019-09-20 17:21+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2019-09-25 06:11+0000\n"
"Last-Translator: Andreas Jaeger <jaegerandi@gmail.com>\n"
"Language-Team: Indonesian\n"
"Language: id\n"
"X-Generator: Zanata 4.3.3\n"
"Plural-Forms: nplurals=1; plural=0\n"
msgid "Current Series Release Notes"
msgstr "Catatan Rilis Seri Saat Ini"
msgid "Stein Series Release Notes"
msgstr "Catatan Rilis Seri Stein"

24
releasenotes/source/locale/ne/LC_MESSAGES/releasenotes.po
View File

@ -1,24 +0,0 @@
# Surit Aryal <aryalsurit@gmail.com>, 2019. #zanata
msgid ""
msgstr ""
"Project-Id-Version: sahara-plugin-storm\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2019-07-23 14:26+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2019-08-02 04:40+0000\n"
"Last-Translator: Surit Aryal <aryalsurit@gmail.com>\n"
"Language-Team: Nepali\n"
"Language: ne\n"
"X-Generator: Zanata 4.3.3\n"
"Plural-Forms: nplurals=2; plural=(n != 1)\n"
msgid "Current Series Release Notes"
msgstr "वर्तमान श्रृंखला रिलीज नोट्स"
msgid "Sahara Storm Plugin Release Notes"
msgstr "Sahara Storm प्लगइन रिलीज नोट्स"
msgid "Stein Series Release Notes"
msgstr "Stein Series रिलीज नोट्स"

6
releasenotes/source/stein.rst
View File

@ -1,6 +0,0 @@
===================================
Stein Series Release Notes
===================================
.. release-notes::
:branch: stable/stein

6
releasenotes/source/train.rst
View File

@ -1,6 +0,0 @@
==========================
Train Series Release Notes
==========================
.. release-notes::
:branch: stable/train

5
releasenotes/source/unreleased.rst
View File

@ -1,5 +0,0 @@
==============================
Current Series Release Notes
==============================
.. release-notes::

6
releasenotes/source/ussuri.rst
View File

@ -1,6 +0,0 @@
===========================
Ussuri Series Release Notes
===========================
.. release-notes::
:branch: stable/ussuri

6
releasenotes/source/victoria.rst
View File

@ -1,6 +0,0 @@
=============================
Victoria Series Release Notes
=============================
.. release-notes::
:branch: stable/victoria

6
releasenotes/source/wallaby.rst
View File

@ -1,6 +0,0 @@
============================
Wallaby Series Release Notes
============================
.. release-notes::
:branch: stable/wallaby

6
releasenotes/source/xena.rst
View File

@ -1,6 +0,0 @@
=========================
Xena Series Release Notes
=========================
.. release-notes::
:branch: stable/xena

6
releasenotes/source/yoga.rst
View File

@ -1,6 +0,0 @@
=========================
Yoga Series Release Notes
=========================
.. release-notes::
:branch: stable/yoga

15
requirements.txt
View File

@ -1,15 +0,0 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
pbr!=2.1.0,>=2.0.0 # Apache-2.0
Babel!=2.4.0,>=2.3.4 # BSD
eventlet>=0.26.0 # MIT
oslo.i18n>=3.15.3 # Apache-2.0
oslo.log>=3.36.0 # Apache-2.0
oslo.serialization!=2.19.1,>=2.18.0 # Apache-2.0
oslo.utils>=3.33.0 # Apache-2.0
requests>=2.14.2 # Apache-2.0
sahara>=10.0.0.0b1
six>=1.10.0 # MIT

0
sahara_plugin_storm/__init__.py
View File

26
sahara_plugin_storm/i18n.py
View File

@ -1,26 +0,0 @@
# Copyright (c) 2014 Mirantis Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# It's based on oslo.i18n usage in OpenStack Keystone project and
# recommendations from https://docs.openstack.org/oslo.i18n/latest/
# user/usage.html
import oslo_i18n
_translators = oslo_i18n.TranslatorFactory(domain='sahara_plugin_storm')
# The primary translation function using the well-known name "_"
_ = _translators.primary

44
sahara_plugin_storm/locale/de/LC_MESSAGES/sahara_plugin_storm.po
View File

@ -1,44 +0,0 @@
# Andreas Jaeger <jaegerandi@gmail.com>, 2019. #zanata
msgid ""
msgstr ""
"Project-Id-Version: sahara-plugin-storm VERSION\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2019-09-20 17:21+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2019-09-25 06:10+0000\n"
"Last-Translator: Andreas Jaeger <jaegerandi@gmail.com>\n"
"Language-Team: German\n"
"Language: de\n"
"X-Generator: Zanata 4.3.3\n"
"Plural-Forms: nplurals=2; plural=(n != 1)\n"
msgid "1 or more"
msgstr "1 oder mehr"
msgid "Push configs to nodes"
msgstr "Push-Konfigurationen zu Knoten"
msgid "Rebalance Topology"
msgstr "Rebalance-Topologie"
#, python-format
msgid "Storm plugin cannot scale nodegroup with processes: %s"
msgstr "Das Storm-Plugin kann Knotengruppen nicht mit Prozessen skalieren:%s"
msgid "Storm {base} required to run {type} jobs"
msgstr "Storm {base} ist zum Ausführen von {type} Jobs erforderlich"
msgid ""
"This plugin provides an ability to launch Storm cluster without any "
"management consoles."
msgstr ""
"Dieses Plugin bietet die Möglichkeit, den Storm-Cluster ohne "
"Verwaltungskonsolen zu starten."
#, python-format
msgid "Unable to get parameter '%(param_name)s' from service %(service)s"
msgstr ""
"Der Parameter '%(param_name)s' konnte nicht vom Service %(service)s "
"abgerufen werden"

42
sahara_plugin_storm/locale/en_GB/LC_MESSAGES/sahara_plugin_storm.po
View File

@ -1,42 +0,0 @@
# Andi Chandler <andi@gowling.com>, 2020. #zanata
msgid ""
msgstr ""
"Project-Id-Version: sahara-plugin-storm VERSION\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2020-10-07 22:08+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2020-11-04 12:50+0000\n"
"Last-Translator: Andi Chandler <andi@gowling.com>\n"
"Language-Team: English (United Kingdom)\n"
"Language: en_GB\n"
"X-Generator: Zanata 4.3.3\n"
"Plural-Forms: nplurals=2; plural=(n != 1)\n"
msgid "1 or more"
msgstr "1 or more"
msgid "Push configs to nodes"
msgstr "Push configs to nodes"
msgid "Rebalance Topology"
msgstr "Rebalance Topology"
#, python-format
msgid "Storm plugin cannot scale nodegroup with processes: %s"
msgstr "Storm plugin cannot scale nodegroup with processes: %s"
msgid "Storm {base} required to run {type} jobs"
msgstr "Storm {base} required to run {type} jobs"
msgid ""
"This plugin provides an ability to launch Storm cluster without any "
"management consoles."
msgstr ""
"This plugin provides an ability to launch Storm cluster without any "
"management consoles."
#, python-format
msgid "Unable to get parameter '%(param_name)s' from service %(service)s"
msgstr "Unable to get parameter '%(param_name)s' from service %(service)s"

43
sahara_plugin_storm/locale/id/LC_MESSAGES/sahara_plugin_storm.po
View File

@ -1,43 +0,0 @@
# suhartono <cloudsuhartono@gmail.com>, 2019. #zanata
msgid ""
msgstr ""
"Project-Id-Version: sahara-plugin-storm VERSION\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2019-09-30 09:31+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2019-10-06 03:16+0000\n"
"Last-Translator: suhartono <cloudsuhartono@gmail.com>\n"
"Language-Team: Indonesian\n"
"Language: id\n"
"X-Generator: Zanata 4.3.3\n"
"Plural-Forms: nplurals=1; plural=0\n"
msgid "1 or more"
msgstr "1 atau lebih"
msgid "Push configs to nodes"
msgstr "Dorong konfigurasi ke node"
msgid "Rebalance Topology"
msgstr "Rebalance Topology"
#, python-format
msgid "Storm plugin cannot scale nodegroup with processes: %s"
msgstr "Plugin Storm tidak dapat menskala nodegroup dengan proses: %s"
msgid "Storm {base} required to run {type} jobs"
msgstr "Storm {base} diperlukan untuk menjalankan jobs {type}"
msgid ""
"This plugin provides an ability to launch Storm cluster without any "
"management consoles."
msgstr ""
"Plugin ini menyediakan kemampuan untuk meluncurkan Storm cluster tanpa "
"konsol manajemen."
#, python-format
msgid "Unable to get parameter '%(param_name)s' from service %(service)s"
msgstr ""
"Tidak dapat memperoleh parameter '%(param_name)s' dari service %(service)s"

41
sahara_plugin_storm/locale/ne/LC_MESSAGES/sahara_plugin_storm.po
View File

@ -1,41 +0,0 @@
# Surit Aryal <aryalsurit@gmail.com>, 2019. #zanata
msgid ""
msgstr ""
"Project-Id-Version: sahara-plugin-storm VERSION\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2019-07-23 14:26+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2019-08-02 04:50+0000\n"
"Last-Translator: Surit Aryal <aryalsurit@gmail.com>\n"
"Language-Team: Nepali\n"
"Language: ne\n"
"X-Generator: Zanata 4.3.3\n"
"Plural-Forms: nplurals=2; plural=(n != 1)\n"
msgid "1 or more"
msgstr "१ वा अधिक"
msgid "Push configs to nodes"
msgstr "कन्फिगरेसन नोडहरूमा पुश गर्नुहोस्"
msgid "Rebalance Topology"
msgstr "सन्तुलन टोपोलजी"
#, python-format
msgid "Storm plugin cannot scale nodegroup with processes: %s"
msgstr "Storm प्लगइनले प्रक्रियाहरूसँग नोड ग्रुप मापन गर्न सक्दैन: %s"
msgid "Storm {base} required to run {type} jobs"
msgstr "Storm {base} required to run {type} jobs"
msgid ""
"This plugin provides an ability to launch Storm cluster without any "
"management consoles."
msgstr ""
"यस प्लगइनले कुनै व्यवस्थापन कन्सोल बिना Storm क्लस्टर सुरुवात गर्न क्षमता प्रदान गर्दछ।"
#, python-format
msgid "Unable to get parameter '%(param_name)s' from service %(service)s"
msgstr "सेवा %(service)s बाट प्यारामिटर '%(param_name)s' प्राप्त गर्न असमर्थ"

0
sahara_plugin_storm/plugins/__init__.py
View File

0
sahara_plugin_storm/plugins/storm/__init__.py
View File

157
sahara_plugin_storm/plugins/storm/config_helper.py
View File

@ -1,157 +0,0 @@
# Copyright (c) 2014 Hoang Do, Phuc Vo, P. Michiardi, D. Venzano
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from oslo_config import cfg
from sahara.plugins import exceptions as ex
from sahara_plugin_storm.i18n import _
CONF = cfg.CONF
def get_config_value(service, name, cluster=None):
if cluster:
for ng in cluster.node_groups:
if (ng.configuration().get(service) and
ng.configuration()[service].get(name)):
return ng.configuration()[service][name]
raise ex.ConfigurationError(
_("Unable to get parameter '%(param_name)s' from "
"service %(service)s") % {'param_name': name, 'service': service})
def get_plugin_configs():
return {}
def generate_storm_config(master_hostname, zk_hostnames, version):
host_cfg = 'nimbus.seeds'
master_value = [master_hostname.encode('ascii', 'ignore')]
cfg = {
host_cfg: master_value,
"worker.childopts": "-Xmx768m -Djava.net.preferIPv4Stack=true",
"nimbus.childopts": "-Xmx1024m -Djava.net.preferIPv4Stack=true",
"supervisor.childopts": "-Djava.net.preferIPv4Stack=true",
"storm.zookeeper.servers": [i.encode('ascii', 'ignore')
for i in zk_hostnames],
"ui.childopts": "-Xmx768m -Djava.net.preferIPv4Stack=true",
"storm.local.dir": "/app/storm"
}
# Since pyleus is built using previous versions of Storm we need this
# option to allow the cluster to be compatible with pyleus topologies as
# well as with topologies built using older versions of Storm
cfg['client.jartransformer.class'] = (
"org.apache.storm.hack.StormShadeTransformer")
return cfg
def generate_pyleus_config():
separator = "\n"
conf = ("[storm]",
"storm_cmd_path: /usr/local/storm/bin/storm")
return separator.join(conf)
def generate_slave_supervisor_conf():
separator = "\n"
conf = ("[program:storm-supervisor]",
'command=bash -exec "cd /usr/local/storm && bin/storm supervisor"',
"user=storm",
"autostart=true",
"autorestart=true",
"startsecs=10",
"startretries=999",
"log_stdout=true",
"log_stderr=true",
"logfile=/var/log/storm/supervisor.out",
"logfile_maxbytes=20MB",
"logfile_backups=10")
return separator.join(conf)
def generate_master_supervisor_conf():
separator = "\n"
seq_n = ("[program:storm-nimbus]",
"command=/usr/local/storm/bin/storm nimbus",
"user=storm",
"autostart=true",
"autorestart=true",
"startsecs=10",
"startretries=999",
"log_stdout=true",
"log_stderr=true",
"logfile=/var/log/storm/supervisor.out",
"logfile_maxbytes=20MB",
"logfile_backups=10")
seq_u = ("[program:storm-ui]",
"command=/usr/local/storm/bin/storm ui",
"user=storm",
"autostart=true",
"autorestart=true",
"startsecs=10",
"startretries=999",
"log_stdout=true",
"log_stderr=true",
"logfile=/var/log/storm/ui.out",
"logfile_maxbytes=20MB",
"logfile_backups=10")
conf_n = separator.join(seq_n)
conf_u = separator.join(seq_u)
conf = (conf_n, conf_u)
return separator.join(conf)
def generate_zookeeper_conf():
separator = "\n"
conf = ("tickTime=2000",
"dataDir=/var/zookeeper",
"clientPort=2181")
return separator.join(conf)
def generate_storm_setup_script(env_configs):
separator = "\n"
script_lines = ["#!/bin/bash -x"]
script_lines.append("echo -n > /usr/local/storm/conf/storm.yaml")
for line in env_configs:
script_lines.append('echo "%s" >> /usr/local/storm/conf/storm.yaml'
% line)
return separator.join(script_lines)
def extract_name_values(configs):
return {cfg['name']: cfg['value'] for cfg in configs}
def _set_config(cfg, gen_cfg, name=None):
if name in gen_cfg:
cfg.update(gen_cfg[name]['conf'])
if name is None:
for name in gen_cfg:
cfg.update(gen_cfg[name]['conf'])
return cfg

52
sahara_plugin_storm/plugins/storm/edp_engine.py
View File

@ -1,52 +0,0 @@
# Copyright (c) 2015 Telles Nobrega.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from sahara.plugins import edp
from sahara.plugins import exceptions as ex
from sahara_plugin_storm.i18n import _
class EdpStormEngine(edp.PluginsStormJobEngine):
edp_base_version = "1.0.1"
@staticmethod
def edp_supported(version):
return version >= EdpStormEngine.edp_base_version
def validate_job_execution(self, cluster, job, data):
if not self.edp_supported(cluster.hadoop_version):
raise ex.PluginInvalidDataException(
_('Storm {base} required to run {type} jobs').format(
base=EdpStormEngine.edp_base_version, type=job.type))
super(EdpStormEngine, self).validate_job_execution(cluster, job, data)
class EdpPyleusEngine(edp.PluginsStormPyleusJobEngine):
edp_base_version = "1.0.1"
@staticmethod
def edp_supported(version):
return version >= EdpPyleusEngine.edp_base_version
def validate_job_execution(self, cluster, job, data):
if not self.edp_supported(cluster.hadoop_version):
raise ex.PluginInvalidDataException(
_('Storm {base} required to run {type} jobs').format(
base=EdpPyleusEngine.edp_base_version, type=job.type))
super(EdpPyleusEngine, self).validate_job_execution(cluster, job, data)

406
sahara_plugin_storm/plugins/storm/plugin.py
View File

@ -1,406 +0,0 @@
# Copyright (c) 2014 Hoang Do, Phuc Vo, P. Michiardi, D. Venzano
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import copy
from oslo_log import log as logging
import six
import yaml
from sahara.plugins import conductor
from sahara.plugins import context
from sahara.plugins import exceptions as ex
from sahara.plugins import provisioning as p
from sahara.plugins import utils
from sahara_plugin_storm.i18n import _
from sahara_plugin_storm.plugins.storm import config_helper as c_helper
from sahara_plugin_storm.plugins.storm import edp_engine
from sahara_plugin_storm.plugins.storm import run_scripts as run
LOG = logging.getLogger(__name__)
class StormProvider(p.ProvisioningPluginBase):
def __init__(self):
self.processes = {
"Zookeeper": ["zookeeper"],
"Storm": ["nimbus", "supervisor"]
}
def get_title(self):
return "Apache Storm"
def get_description(self):
return (
_("This plugin provides an ability to launch Storm "
"cluster without any management consoles."))
def get_labels(self):
default = {'enabled': {'status': True}, 'stable': {'status': True}}
deprecated = {'enabled': {'status': True},
'deprecated': {'status': True}}
result = {'plugin_labels': copy.deepcopy(default)}
result['version_labels'] = {
'1.2': copy.deepcopy(default),
'1.1.0': copy.deepcopy(default),
'1.0.1': copy.deepcopy(deprecated),
}
return result
def get_versions(self):
return ['1.0.1', '1.1.0', '1.2']
def get_configs(self, storm_version):
return c_helper.get_plugin_configs()
def get_node_processes(self, storm_version):
return self.processes
def validate(self, cluster):
# validate Storm Master Node and Storm Slaves
sm_count = sum([ng.count for ng
in utils.get_node_groups(cluster, "nimbus")])
if sm_count < 1:
raise ex.RequiredServiceMissingException("Storm nimbus")
if sm_count >= 2:
raise ex.InvalidComponentCountException("Storm nimbus", "1",
sm_count)
sl_count = sum([ng.count for ng
in utils.get_node_groups(cluster, "supervisor")])
if sl_count < 1:
raise ex.InvalidComponentCountException("Storm supervisor",
_("1 or more"),
sl_count)
def update_infra(self, cluster):
pass
def configure_cluster(self, cluster):
self._setup_instances(cluster)
def start_cluster(self, cluster):
sm_instance = utils.get_instance(cluster, "nimbus")
sl_instances = utils.get_instances(cluster, "supervisor")
zk_instances = utils.get_instances(cluster, "zookeeper")
# start zookeeper processes
self._start_zookeeper_processes(zk_instances)
# start storm master
if sm_instance:
self._start_storm_master(sm_instance)
# start storm slaves
self._start_slave_processes(sl_instances)
LOG.info("Cluster {cluster} has been started successfully".format(
cluster=cluster.name))
self._set_cluster_info(cluster)
def get_edp_engine(self, cluster, job_type):
if job_type in edp_engine.EdpStormEngine.get_supported_job_types():
return edp_engine.EdpStormEngine(cluster)
if job_type in edp_engine.EdpPyleusEngine.get_supported_job_types():
return edp_engine.EdpPyleusEngine(cluster)
return None
def get_edp_job_types(self, versions=None):
res = {}
for vers in self.get_versions():
if not versions or vers in versions:
storm_engine = edp_engine.EdpStormEngine
pyleus_engine = edp_engine.EdpPyleusEngine
res[vers] = (storm_engine.get_supported_job_types() +
pyleus_engine.get_supported_job_types())
return res
def get_edp_config_hints(self, job_type, version):
if edp_engine.EdpStormEngine.edp_supported(version):
return edp_engine.EdpStormEngine.get_possible_job_config(job_type)
if edp_engine.EdpPyleusEngine.edp_supported(version):
return edp_engine.EdpPyleusEngine.get_possible_job_config(job_type)
return {}
def get_open_ports(self, node_group):
ports_map = {
'nimbus': [8080]
}
ports = []
for process in node_group.node_processes:
if process in ports_map:
ports.extend(ports_map[process])
return ports
def _extract_configs_to_extra(self, cluster):
st_master = utils.get_instance(cluster, "nimbus")
zk_servers = utils.get_instances(cluster, "zookeeper")
extra = dict()
config_instances = ''
if st_master is not None:
if zk_servers is not None:
zknames = []
for zk in zk_servers:
zknames.append(zk.hostname())
config_instances = c_helper.generate_storm_config(
st_master.hostname(),
zknames,
cluster.hadoop_version)
config = self._convert_dict_to_yaml(config_instances)
supervisor_conf = c_helper.generate_slave_supervisor_conf()
nimbus_ui_conf = c_helper.generate_master_supervisor_conf()
zk_conf = c_helper.generate_zookeeper_conf()
pyleus_conf = c_helper.generate_pyleus_config()
for ng in cluster.node_groups:
extra[ng.id] = {
'st_instances': config,
'slave_sv_conf': supervisor_conf,
'master_sv_conf': nimbus_ui_conf,
'zk_conf': zk_conf,
'pyleus_conf': pyleus_conf
}
return extra
@utils.event_wrapper(
True, step=utils.start_process_event_message("StormMaster"))
def _start_storm_master(self, sm_instance):
with utils.get_remote(sm_instance) as r:
run.start_storm_nimbus_and_ui(r)
LOG.info("Storm master at {host} has been started".format(
host=sm_instance.hostname()))
def _start_slave_processes(self, sl_instances):
if len(sl_instances) == 0:
return
utils.add_provisioning_step(
sl_instances[0].cluster_id,
utils.start_process_event_message("Slave"), len(sl_instances))
with context.PluginsThreadGroup() as tg:
for i in sl_instances:
tg.spawn('storm-start-sl-%s' % i.instance_name,
self._start_slaves, i)
@utils.event_wrapper(True)
def _start_slaves(self, instance):
with instance.remote() as r:
run.start_storm_supervisor(r)
def _start_zookeeper_processes(self, zk_instances):
if len(zk_instances) == 0:
return
utils.add_provisioning_step(
zk_instances[0].cluster_id,
utils.start_process_event_message("Zookeeper"), len(zk_instances))
with context.PluginsThreadGroup() as tg:
for i in zk_instances:
tg.spawn('storm-start-zk-%s' % i.instance_name,
self._start_zookeeper, i)
@utils.event_wrapper(True)
def _start_zookeeper(self, instance):
with instance.remote() as r:
run.start_zookeeper(r)
def _setup_instances(self, cluster, instances=None):
extra = self._extract_configs_to_extra(cluster)
if instances is None:
instances = utils.get_instances(cluster)
self._push_configs_to_nodes(cluster, extra, instances)
def _push_configs_to_nodes(self, cluster, extra, new_instances):
all_instances = utils.get_instances(cluster)
utils.add_provisioning_step(
cluster.id, _("Push configs to nodes"), len(all_instances))
with context.PluginsThreadGroup() as tg:
for instance in all_instances:
if instance in new_instances:
tg.spawn('storm-configure-%s' % instance.instance_name,
self._push_configs_to_new_node, cluster,
extra, instance)
else:
tg.spawn('storm-reconfigure-%s' % instance.instance_name,
self._push_configs_to_existing_node, cluster,
extra, instance)
def _convert_dict_to_yaml(self, dict_to_convert):
new_dict = dict_to_convert.copy()
for key in dict_to_convert:
if isinstance(dict_to_convert[key], six.string_types):
new_dict[key] = "\"" + dict_to_convert[key] + "\""
stream = yaml.dump(new_dict, default_flow_style=False)
stream = stream.replace("\'", "")
return stream
@utils.event_wrapper(True)
def _push_configs_to_new_node(self, cluster, extra, instance):
ng_extra = extra[instance.node_group.id]
files_supervisor = {
'/etc/supervisor/supervisord.conf': ng_extra['slave_sv_conf']
}
files_storm = {
'/usr/local/storm/conf/storm.yaml': ng_extra['st_instances']
}
files_zk = {
'/opt/zookeeper/zookeeper/conf/zoo.cfg': ng_extra['zk_conf']
}
files_supervisor_master = {
'/etc/supervisor/supervisord.conf': ng_extra['master_sv_conf']
}
file_pyleus_conf = {
'/home/ubuntu/.pyleus.conf': ng_extra['pyleus_conf']
}
with utils.get_remote(instance) as r:
node_processes = instance.node_group.node_processes
r.write_files_to(files_storm, run_as_root=True)
if 'zookeeper' in node_processes:
self._push_zk_configs(r, files_zk)
if 'nimbus' in node_processes:
self._push_supervisor_configs(r, files_supervisor_master)
self._push_supervisor_configs(r, file_pyleus_conf)
if 'supervisor' in node_processes:
self._push_supervisor_configs(r, files_supervisor)
@utils.event_wrapper(True)
def _push_configs_to_existing_node(self, cluster, extra, instance):
node_processes = instance.node_group.node_processes
need_storm_update = ('nimbus' in node_processes or
'supervisor' in node_processes)
need_zookeeper_update = 'zookeeper' in node_processes
ng_extra = extra[instance.node_group.id]
r = utils.get_remote(instance)
if need_storm_update:
storm_path = '/usr/local/storm/conf/storm.yaml'
files_storm = {storm_path: ng_extra['st_instances']}
r.write_files_to(files_storm)
if need_zookeeper_update:
zk_path = '/opt/zookeeper/zookeeper/conf/zoo.cfg'
files_zookeeper = {zk_path: ng_extra['zk_conf']}
self._push_zk_configs(r, files_zookeeper)
def _set_cluster_info(self, cluster):
st_master = utils.get_instance(cluster, "nimbus")
info = {}
if st_master:
port = "8080"
info['Storm'] = {
'Web UI': 'http://%s:%s' % (
st_master.get_ip_or_dns_name(), port)
}
ctx = context.ctx()
conductor.cluster_update(ctx, cluster, {'info': info})
def _push_zk_configs(self, r, files):
r.write_files_to(files, run_as_root=True)
def _push_supervisor_configs(self, r, files):
r.append_to_files(files, run_as_root=True)
# Scaling
def _get_running_topologies_names(self, cluster):
master = utils.get_instance(cluster, "nimbus")
cmd = (
"%(storm)s -c nimbus.host=%(host)s "
"list | grep ACTIVE | awk '{print $1}'") % (
{
"storm": "/usr/local/storm/bin/storm",
"host": master.hostname()
})
with utils.get_remote(master) as r:
ret, stdout = r.execute_command(cmd)
names = stdout.split('\n')
topology_names = names[0:len(names)-1]
return topology_names
@utils.event_wrapper(True, step=_("Rebalance Topology"),
param=('cluster', 1))
def rebalance_topology(self, cluster):
topology_names = self._get_running_topologies_names(cluster)
master = utils.get_instance(cluster, "nimbus")
for topology_name in topology_names:
cmd = (
'%(rebalance)s -c nimbus.host=%(host)s %(topology_name)s') % (
{
"rebalance": "/usr/local/storm/bin/storm rebalance",
"host": master.hostname(),
"topology_name": topology_name
})
with utils.get_remote(master) as r:
ret, stdout = r.execute_command(cmd)
def validate_scaling(self, cluster, existing, additional):
self._validate_existing_ng_scaling(cluster, existing)
self._validate_additional_ng_scaling(cluster, additional)
def scale_cluster(self, cluster, instances):
self._setup_instances(cluster, instances)
# start storm slaves
self._start_slave_processes(instances)
self.rebalance_topology(cluster)
LOG.info("Storm scaling has been started.")
def _get_scalable_processes(self):
return ["supervisor"]
def _validate_additional_ng_scaling(self, cluster, additional):
scalable_processes = self._get_scalable_processes()
for ng_id in additional:
ng = utils.get_by_id(cluster.node_groups, ng_id)
if not set(ng.node_processes).issubset(scalable_processes):
raise ex.NodeGroupCannotBeScaled(
ng.name, _("Storm plugin cannot scale nodegroup"
" with processes: %s") %
' '.join(ng.node_processes))
def _validate_existing_ng_scaling(self, cluster, existing):
scalable_processes = self._get_scalable_processes()
for ng in cluster.node_groups:
if ng.id in existing:
if not set(ng.node_processes).issubset(scalable_processes):
raise ex.NodeGroupCannotBeScaled(
ng.name, _("Storm plugin cannot scale nodegroup"
" with processes: %s") %
' '.join(ng.node_processes))

54
sahara_plugin_storm/plugins/storm/run_scripts.py
View File

@ -1,54 +0,0 @@
# Copyright (c) 2014 Hoang Do, Phuc Vo, P. Michiardi, D. Venzano
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
def start_zookeeper(remote):
remote.execute_command("sudo %s %s" % (
"/opt/zookeeper/zookeeper/bin/zkServer.sh",
"start"))
def start_storm_supervisor(node):
_create_supervisor_log_file(node)
_stop_supervisor_deamon(node)
_start_supervisor_deamon(node)
def start_storm_nimbus_and_ui(node):
_create_supervisor_log_file(node)
_stop_supervisor_deamon(node)
_start_supervisor_deamon(node)
def stop_storm_nimbus_and_ui(node):
_stop_supervisor_deamon(node)
def stop_storm_supervisor(node):
_stop_supervisor_deamon(node)
def _start_supervisor_deamon(node):
node.execute_command("sudo service supervisor start")
def _stop_supervisor_deamon(node):
node.execute_command("sudo service supervisor stop")
def _create_supervisor_log_file(node):
node.execute_command("sudo mkdir -p /var/log/storm")
node.execute_command("sudo chmod -R 777 /var/log/storm")
node.execute_command("sudo chown -R storm:storm /var/log/storm")

17
sahara_plugin_storm/tests/__init__.py
View File

@ -1,17 +0,0 @@
# Copyright (c) 2014 Mirantis Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from sahara_plugin_storm.utils import patches
patches.patch_all()

0
sahara_plugin_storm/tests/unit/__init__.py
View File

74
sahara_plugin_storm/tests/unit/base.py
View File

@ -1,74 +0,0 @@
# Copyright (c) 2013 Mirantis Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from unittest import mock
from oslotest import base
from sahara import context
from sahara.db import api as db_api
from sahara import main
from sahara.utils import rpc
class SaharaTestCase(base.BaseTestCase):
def setUp(self):
super(SaharaTestCase, self).setUp()
self.setup_context()
rpc.setup('all-in-one')
def setup_context(self, username="test_user", tenant_id="tenant_1",
auth_token="test_auth_token", tenant_name='test_tenant',
service_catalog=None, **kwargs):
self.addCleanup(context.set_ctx,
context.ctx() if context.has_ctx() else None)
context.set_ctx(context.Context(
username=username, tenant_id=tenant_id,
auth_token=auth_token, service_catalog=service_catalog or {},
tenant_name=tenant_name, **kwargs))
def override_config(self, name, override, group=None):
main.CONF.set_override(name, override, group)
self.addCleanup(main.CONF.clear_override, name, group)
class SaharaWithDbTestCase(SaharaTestCase):
def setUp(self):
super(SaharaWithDbTestCase, self).setUp()
self.override_config('connection', "sqlite://", group='database')
db_api.setup_db()
self.addCleanup(db_api.drop_db)
class _ConsecutiveThreadGroup(context.ThreadGroup):
def __init__(self, _thread_pool_size=1000):
pass
def spawn(self, thread_description, func, *args, **kwargs):
func(*args, **kwargs)
def __enter__(self):
return self
def __exit__(self, *ex):
pass
def mock_thread_group(func):
return mock.patch('sahara.context.ThreadGroup',
new=_ConsecutiveThreadGroup)(func)

0
sahara_plugin_storm/tests/unit/plugins/__init__.py
View File

0
sahara_plugin_storm/tests/unit/plugins/storm/__init__.py
View File

55
sahara_plugin_storm/tests/unit/plugins/storm/test_config_helper.py
View File

@ -1,55 +0,0 @@
# Copyright 2017 Massachusetts Open Cloud
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from testtools import testcase
from sahara_plugin_storm.plugins.storm import config_helper as s_config
from sahara_plugin_storm.plugins.storm import plugin as s_plugin
class TestStormConfigHelper(testcase.TestCase):
def test_generate_storm_config(self):
STORM_101 = '1.0.1'
STORM_110 = '1.1.0'
STORM_120 = '1.2'
tested_versions = []
master_hostname = "s-master"
zk_hostnames = ["s-zoo"]
configs_101 = s_config.generate_storm_config(
master_hostname, zk_hostnames, STORM_101)
self.assertNotIn('nimbus.host', configs_101.keys())
self.assertIn('nimbus.seeds', configs_101.keys())
self.assertIn('client.jartransformer.class', configs_101.keys())
self.assertEqual(configs_101['client.jartransformer.class'],
'org.apache.storm.hack.StormShadeTransformer')
tested_versions.append(STORM_101)
configs_110 = s_config.generate_storm_config(
master_hostname, zk_hostnames, STORM_110)
self.assertNotIn('nimbus.host', configs_110.keys())
self.assertIn('nimbus.seeds', configs_110.keys())
self.assertIn('client.jartransformer.class', configs_110.keys())
self.assertEqual(configs_110['client.jartransformer.class'],
'org.apache.storm.hack.StormShadeTransformer')
tested_versions.append(STORM_110)
configs_120 = s_config.generate_storm_config(
master_hostname, zk_hostnames, STORM_120)
self.assertNotIn('nimbus.host', configs_120.keys())
self.assertIn('nimbus.seeds', configs_120.keys())
self.assertIn('client.jartransformer.class', configs_120.keys())
self.assertEqual(configs_120['client.jartransformer.class'],
'org.apache.storm.hack.StormShadeTransformer')
tested_versions.append(STORM_120)
storm = s_plugin.StormProvider()
self.assertEqual(storm.get_versions(), tested_versions)

56
sahara_plugin_storm/tests/unit/plugins/storm/test_plugin.py
View File

@ -1,56 +0,0 @@
# Copyright (c) 2015 TellesNobrega
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from unittest import mock
from sahara.plugins import base as pb
from sahara_plugin_storm.plugins.storm import plugin as pl
from sahara_plugin_storm.tests.unit import base
class StormPluginTest(base.SaharaWithDbTestCase):
def setUp(self):
super(StormPluginTest, self).setUp()
self.override_config("plugins", ["storm"])
self.master_host = "master"
self.master_inst = "6789"
self.storm_topology_name = 'topology1'
pb.setup_plugins()
def _make_master_instance(self, return_code=0):
master = mock.Mock()
master.execute_command.return_value = (return_code,
self.storm_topology_name)
master.hostname.return_value = self.master_host
master.id = self.master_inst
return master
def _get_cluster(self, name, version):
cluster_dict = {
'name': name,
'plugin_name': 'storm',
'hadoop_version': version,
'node_groups': []}
return cluster_dict
def test_get_open_port(self):
plugin_storm = pl.StormProvider()
cluster = mock.Mock()
ng = mock.Mock()
ng.node_processes = ['nimbus']
cluster.node_groups = [ng]
ng.cluster = cluster
ports = plugin_storm.get_open_ports(ng)
self.assertEqual([8080], ports)

0
sahara_plugin_storm/utils/__init__.py
View File

108
sahara_plugin_storm/utils/patches.py
View File

@ -1,108 +0,0 @@
# Copyright (c) 2013 Mirantis Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import eventlet
EVENTLET_MONKEY_PATCH_MODULES = dict(os=True,
select=True,
socket=True,
thread=True,
time=True)
def patch_all():
"""Apply all patches.
List of patches:
* eventlet's monkey patch for all cases;
* minidom's writexml patch for py < 2.7.3 only.
"""
eventlet_monkey_patch()
patch_minidom_writexml()
def eventlet_monkey_patch():
"""Apply eventlet's monkey patch.
This call should be the first call in application. It's safe to call
monkey_patch multiple times.
"""
eventlet.monkey_patch(**EVENTLET_MONKEY_PATCH_MODULES)
def eventlet_import_monkey_patched(module):
"""Returns module monkey patched by eventlet.
It's needed for some tests, for example, context test.
"""
return eventlet.import_patched(module, **EVENTLET_MONKEY_PATCH_MODULES)
def patch_minidom_writexml():
"""Patch for xml.dom.minidom toprettyxml bug with whitespaces around text
We apply the patch to avoid excess whitespaces in generated xml
configuration files that brakes Hadoop.
(This patch will be applied for all Python versions < 2.7.3)
Issue: http://bugs.python.org/issue4147
Patch: http://hg.python.org/cpython/rev/cb6614e3438b/
Description: http://ronrothman.com/public/leftbraned/xml-dom-minidom-\
toprettyxml-and-silly-whitespace/#best-solution
"""
import sys
if sys.version_info >= (2, 7, 3):
return
import xml.dom.minidom as md
def element_writexml(self, writer, indent="", addindent="", newl=""):
# indent = current indentation
# addindent = indentation to add to higher levels
# newl = newline string
writer.write(indent + "<" + self.tagName)
attrs = self._get_attributes()
a_names = list(attrs.keys())
a_names.sort()
for a_name in a_names:
writer.write(" %s=\"" % a_name)
md._write_data(writer, attrs[a_name].value)
writer.write("\"")
if self.childNodes:
writer.write(">")
if (len(self.childNodes) == 1
and self.childNodes[0].nodeType == md.Node.TEXT_NODE):
self.childNodes[0].writexml(writer, '', '', '')
else:
writer.write(newl)
for node in self.childNodes:
node.writexml(writer, indent + addindent, addindent, newl)
writer.write(indent)
writer.write("</%s>%s" % (self.tagName, newl))
else:
writer.write("/>%s" % (newl))
md.Element.writexml = element_writexml
def text_writexml(self, writer, indent="", addindent="", newl=""):
md._write_data(writer, "%s%s%s" % (indent, self.data, newl))
md.Text.writexml = text_writexml

43
setup.cfg
View File

@ -1,43 +0,0 @@
[metadata]
name = sahara-plugin-storm
summary = Storm Plugin for Sahara Project
description-file = README.rst
license = Apache Software License
python-requires = >=3.8
classifiers =
Programming Language :: Python
Programming Language :: Python :: Implementation :: CPython
Programming Language :: Python :: 3 :: Only
Programming Language :: Python :: 3
Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9
Environment :: OpenStack
Intended Audience :: Information Technology
Intended Audience :: System Administrators
License :: OSI Approved :: Apache Software License
Operating System :: POSIX :: Linux
author = OpenStack
author-email = openstack-discuss@lists.openstack.org
home-page = https://docs.openstack.org/sahara/latest/
[files]
packages =
sahara_plugin_storm
[entry_points]
sahara.cluster.plugins =
storm = sahara_plugin_storm.plugins.storm.plugin:StormProvider
[compile_catalog]
directory = sahara_plugin_storm/locale
domain = sahara_plugin_storm
[update_catalog]
domain = sahara_plugin_storm
output_dir = sahara_plugin_storm/locale
input_file = sahara_plugin_storm/locale/sahara_plugin_storm.pot
[extract_messages]
keywords = _ gettext ngettext l_ lazy_gettext
mapping_file = babel.cfg
output_file = sahara_plugin_storm/locale/sahara_plugin_storm.pot

20
setup.py
View File

@ -1,20 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import setuptools
setuptools.setup(
setup_requires=['pbr>=2.0.0'],
pbr=True)

16
test-requirements.txt
View File

@ -1,16 +0,0 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
hacking>=3.0.1,<3.1.0 # Apache-2.0
bandit>=1.1.0 # Apache-2.0
bashate>=0.5.1 # Apache-2.0
coverage!=4.4,>=4.0 # Apache-2.0
doc8>=0.6.0 # Apache-2.0
fixtures>=3.0.0 # Apache-2.0/BSD
oslotest>=3.2.0 # Apache-2.0
stestr>=1.0.0 # Apache-2.0
pylint==1.4.5 # GPLv2
testscenarios>=0.4 # Apache-2.0/BSD
testtools>=2.4.0 # MIT

99
tox.ini
View File

@ -1,99 +0,0 @@
[tox]
envlist = py38,pep8
minversion = 3.1.1
skipsdist = True
# this allows tox to infer the base python from the environment name
# and override any basepython configured in this file
ignore_basepython_conflict = true
[testenv]
basepython = python3
usedevelop = True
install_command = pip install {opts} {packages}
setenv =
VIRTUAL_ENV={envdir}
DISCOVER_DIRECTORY=sahara_plugin_storm/tests/unit
deps =
-c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
-r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands = stestr run {posargs}
passenv =
http_proxy
https_proxy
no_proxy
[testenv:debug-py36]
basepython = python3.6
commands = oslo_debug_helper -t sahara_plugin_storm/tests/unit {posargs}
[testenv:debug-py37]
basepython = python3.7
commands = oslo_debug_helper -t sahara_plugin_storm/tests/unit {posargs}
[testenv:pep8]
deps =
-c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
-r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
-r{toxinidir}/doc/requirements.txt
commands =
flake8 {posargs}
doc8 doc/source
[testenv:venv]
commands = {posargs}
[testenv:docs]
deps =
-c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
-r{toxinidir}/doc/requirements.txt
commands =
rm -rf doc/build/html
sphinx-build -W -b html doc/source doc/build/html
allowlist_externals =
rm
[testenv:pdf-docs]
deps = {[testenv:docs]deps}
commands =
rm -rf doc/build/pdf
sphinx-build -W -b latex doc/source doc/build/pdf
make -C doc/build/pdf
allowlist_externals =
make
rm
[testenv:releasenotes]
deps =
-c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
-r{toxinidir}/doc/requirements.txt
commands =
rm -rf releasenotes/build releasenotes/html
sphinx-build -a -E -W -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html
allowlist_externals = rm
[testenv:debug]
# It runs tests from the specified dir (default is sahara_plugin_storm/tests)
# in interactive mode, so, you could use pbr for tests debug.
# Example usage: tox -e debug -- -t sahara_plugin_storm/tests/unit some.test.path
# https://docs.openstack.org/oslotest/latest/features.html#debugging-with-oslo-debug-helper
commands = oslo_debug_helper -t sahara_plugin_storm/tests/unit {posargs}
[flake8]
show-source = true
builtins = _
exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,tools
# [H904] Delay string interpolations at logging calls
# [H106] Don't put vim configuration in source files
# [H203] Use assertIs(Not)None to check for None.
# [H204] Use assert(Not)Equal to check for equality
# [H205] Use assert(Greater|Less)(Equal) for comparison
enable-extensions=H904,H106,H203,H204,H205
# [E123] Closing bracket does not match indentation of opening bracket's line
# [E226] Missing whitespace around arithmetic operator
# [E402] Module level import not at top of file
# [E731] Do not assign a lambda expression, use a def
# [W503] Line break occurred before a binary operator
# [W504] line break after binary operator
ignore=E123,E226,E402,E731,W503,W504