This commit adds the functionality of sahara-status CLI for performing
upgrade checks as part of the Stein cycle upgrade-checkers goal.
It only includes a sample check which must be replaced by real checks in
future.
Change-Id: Idcb8d9eaf689800812cf6087e9c5937058c89ea6
Story: 2003657
Task: 26152
* Create S3 data source type for EDP
* Support storing S3 secret key in Castellan
* Unit tests for new data source type
* Document new data source type and related ideas
* Add support of S3 configs into Spark and Oozie workflows
* Hide S3 credentials in job execution info, like for Swift
* Release note
Change-Id: I3ae5b9879b54f81d34bc7cd6a6f754347ce82f33
* Create common module for managing S3 job binaries
* Add new dependency on botocore
* Use common S3 library to create S3 job binary type for EDP
* Use common S3 library to create S3 job binary retriever
* Support storing S3 secret key in Castellan
* Document new job binary type (and foreshadow the S3 data source type)
* Unit tests for new code
Change-Id: I6781203d802305446ba1418ed6999186db4dfe9b
Partially-Implements: bp sahara-support-s3
In order to make it simpler to use the default
configuration files when deploying services
from source, the files are added to pbr's
data_files section so that the files are
included in the built wheels and therefore
deployed with the code. Packaging and deployment
tools can then more easily use the default files
if they wish to.
This pattern is already established with similar
files for neutron, designate and glance as has
been mentioned in the related bug report.
Change-Id: Iae987bdae55b75143f192f9a642cf1ff564c62b6
Closes-Bug: #1718356
The gating on python 3.4 is restricted to <= Mitaka. This is due to
the change from Ubuntu Trusty to Xenial, where only python3.5 is
available. There is no need to continue to keep these settings.
Change-Id: Ifbe877705dbd87dea34d2b1974d2ae3ed7cac9a3
In patch [1], tempest-like test of sahara client has been removed.
But in setup.cfg, there still is entry point of it. Remove it!
[1]:https://review.openstack.org/#/c/367980/
Change-Id: I802e27b91486baa43286bc949e304448ad1610f1
Now that there is a passing gate job, we can claim support for
Python 3.5 in the classifier. This patch also adds the convenience
py35 venv.
Change-Id: I15a761c0b6d5769b2dba9f521ef96cf43c99e552
Introduces a new libfuestfs-driven CLI for packing of
Sahara images, using the same recipe definition scheme
introduced in the validate-image-spi blueprint.
Documentation and plugin image packing configurability
can and will be provided in separate patches for ease
of review.
Partially-implements: blueprint image-generation-cli
Change-Id: I6788108e3fb6232045fc56937639a6348768a7bc
Newton release is opened, so we can remove this
plugin from sahara codebase
Implements blueprint: remove-hdp206
Change-Id: I1ebf90e716964151c88de53d127789251bef066b
No config generator hooks should ever be registered with a name that
belongs to another project. In this case, using oslo.middleware.cors
means that *every other project* that loads the middleware gets this
application's defaults when the generator is run on a system with
everything installed (such as a dev box with devstack). Use the name
of the app instead, to ensure that the defaults are only set when this
app's sample config and documentation are being generated.
Change-Id: I6a8c7d44b9db9325003ff2fdb667b0ced7739e96
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
The default values needed for sahara's implementation of cors
middleware have been moved from paste.ini into the configuration
hooks provided by oslo.config. Furthermore, these values have been
added to the default initialization procedure. This ensures
that if a value remains unset in the configuration file, it will
fallback to using sane defaults. It also ensures that an operator
modifying the configuration will be presented with that same
set of defaults.
Change-Id: Iedcaa002ff3d40cf61168769bc3946f8c6e42b87
Closes-Bug: 1551836
This patch adds support of running sahara-api as wsgi application.
Tested with apache + mod_wsgi web server.
Change-Id: I7355b09ebedd44e51242a252e96ac3a51073b2e6
Per [1] we now want to use the git repository for knowing the version instead
of just trying to modify the setup.cfg file.
[1] http://lists.openstack.org/pipermail/openstack-dev/2015-November/080692.html
Change-Id: I790506ea15e67f0cdeaed62b5b87a7d51c589e7b
Depends-On: I001a18895489a56eb55a5eb8c73ab6a3d4b1d4d0
This change remove support of direct engine in Sahara.
Unit tests related to direct engine were removed.
Implements blueprint: remove-direct-engine
Change-Id: Iba292996dd052a069a98f1024441255a15c0f734
Bump preversion to mark the start of the Mitaka development branch.
The liberty release branch will be cut from the previous commit.
Change-Id: I2e08186efb1d4f0d280bce11f2c3ac710bfa0a5d
Use the Tempest plugin interface for python client tests (which
are already tempest-based) instead of copying the tests inside
the tempest tree.
Inspired by the same type of change applied to Manila,
see Ie5ed64a6777ed1acf8dd56522c26705ae897596d
Depends-On: I06f1e13207cc6d661c078c4c4cf1ae7974ecf3da
Change-Id: I6073a528413aefd96882179a1eb8bbe715c6417b
Current features:
* Set up Ambari server and Ambari agents from pre-built image
* Support Ambari 2.0 / 2.1
* Password generation for management console
partially implements bp: hdp-22-support
Change-Id: I53084a83cccde52654a42911f449cc8b7d769bea
Bump pre-version in setup.cfg to formally open Liberty development.
Kilo release branch will be cut from the previous commit.
Change-Id: I6fa1e9e45fbaae62f7feb64039565266cbc86788
The first official oslo.policy was released and so we're ready to
migrate from the copy-pasted oslo-incubator code to the separated
lib.
Change-Id: I8e404d2b622876389d9b1019ae50f1e1a8cecdf1
This change adds a CLI called sahara-templates to manage default
templates.
Partial-implements: blueprint default-templates
Change-Id: I4f30bd6bc378d90fda41b512c04afe8023d2b4b2
* Extracted heat template generation from
'utils/openstack/heat.py' to 'service/heat/templates.py'.
Now heat.py is for heat client only.
* Moved heat template resources from 'resources' to
'service/heat/resources'
* Separated tests for heat templates and heat client.
Change-Id: I4c7a561f1648f34e71574a556cbc07ed2cf9b173
Closes-Bug: #1373075
Log module was removed from oslo-incubator,
so let's sync with latest oslo-incubator and remove
log module.
Change-Id: Id27ade473ed8597f2741ed0c746fa6b7ef6c7f06
Related-bug: #1412673
It can be automatically generated from console_scripts entry in
setup.cfg. So we don't need maintain it for sahara.
Change-Id: Ibe2a170cc60eb69b44aec43fb61f367e53c47b9f
This change allows the stevedore package to find the Storm plugin when
enabling extensions.
Change-Id: I1748609caff74e073171d5e4a1b76fda9efef1d6
Closes-Bug: 1407120
1. Sahara opts are listed once
2. Oslo incubator opts are listed separately
3. Groups are referenced by variables
Still things to improve:
1. opt-group mapping is listed twice now (in reg and in list)
2. registering spread across the project (should be in one place)
3. dependancies are not managed (it should be one place with CONF)
Change-Id: I4b8edf26187b3c23d08f9efbee42086652efea5a
Closes-Bug: #1407834
Features:
* cluster provisioning
* scaling
* edp
* validation
* swift support
TODO (in other CRs):
* unit tests
* integration tests
* data locality
partially implement: blueprint cdh-plugin
Change-Id: Ie231c434d61ba9a379a6ee2fd0f0bf2af21ce44d
Integrate the Spark plugin in Sahara. It adds the capability of
provisioning Spark clusters with (Cloudera) HDFS. The plugin
assumes the use of VM images generated with the Savanna
diskimage-builder and the '-p spark' option.
Implements: blueprint spark-plugin
Wiki: https://wiki.openstack.org/wiki/Sahara/SparkPlugin
This code is running on our Bigfoot research cluster.
Change-Id: Ic105f7de64248bdfb05879ededf35503bc04925b
Split monolitic sahara-api into two services: sahara-api and
sahara-engine. The former is a user-facing interface, the later -
service doing all the work. Sahara-api sends tasks to sahara-engine
via oslo.messaging. See the blueprint for details.
Notes:
* Used the following Climate RPC code as a baseline:
https://github.com/stackforge/climate/blob/master/climate/utils/service.py
hence added Julien Danjou to licence header.
* Removed the old contents of sahara/utils/rpc.py - that is a prehistoric
stuff not used anywhere.
* sahara-api still depends on eventlet, while we want to drop this dependency
in the end.
* periodics run in sahara-api. They shold be moved to engine later.
Partially Implements: blueprint scalable-savanna
Change-Id: I64275a757b539f3fcddd6e993d6614d492745226
In the nearest future we are going to split current sahara-api process into
two: sahara-api and sahara-engine. The new process sahara-all will replace
the old sahara-api in that it will run standalone Sahara all-in-one, which
does not require running several different processes.
This patch set will help us make a painless transition from sahara-api
to sahara-all in our CI. The plan is:
* merge current patch set, which just adds sahara-all alias to
sahara-api
* change CI to run sahara-all instead of sahara-api
* merge patch set which splits sahara-api and sahara-engine
Change-Id: I4fa68364b756c803a3fa8a0b92813785636ea13c