openstack/deb-sahara
Code Issues Proposed changes
64583e4b83
deb-sahara/sahara/plugins/spark/resources
History
Ethan Gafford 7cf61bd313 Spark Temporary Job Data Retention and Cleanup 
Introduces a periodic task for the cleanup of data from Spark jobs, in order
to ease maintenance of long-lived clusters.

Change-Id: Ia7dc2dde54ab62199a630c3d1b64c76f08698181
Implements: blueprint spark-cleanup
2015-01-27 13:14:33 -05:00
..
core-default.xml Add Spark plugin to Sahara 2014-05-27 19:34:39 +00:00
hdfs-default.xml Add Spark plugin to Sahara 2014-05-27 19:34:39 +00:00
README.rst Add Spark plugin to Sahara 2014-05-27 19:34:39 +00:00
spark-cleanup.cron Spark Temporary Job Data Retention and Cleanup 2015-01-27 13:14:33 -05:00
spark-env.sh.template Add Spark plugin to Sahara 2014-05-27 19:34:39 +00:00
tmp-cleanup.sh.template Spark Temporary Job Data Retention and Cleanup 2015-01-27 13:14:33 -05:00
topology.sh Fix bashate errors 2014-10-08 18:21:49 +04:00

README.rst

Apache Spark and HDFS Configurations for Sahara

This directory contains default XML configuration files and Spark scripts:

  • core-default.xml,
  • hdfs-default.xml,
  • spark-env.sh.template,
  • topology.sh

These files are used by Sahara's plugin for Apache Spark and Cloudera HDFS. XML config files were taken from here: * https://github.com/apache/hadoop-common/blob/release-1.2.1/src/core/core-default.xml * https://github.com/apache/hadoop-common/blob/release-1.2.1/src/hdfs/hdfs-default.xml

Cloudera packages use the same configuration files as standard Apache Hadoop.

XML configs are used to expose default Hadoop configurations to the users through Sahara's REST API. It allows users to override some config values which will be pushed to the provisioned VMs running Hadoop services as part of appropriate xml config.