sahara-extra/edp-adapt-for-spark
Trevor McKay d4b2f376e4 Add a SparkWrapper class
This class takes the name of an XML file to add to the default Hadoop
configuration for a Spark job before invoking the job's main class.

This class is based on the MainWrapper used for Oozie Java actions
but is simpler. There is no need to modify the SecurityManager, and
the name of the file may be anything. This class is broken out
separately so that the wrappers can vary independently in
separate JARs.

Change-Id: I5f34fcf525f4f16b095d52fab5059b9be59896f7
Partial-Implements: blueprint edp-spark-swift-integration
2015-01-21 14:43:17 -05:00
..
src/main/java/org/openstack/sahara/edp Add a SparkWrapper class 2015-01-21 14:43:17 -05:00
README.rst Add a SparkWrapper class 2015-01-21 14:43:17 -05:00
pom.xml Add a SparkWrapper class 2015-01-21 14:43:17 -05:00

README.rst

Sources for main function wrapper for Spark

The Hadoop configuration for a Spark job must be modified if the Spark job is going to access Swift paths. Specifically, the Hadoop configuration must contain values that allow the job to authenticate to the Swift service.

This wrapper adds a specified xml file to the default Hadoop Configuration resource list and then calls the specified main class. Any necessary Hadoop configuration values can be added to the xml file. This allows the main class to be run and access Swift paths without alteration.