Trevor McKay bfe01ead79 Add Swift integration with Spark ...

This change allows Spark jobs to access Swift URLs without
any need to modify the Spark job code itself. There are a
number of things necessary to make this work:

* add a "edp.spark.adapt_for_swift" config value to control the
  feature
* generate a modified spark-submit command when the feature is
  enabled
* add the hadoop-swift.jar to the Spark classpaths for the
  driver and executors (cluster launch)
* include the general Swift configs in the Hadoop core-site-xml
  and make Spark read the Hadoop core-site.xml (cluster launch)
* upload an xml file containing the Swift authentication configs
  for Hadoop
* run a wrapper class that reads the extra Hadoop configuration
  and adds it to the configuration for the job

Changes in other CRs:
* add the hadoop-swift.jar to the Spark images
* add the SparkWrapper code to sahara-extra

Partial-Implements: blueprint edp-spark-swift-integration
Change-Id: I03dca4400c832f3ba8bc508d4fb2aa98dede8d80

2015-02-03 10:34:32 -05:00

3.4 KiB

Raw History

View Raw