monasca-transform/monasca_transform/component
Ashwin Agate 022bd11a4d Switch to using Spark version 2.2.0
Following changes were required:

1.)
By default the pre-built distribution
for Spark 2.2.0 is compiled with Scala 2.11.
monasca-transform requires Spark compiled with
Scala 2.10 since we use spark streaming to
pull data from Kafka and the version of Kafka
is compatible with Scala 2.10.
The recommended way is to compile Spark
with Scala 2.10, but for purposes of devstack
plugin made changes to pull the required jars
from mvn directly.
(see SPARK_JARS and SPARK_JAVA_LIB variables in
settings)
All jars get moved to
<SPARK_HOME>/assembly/target/assembly/
target/scala_2.10/jars/
Note: <SPARK_HOME>/jars gets renamed
to <SPARK_HOME>/jars_original.
spark-submit defaults to assembly location
if <SPARK_HOME>/jars directory is missing.

2.) Updated start up scripts for spark
worker and spark master with a new env variable
SPARK_SCALA_VERSIOn=2.10. Also updated
PYTHONPATH variable to add new
py4j-0.10.4-src.zip file

3.) Some changes to adhere to deprecated pyspark
function calls which were removed in Spark 2.0

Change-Id: I8f8393bb91307d55f156b2ebf45225a16ae9d8f4
2017-08-21 11:18:22 -07:00
..
insert Started adding kubernetes metrics aggregation 2017-03-02 15:38:25 +00:00
setter Started adding kubernetes metrics aggregation 2017-03-02 15:38:25 +00:00
usage Switch to using Spark version 2.2.0 2017-08-21 11:18:22 -07:00
__init__.py monasca-transform initial commit 2016-05-26 00:10:37 +00:00
component_utils.py monasca-transform initial commit 2016-05-26 00:10:37 +00:00