openstack/sahara-image-elements
Code Issues Proposed changes
63af6d0852
sahara-image-elements/elements/hadoop
History
Jeremy Freudberg a77a9a978a Add S3 jar to Hadoop classpath 
As prereq of support for S3 datasource, the hadoop-aws jar needs to be
in the Hadoop classpath. The jar is copied into the proper folder when
possible on the appropriate plugins, and otherwise can be provided from
a download URL by the user.

Additionally, set the correct value of DIB_HDFS_LIB_DIR on the Vanilla
plugin to avoid any unnecessary simlinking.

Partially-Implements: bp sahara-support-s3

Change-Id: I94c5b0055b87f6a4e1382118d0718e588fccfe87
2017-07-28 14:01:01 +00:00
..
install.d Add S3 jar to Hadoop classpath 2017-07-28 14:01:01 +00:00
root.d Adding rhel7 to elements checks 2016-09-20 17:28:02 -03:00
element-deps Start switching to declarative package-installs 2015-04-15 00:44:54 +08:00
package-installs.yaml fix firstboot script 2016-12-22 08:58:20 +00:00
pkg-map fix firstboot script 2016-12-22 08:58:20 +00:00
README.rst drop vanilla 2.6.0 support from elements 2016-08-22 17:41:05 +03:00

README.rst

hadoop

Installs Java and Hadoop, configures SSH.

HOWTO build Hadoop Native Libs

  • Install: jdk >= 6, maven, cmake and protobuf >= 2.5.0

  • Get Hadoop source code:

    wget http://archive.apache.org/dist/hadoop/core/hadoop-2.7.1/hadoop-2.7.1-src.tar.gz

  • Unpack source:

    tar xvf hadoop-2.7.1-src.tar.gz

  • Build Hadoop:

    cd hadoop-2.7.1-src
mvn package -Pdist,native -DskipTests

  • Create tarball with Hadoop Native Libs:

    cd hadoop-dist/target/hadoop-2.7.1/lib
tar -czvf hadoop-native-libs-2.7.1.tar.gz native