mac os x 编译spark-2.1.0 for hadoop-2.7.3
mac os x maven编译spark-2.1.0for hadoop-2.7.31.官方文档中要求安装Maven 3.3.9+ 和Java 8 ;
2.执行
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"
3.cd spark2.1.0源码根目录下
./build/mvn -Pyarn-Phadoop-2.7-Dhadoop.version=2.7.3-Dscala-2.11 -Phive-Phive-thriftserver -DskipTests clean package
--
------------------------------------------------------------------------
Reactor Summary:
Spark Project Parent POM ........................... SUCCESS
Spark Project Tags ................................. SUCCESS [ 32.297 s]
Spark Project Sketch ............................... SUCCESS
Spark Project Networking ........................... SUCCESS [ 22.433 s]
Spark Project Shuffle Streaming Service ............ SUCCESS
Spark Project Unsafe ............................... SUCCESS [ 10.949 s]
Spark Project Launcher ............................. SUCCESS [ 41.786 s]
Spark Project Core ................................. SUCCESS
Spark Project ML Local Library ..................... SUCCESS [ 25.018 s]
Spark Project GraphX ............................... SUCCESS [ 10.133 s]
Spark Project Streaming ............................ SUCCESS [ 22.240 s]
Spark Project Catalyst ............................. SUCCESS
Spark Project SQL .................................. SUCCESS
Spark Project ML Library ........................... SUCCESS [ 58.186 s]
Spark Project Tools ................................ SUCCESS
Spark Project Hive ................................. SUCCESS
Spark Project REPL ................................. SUCCESS
Spark Project YARN Shuffle Service ................. SUCCESS
Spark Project YARN ................................. SUCCESS [ 20.450 s]
Spark Project Hive Thrift Server ................... SUCCESS [ 19.625 s]
Spark Project Assembly ............................. SUCCESS
Spark Project External Flume Sink .................. SUCCESS [ 15.382 s]
Spark Project External Flume ....................... SUCCESS
Spark Project External Flume Assembly .............. SUCCESS
Spark Integration for Kafka 0.8 .................... SUCCESS [ 15.826 s]
Spark Project Examples ............................. SUCCESS [ 14.344 s]
Spark Project External Kafka Assembly .............. SUCCESS
Spark Integration for Kafka 0.10 ................... SUCCESS [ 13.367 s]
Spark Integration for Kafka 0.10 Assembly .......... SUCCESS
Kafka 0.10 Source for Structured Streaming ......... SUCCESS
Spark Project Java 8 Tests ......................... SUCCESS
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time: 15:32 min
Finished at: 2017-05-01T14:11:28+08:00
Final Memory: 88M/981M
------------------------------------------------------------------------
--
The requested profile "hadoop-2.8" could not be activated because it does not exist.
mvn 从github 下载 hadoop-2.8.0 的lib文件，这些lib文件还没人准备。。。因此下载不到，不会编译。
4切换到编译完的dev目录下执行下面命令打包
测试mvn编译成功后，可以用如下命令生成安装包：(当然直接执行生成安装包，但安全起见，还是先执测试一下)
./dev/make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver-Pyarn
参考：
http://spark.apache.org/docs/latest/building-spark.html
页:
[1]