Hadoop学习日志（一）

xiaoyu28

Hadoop安装
0.安装java环境
<!--[if !supportLists]-->1.       <!--[endif]-->下载版本1.2.1版本Hadoop
<!--[if !supportLists]-->2.       <!--[endif]-->tar zxvf  解压缩hadoop到相应目录
<!--[if !supportLists]-->3.       <!--[endif]-->需要配置的4个文件：hadoop-env.sh、core-site.xml、hdfs-site.xml、mapred-site.xml、masters、slaves
4、文件说明：
hadoop-env.sh是hadoop启动环境设置
hdfs-site.xml  是分布式文件系统相关配置
mapred-site.xml是map-reduce任务相关配置
master 表示主从机器（主机器是nameNode，丛就是dataNode）
slaver 表示丛机器
5、hadoop-env.sh配置详细：
 
# Set Hadoop-specific environment variables here.
 
# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
 
# The java implementation to use.  Required. 配置该项目，设置jDK所在目录
export JAVA_HOME=/usr/local/jdk/jdk1.6.0_45
 
# Extra Java CLASSPATH elements.  Optional.可选的配置CLASSPATH
# export HADOOP_CLASSPATH=
 
# The maximum amount of heap to use, in MB. Default is 1000. 配置虚拟机内存
# export HADOOP_HEAPSIZE=2000
 
# Extra Java runtime options.  Empty by default. 配置模式，肯定是server模式了，client说明内存太小了，是java虚拟机模式，设计的内存回收等相关算法
# export HADOOP_OPTS=-server
 
# Command specific options appended to HADOOP_OPTS when specified 配置JMX管理配置
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS
 
# Extra ssh options.  Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
 
# Where log files are stored.  $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
 
# File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
 
# host:path where hadoop code should be rsync'd from.  Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop
 
# Seconds to sleep between slave commands.  Unset by default.  This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1
 
# The directory where pid files are stored. /tmp by default.
# NOTE: this should be set to a directory that can only be written to by
#       the users that are going to run the hadoop daemons.  Otherwise there is
#       the potential for a symlink attack.
# export HADOOP_PID_DIR=/var/hadoop/pids
 
# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=$USER
 
# The scheduling priority for daemon processes.  See 'man nice'.
# export HADOOP_NICENESS=10
 
 
由于以上配置设计到 ${HADOOP_HOME} 所以需要在/etc/profile  配置
export HADOOP_HOME=/usr/local/hadoop/hadoop 就是Hadoop所在的目录
 
 
5、core-site.xml核心配置：
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<!-- Put site-specific property overrides in this file. -->
<!—配置端口号 hadoop1 是本机机器名称需要在hosts文件夹下映射配置，也可以配置IP地址   - ->
<configuration>
  <property>
   <name>fs.default.name</name>
    <value>hdfs://hadoop1:9000</value>
  </property>
<!--  配置文件系统所在目录 à
  <property> 
  <name>hadoop.tmp.dir</name> 
  <value>/usr/local/hadoop/hadoop/tmp</value>
  </property> 
</configuration>
 
6、配置hdfs-site.xml：
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<!-- Put site-specific property overrides in this file. -->
 
<configuration>
<!--  是否打开权限检查 à
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<!--  副本复制的数目 à
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
 
7、配置mapred-site.xml：
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<!-- Put site-specific property overrides in this file. -->
 
<configuration>
<!--  配置jobtracker端口号，dataNode会使用 à
<property>
<name>mapred.job.tracker</name>
<value>hdfs://hadoop1:9001</value>
</property>
</configuration>
8、master配置：
配置那些是master可以配置机器名称也可以配置IP地址
例如：
192.168.100.101
192.168.100.102
9、salver配置：
配置那些事slaver
192.168.100.101
192.168.100.102
10、配置SSH无密登录
免密码ssh设置（合适的权限很重要）
登入hadoop账户，建立ssh文件夹    mkdir .ssh
现在确认能否不输入口令就用ssh登录本机:
$ ssh namenode
如果不输入口令就无法用ssh登陆namenode，执行下面的命令：
$ ssh-keygen -t rsa –f ~/.ssh/id_rsa 
回车设置密钥，可以设置一个密钥，也可以设置为空密钥。但是安全起见，设置密钥为hadoop，下面利用ssh-agent来设置免密钥登陆集群中其他机器。
私钥放在由-f选项指定的文件之中，例如~/.ssh/id_rsa。存放公钥的文件名称与私钥类似，但是以’.pub’为后缀，本例为~/.ssh/id_rsa.pub
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
将ssh密钥存放在namenode机器的~/.ssh/authorized_keys文件中。
再用scp命令将authorized_keys和id_rsa.pub分发到其它机器的相同文件夹下，例如，将authorized_keys文件分发给datanode1机器的.ssh文件夹的命令为：
$scp ~/.ssh/id_dsa.pub  hadoop@datanode1:/home/hadoop/.ssh
到datanode1机器上  cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
理论上要将公钥发放给各机器然后再各个机器上生成authorized_keys文件，实际上直接分发authorized_keys文件也可以
然后赋予文件权限，赋予各机器的.ssh文件夹权限为711，authorized_keys的权限为644。过高的权限不能通过ssh-add命令，过低的权限无法实现免密码登录。
 
11、启动start-all.sh 就可以启动hadoop 集群了。其他丛机器会自动被ssh无密码登录启动起来。nameNode会启动其他子节点的DataNode。
12、使用jps查看
主机器：
[iyunv@hadoop1 conf]# jps
32365 JobTracker
32090 NameNode
25900 Jps
3017 Bootstrap
32269 SecondaryNameNode
丛机器：
[iyunv@hadoop2 ~]# jps
10009 TaskTracker
9901 DataNode
27852 Jps
恭喜你启动成功了
 
 
 
Hbase安装
由于hbase 和hadoop需要搭配版本的，我用的是1.2.1那么hbase用的是hbase-0.94.23
<!--[if !supportLists]-->1.       <!--[endif]-->tar  zxvf 解压缩 hbase-0.94.23.tar.gz
<!--[if !supportLists]-->2.       <!--[endif]-->修改配置文件hbase-env.sh、hbase-site.xml
<!--[if !supportLists]-->3.       <!--[endif]-->Hbase-env.sh
#
#/**
# * Copyright 2007 The Apache Software Foundation
# *
# * Licensed to the Apache Software Foundation (ASF) under one
# * or more contributor license agreements.  See the NOTICE file
# * distributed with this work for additional information
# * regarding copyright ownership.  The ASF licenses this file
# * to you under the Apache License, Version 2.0 (the
# * "License"); you may not use this file except in compliance
# * with the License.  You may obtain a copy of the License at
# *
# *     http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# */
 
# Set environment variables here.
 
# This script sets variables multiple times over the course of starting an hbase process,
# so try to keep things idempotent unless you want to take an even deeper look
# into the startup scripts (bin/hbase, etc.)
 
# The java implementation to use.  Java 1.6 required.指定JDK所在的目录
# export JAVA_HOME=/usr/local/jdk/jdk1.6.0_45/
 
# Extra Java CLASSPATH elements.  Optional.
# export HBASE_CLASSPATH=
 
# The maximum amount of heap to use, in MB. Default is 1000.
# export HBASE_HEAPSIZE=1000
 
# Extra Java runtime options.
# Below are what we set by default.  May only work with SUN JVM.
# For more on why as well as other possible settings,
# see http://wiki.apache.org/hadoop/PerformanceTuning
export HBASE_OPTS="-XX:+UseConcMarkSweepGC"
 
# Uncomment one of the below three options to enable java garbage collection logging for the server-side processes.
 
# This enables basic gc logging to the .out file.
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps"
 
# This enables basic gc logging to its own file.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH>"
 
# This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M"
 
# Uncomment one of the below three options to enable java garbage collection logging for the client processes.
 
# This enables basic gc logging to the .out file.
# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps"
 
# This enables basic gc logging to its own file.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH>"
 
# This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M"
 
# Uncomment below if you intend to use the EXPERIMENTAL off heap cache.
# export HBASE_OPTS="$HBASE_OPTS -XX:MaxDirectMemorySize="
# Set hbase.offheapcache.percentage in hbase-site.xml to a nonzero value.
 
 
# Uncomment and adjust to enable JMX exporting
# See jmxremote.password and jmxremote.access in $JRE_HOME/lib/management to configure remote password access.
# More details at: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
#
# export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10101"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10102"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10103"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10104"
 
# File naming hosts on which HRegionServers will run.  $HBASE_HOME/conf/regionservers by default.
# export HBASE_REGIONSERVERS=${HBASE_HOME}/conf/regionservers
 
# File naming hosts on which backup HMaster will run.  $HBASE_HOME/conf/backup-masters by default.
# export HBASE_BACKUP_MASTERS=${HBASE_HOME}/conf/backup-masters
 
# Extra ssh options.  Empty by default.
# export HBASE_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HBASE_CONF_DIR"
 
# Where log files are stored.  $HBASE_HOME/logs by default.
# export HBASE_LOG_DIR=${HBASE_HOME}/logs
 
# Enable remote JDWP debugging of major HBase processes. Meant for Core Developers
# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
 
# A string representing this instance of hbase. $USER by default.
# export HBASE_IDENT_STRING=$USER
 
# The scheduling priority for daemon processes.  See 'man nice'.
# export HBASE_NICENESS=10
 
# The directory where pid files are stored. /tmp by default.
# export HBASE_PID_DIR=/var/hadoop/pids
 
# Seconds to sleep between slave commands.  Unset by default.  This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HBASE_SLAVE_SLEEP=0.1
 
# Tell HBase whether it should manage it's own instance of Zookeeper or not.
# export HBASE_MANAGES_ZK=true
 
4、配置hbase-site.xml
<configuration>
 
<property>
 <name>hbase.rootdir</name>
 <value>hdfs://hadoop1:9000/hbase</value>
 </property>
<!-- 设置hbase数据存在位置  -->
<property>
<name>hbase.rootdir</name>
<value>file:///usr/local/hadoop/hbase/hbase-0.94.23/data</value>
</property>
 
<!-- 配置Hbase是分布式模式  -->
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
 
 
<property>
<name>hbase.master</name>
<value>hdfs://hadoop1:60000</value>
</property>
 
<property>
<name>hbase.zookeeper.quorum</name>
<value>hadoop1,hadoop2</value>
</property>
 
</configuration>
~