HBase配置LZO压缩
系统： gentooHDFS: hadoop:hadoop-0.20.2-cdh3u0
HBase: HBase:0.90.3
安装过程：
1、安装lzo
emerge dev-libs/lzo
2.1 下载
download from:
wget http://www.apache.org/dist//hbase/hbase-0.90.3/hbase-0.90.3.tar.gz
wget http://hadoop-gpl-compression.apache-extras.org.codespot.com/files/hadoop-gpl-compression-0.1.0-rc0.tar.gz
2.2 编译
tar -zxvf hadoop-gpl-compression-0.1.0-rc0.tar.gz
cd hadoop-gpl-compression-0.1.0
cp $HADOOP_HOME//hadoop-core-0.20.2-cdh3u0.jar lib/
nano -w src/java/com/hadoop/compression/lzo/LzoCompressor.java
......
import org.apache.hadoop.conf.Configuration;
.....
public void reinit(Configuration conf) {
// do nothing
}
ant compile-native tar
2.3 拷贝需要的文件
cp build/hadoop-gpl-compression-0.1.0-dev.jar $HADOOP_HOME/lib/
cp build/hadoop-gpl-compression-0.1.0-dev.jar $HBASE_HOME/lib/
rsync -auPv lib/native $HADOOP_HOME/lib/
rsync -auPv lib/native $HBASE_HOME/lib/
3.修改配置重启HBase和Hadoop
nano core-site.xml
<property>
<name>io.compression.codecs</name>
<value>com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
4 测试是否成功
bin/hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://test1.namenode.uc.uuwatch.com:9000/test_path lzo
5 参考资料
http://wiki.apache.org/hadoop/UsingLzoCompression
http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport
http://koven2049.iyunv.com/blog/983502
页:
[1]