hadoop 下的debug

wendu

2011.12.20
1. hadoop 集群环境中对namenode,datanode,jobtracker,tasktracker的监控甚至是debug.
  要修改hadoop-env.sh文件:

export HADOOP_TASKTRACKER_OPTS=" -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.port=9051 -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9052 "

export HADOOP_TASKTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_TASKTRACKER_OPTS"

这样可以通过 jconsole远程连接 tasktracker_server_ip:9051 查看jvm的运行状态,包括线程,堆栈,运行参数等.
可以通过Eclipse Remote debug 连接tasktracker_server_ip:9052,打断点来跟踪TaskTracker的程序运行步骤.

2. 当设置 mapred.job.reuse.jvm.num.tasks 设置为-1时,只有属于同一个job的多个map(分配给一台同样的服务器)才会在同一个jvm中运行,对于属于同一个job的map,reduce还未知.

3. 对于map及reduce任务,都是由TaskController(默认为DefaultTaskController)生成一个taskjvm.sh 文件,然后由java的Process类来执行,通过process.waitFor()来返回exitCode,如果exitCode==0说明进程正常结束,如果返回非0,说明进程非正常退出,在taskjvm.sh可以看到所有的运行参数和变量(可以通过grep tasktracker.log日志找到),通过查看task日志来分析原因,每个task的日志在 ${hadoop.log.dir}/userlogs/${job_id}/${task_id}/ 目录下,有stderr 错误输出,stdout 标准输出,syslog 运行日志,log.index 前面三个文件的数据大小.

4. 如果jvm设置为重用,对于一个job的debug可以通过设置属性来对job进行jmx和debug
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx512m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.port=21000 -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=22000 </value>
</property>

<property>
<name>mapred.reduce.child.java.opts</name>
<value>-Xmx512m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.port=10011 -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=10012 </value>
</property>

2012.01.19
  1. hadoop -D参数的设置位置:
hadoop  jar hbase-json-test.jar -Dmapred.output.compress=true -Dmapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec /maxthon/mac_export/20120115/ duming/mac_hfile/20120121/

2012.05.11
  1. hadoop distcp
hadoop distcp -Dmapred.map.child.java.opts=" -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=22000 " -skipcrccheck -update -m 1 hftp://namenode-bj.hadoop.maxthon.cn:50070/user/yangfeng/tttt.txt /user/hadoop/duming/
  不版本间的数据迁移要注意:
  a. source 用hftp协议.
  b. 迁移到的hadoop集群版本比较高, 最好设置-skipcrccheck选项也-update选项, skipcrccheck忽略FileChecksum校验, 因为版本的升级可能带来Checksum值不一样, cdh4与cdh3就是这样.

2012.06.01
1. load数据到现有的partition 下, 报错误, 但最终还是能将数据转移过去, 最终原因是没有使用use views;

2012.06.09
org.apache.hadoop.hive.ql.parse.SemanticException: org.apache.hadoop.hive.ql.metadata.HiveException: Invalid partition for table chukwa
at org.apache.hadoop.hive.ql.optimizer.pcr.PcrOpProcFactory$FilterPCR.process(PcrOpProcFactory.java:122)
at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102)
at org.apache.hadoop.hive.ql.optimizer.pcr.PartitionConditionRemover.transform(PartitionConditionRemover.java:78)
at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:87)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7306)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:889)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:200)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: Invalid partition for table chukwa
at org.apache.hadoop.hive.ql.metadata.Hive.getPartitionsByNames(Hive.java:1696)
at org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.pruneBySequentialScan(PartitionPruner.java:376)
at org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:233)
at org.apache.hadoop.hive.ql.optimizer.pcr.PcrOpProcFactory$FilterPCR.process(PcrOpProcFactory.java:112)
... 21 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Invalid partition for table chukwa
at org.apache.hadoop.hive.ql.metadata.Partition.initialize(Partition.java:207)
at org.apache.hadoop.hive.ql.metadata.Partition.<init>(Partition.java:106)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartitionsByNames(Hive.java:1691)
... 24 more
Caused by: MetaException(message:Invalid partition key & values)
at org.apache.hadoop.hive.metastore.Warehouse.makePartName(Warehouse.java:407)
at org.apache.hadoop.hive.metastore.Warehouse.makePartName(Warehouse.java:392)
at org.apache.hadoop.hive.ql.metadata.Partition.initialize(Partition.java:190)
... 26 more
是因为数据库 PARTITION_KEY_VALS中对应某个PART_ID的分区数据少一个logType记录,
hive存储的meta信息位于数据库中, 类为ObjectStore,
2012.07.03
  hadoop  jar  hbase-promote-0.0.1-SNAPSHOT.jar com.maxthon.hbase.promote.etl.EtlPromoteDriver -Dmapred.map.child.java.opts="-Xdebug -Xrunjdwp:transport=dt_socket,address=9090,server=y,suspend=y,address=22000 " /hive/warehouse/maxthon.db/chukwa/logtype=com_ios.g.maxthon.com/logday=20120630/loghour=00 /user/yangfeng/promote_ios/01

2012.07.04
mapred.min.split.sizeMxCombineFileInputFormatShim.getSplits
mapred.min.split.size.per.nodeMxCombineFileInputFormatShim.getSplits
mapred.max.split.size  256000000
mapreduce.input.fileinputformat.split.minsize.per.node  1
mapreduce.input.fileinputformat.split.maxsize  256000000  每个block块的大小 MxCombineFileInputFormatNew.getSplits, MxCombineFileInputFormatNew.OneFileInfo判断

2012.07.10
  1. 通过-D更改参数, main方法调用, 参考DistCp
    JobConf job = new JobConf(DistCp.class);
    DistCp distcp = new DistCp(job);
    int res = ToolRunner.run(distcp, args);