Hadoop Oozie学习笔记(二) 使用OOZIE,通过命令行运行example

xxxmenger

　　本博客属原创文章,转载请注明出处:http://guoyunsky.iyunv.com/blog/1245092
　　欢迎加入Hadoop超级群: 180941958   
　　 
　　    Oozie下面有很多例子,提供测试.也可以将源码放入Eclipse中启动提交.这里就一起试下.但发现有些问题,一一解决吧.
　　     运行Oozie例子map-reduce,命令:
　　     $OOZIE_HOME/bin/oozie job -oozie http://localhost:11000/oozie -config /home/guoyun/hadoop/oozie-3.0.2/examples/apps/map-reduce/job.properties -run
　　     本以为会很顺利,但控制台报以下错误:
　　     Error: E0902 : E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
　　    跑到日志中$OOZIE_HOME/logs/oozie.log发现有以下报错:
　　  2011-11-09 10:42:30,721 WARN V1JobsServlet:539 - USER[?] GROUP[users] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://localhost:11000/oozie/v1/jobs?action=start] error[E0902], E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]

org.apache.oozie.servlet.XServletException: E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:196)
at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:89)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:281)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.oozie.service.AuthorizationException: E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:320)
at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:185)
... 16 more
Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
at org.apache.oozie.service.KerberosHadoopAccessorService.createFileSystem(KerberosHadoopAccessorService.java:208)
at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:285)
... 17 more
Caused by: java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1131)
at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy22.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:398)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:384)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:111)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:213)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:180)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1514)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1548)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1530)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
at org.apache.oozie.service.KerberosHadoopAccessorService$3.run(KerberosHadoopAccessorService.java:200)
at org.apache.oozie.service.KerberosHadoopAccessorService$3.run(KerberosHadoopAccessorService.java:192)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.oozie.service.KerberosHadoopAccessorService.createFileSystem(KerberosHadoopAccessorService.java:192)
... 18 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:425)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:532)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:210)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1244)
at org.apache.hadoop.ipc.Client.call(Client.java:1075)
... 37 more
　　  从Oozie的FAQ找到了解决办法:https://github.com/yahoo/oozie/wiki/FAQ,需要将oozie-default.xml中的 oozie.services属性拷贝到oozie-site.xml中,也就是以下属性:

     <property>
<name>oozie.services</name>
<value>
org.apache.oozie.service.SchedulerService,
org.apache.oozie.service.InstrumentationService,
org.apache.oozie.service.CallableQueueService,
org.apache.oozie.service.UUIDService,
org.apache.oozie.service.ELService,
org.apache.oozie.service.AuthorizationService,
org.apache.oozie.service.KerberosHadoopAccessorService,
org.apache.oozie.service.MemoryLocksService,
org.apache.oozie.service.DagXLogInfoService,
org.apache.oozie.service.SchemaService,
org.apache.oozie.service.LiteWorkflowAppService,
org.apache.oozie.service.JPAService,
org.apache.oozie.service.StoreService,
org.apache.oozie.service.CoordinatorStoreService,
org.apache.oozie.service.SLAStoreService,
org.apache.oozie.service.DBLiteWorkflowStoreService,
org.apache.oozie.service.CallbackService,
org.apache.oozie.service.ActionService,
org.apache.oozie.service.ActionCheckerService,
org.apache.oozie.service.RecoveryService,
org.apache.oozie.service.PurgeService,
org.apache.oozie.service.CoordinatorEngineService,
org.apache.oozie.service.BundleEngineService,
org.apache.oozie.service.DagEngineService,
org.apache.oozie.service.CoordMaterializeTriggerService,
org.apache.oozie.service.StatusTransitService,
org.apache.oozie.service.PauseTransitService
</value>
<description>
All services to be created and managed by Oozie Services singleton.
Class names must be separated by commas.
</description>
</property>
　　   拷贝完成后重启OOZIE,继续跑map-reduce例子,控制台上却又报一样的错误:
　　   Error: E0902 : E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]

  还是看继续看日志吧,还是同样的错误.https://github.com/yahoo/oozie/wiki/FAQ中貌似也有解决办法,于是按照说明将org.apache.oozie.service.HadoopAccessorService 添加到oozie-size.xml中的oozie.services属性中.继续重启.

   还是一样的错误,怒了!难道不是Hadoop授权问题?而且本身来说我都是本地启动,不存在授权的问题啊.于是看下端口,lsof -i:8020,果真空空如也.早就应该想到这点了.于是去$HADOOP_HOME/conf/下查看core-site.xml发现HDFS的端口是9000,并不是这里默认的8020.同时看下maped-site.xml发现jobTracker端口是9001.于是去对应的app下,也就是我这里的$OOZIE_HOME/examples/apps/map-reduce下将job.properties中修改如下:

nameNode=hdfs://localhost:9000
jobTracker=localhost:9001
queueName=default
examplesRoot=hadoop/oozie-3.0.2/examples
oozie.coord.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/aggregator
outputDir=map-reduce
　　  于是继续运行这个example,报如下错误:
　　   Error: E0504 : E0504: App directory [hdfs://localhost:9000/user/guoyun/hadoop/oozie-3.0.2/examples/apps/aggregator] does not exist
　　  想想我现在是伪分布式下运行Hadoop,从HDFS中拿文件则需要先存入到HDFS中.还是直接调用本地文件方便调试,于是将job.properties修改如下:

nameNode=file://
jobTracker=localhost:9001
queueName=default
examplesRoot=hadoop/oozie-3.0.2/examples
oozie.wf.application.path=${nameNode}/home/${user.name}/${examplesRoot}/apps/map-reduce
outputDir=map-reduce
　　   再次运行得到以下结果:
　　   job: 0000001-111109111632287-oozie-guoy-W
　　   再去http://localhost:11000/oozie查看也能看到该job,并有相应的状态.
　　   OK!example运行成功!
更多文章、感悟、分享、勾搭，请用微信扫描: