python基于Hadoop Streaming实现简单的WordCount
1. Eclipse下配置python插件PyDev参考：http://blog.iyunv.com/uid-11121450-id-1476897.html
2. Hadoop Streaming编程
Hadoop Streaming是利用“标准输入”和“标准输出”与我们编写的Map和Reduce进行数据的交换。
那么，任何能够使用“标准输入”和“标准输出”的编程语言都应该可以用来编写MapReduce程序。
3. Python实现简单的Word Count程序
需要注意python中对分隔符的要求比较严格，尽量不要混用tab和空格，很容易导致编译失败
mapper.py
#!/usr/bin/python
import sys
for line in sys.stdin:
#去除字符串两边的空格
line = line.strip()
#按照空格去划分单词
words = line.split()
for word in words:
print '%s %s' % (word, 1)
reducer.py
#!/usr/bin/python
from operator import itemgetter
import sys
word2count = {}
for line in sys.stdin:
line = line.strip()
word, count = line.split(' ', 1)
try:
count = int(count)
word2count = word2count.get(word, 0) + count
except ValueError:
pass
sorted_word2count = sorted(word2count.items(), key=itemgetter(0))
for word, count in sorted_word2count:
print '%s %s'% (word, count)
4. 在hadoop上运行python脚本
#在Hadoop Home路径下运行以下命令
bin/hadoop jar contrib/streaming/hadoop-streaming-1.0.0.jar -mapper /usr/local/mapper.py -reducer /usr/local/reducer.py -input /test.txt -output /result
页:
[1]