flume-ng demo配置详解
一、简单测试配置信息flume-ng启动agent的命令为：
$ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template
添加如下配置文件
example.conf
# example.conf: A single-node Flume configuration
# Name the components on this agent
agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1
# Describe/configure source1
agent1.sources.source1.type = netcat
agent1.sources.source1.bind = localhost
agent1.sources.source1.port = 44444
# Describe sink1
agent1.sinks.sink1.type = logger
# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000
agent1.channels.channel1.transactionCapactiy = 100
# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1
以上配置信息描述的是：
该配置文件中 配置了一个代理agent1
在代理agent中配置了一个source（源）一个sink（接收器）和一个channel（通道），分别为：source1，sink1，channel1
source1的类型定义为netcat，对应该类型参数为bind和port 分别为localhost和44444
sink1的类型定义为logger，直接输出到日志文件中
channel的类型定义为内存方式，设置其参数capactiy和transactionCapacity分别为1000和100
指定source1和sink1的channel为channel1
现在我们可以启动看看我们的flume-ng是否成功。
在flume-ng的home目录下
$ bin/flume-ng agent --conf-file example.conf --name agent1 -Dflume.root.logger=INFO,console --conf = conf
启动之后日志信息如下：
2012-07-30 15:42:30,708 (main) Starting lifecycle supervisor 1
2012-07-30 15:42:30,713 (main) Flume node starting - agent1
2012-07-30 15:42:30,715 (lifecycleSupervisor-1-1) Node manager starting
2012-07-30 15:42:30,716 (lifecycleSupervisor-1-2) Configuration provider starting
2012-07-30 15:42:30,718 (lifecycleSupervisor-1-1) Starting lifecycle supervisor 11
2012-07-30 15:42:30,720 (conf-file-poller-0) Reloading configuration file:example.conf
2012-07-30 15:42:30,726 (conf-file-poller-0) Processing:sink1
2012-07-30 15:42:30,726 (conf-file-poller-0) Processing:sink1
2012-07-30 15:42:30,726 (conf-file-poller-0) Added sinks: sink1 Agent: agent1
2012-07-30 15:42:30,747 (conf-file-poller-0) Post-validation flume configuration contains configurationfor agents:
2012-07-30 15:42:30,747 (conf-file-poller-0) Creating channels
2012-07-30 15:42:30,808 (conf-file-poller-0) Monitoried counter group for type: CHANNEL, name: channel1, registered successfully.
2012-07-30 15:42:30,808 (conf-file-poller-0) created channel channel1
2012-07-30 15:42:30,828 (conf-file-poller-0) Creating instance of sink: sink1, type: logger
2012-07-30 15:42:30,832 (conf-file-poller-0) Starting new configuration:{ sourceRunners:{source1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource@18f1d7e }} sinkRunners:{sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@d9660d counterGroup:{ name:null counters:{} } }} channels:{channel1=org.apache.flume.channel.MemoryChannel@bb0d0d} }
2012-07-30 15:42:30,832 (conf-file-poller-0) Starting Channel channel1
2012-07-30 15:42:30,835 (lifecycleSupervisor-1-0) Component type: CHANNEL, name: channel1 started
2012-07-30 15:42:30,835 (conf-file-poller-0) Waiting for channel: channel1 to start. Sleeping for 500 ms
2012-07-30 15:42:31,338 (conf-file-poller-0) Starting Sink sink1
2012-07-30 15:42:31,338 (conf-file-poller-0) Starting Source source1
2012-07-30 15:42:31,339 (lifecycleSupervisor-1-1) Source starting
2012-07-30 15:42:31,358 (lifecycleSupervisor-1-1) Created serverSocket:sun.nio.ch.ServerSocketChannelImpl
我们打开另外一个console：执行 telnet localhost 44444
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
haha
OK
在日志控制台，我们看到如下信息
2012-07-30 15:42:40,780 (SinkRunner-PollingRunner-DefaultSinkProcessor) Event: { headers:{} body: 68 61 68 61 0D haha. }
说明我们的flume-ng运行良好。可以继续下一步操作…………
页:
[1]