论坛 › nginx › Flume采集Nginx日志到HDFS

lakers009 发表于 2017-12-23 20:15:37

Flume采集Nginx日志到HDFS

# 配置Agent　　a1.sources = r1
　　a1.sinks = k1
　　a1.channels = c1
　　# 配置Source
　　a1.sources.r1.type = exec
　　a1.sources.r1.channels = c1
　　a1.sources.r1.deserializer.outputCharset = UTF-8
　　# 配置需要监控的日志输出目录
　　a1.sources.r1.command = tail -F /usr/local/nginx/log/access.log
　　# 配置Sink
　　a1.sinks.k1.type = hdfs
　　a1.sinks.k1.channel = c1
　　a1.sinks.k1.hdfs.useLocalTimeStamp = true
　　a1.sinks.k1.hdfs.path = hdfs://master:9000/flume/events/%Y-%m
　　a1.sinks.k1.hdfs.filePrefix = %Y-%m-%d-%H
　　a1.sinks.k1.hdfs.fileSuffix = .log
　　a1.sinks.k1.hdfs.minBlockReplicas = 1
　　a1.sinks.k1.hdfs.fileType = DataStream
　　a1.sinks.k1.hdfs.writeFormat = Text
　　a1.sinks.k1.hdfs.rollInterval = 86400
　　a1.sinks.k1.hdfs.rollSize = 1000000
　　a1.sinks.k1.hdfs.rollCount = 10000
　　a1.sinks.k1.hdfs.idleTimeout = 0
　　# 配置Channel
　　a1.channels.c1.type = memory
　　a1.channels.c1.capacity = 1000
　　a1.channels.c1.transactionCapacity = 100
　　# 将三者连接
　　a1.sources.r1.channel = c1
　　a1.sinks.k1.channel = c1

查看完整版本: Flume采集Nginx日志到HDFS