lx86 发表于 2017-12-23 22:22:46

Logstash收集nginx日志之使用grok过滤插件解析日志

　　grok作为一个logstash的过滤插件，支持根据模式解析文本日志行，拆成字段。


[*]nginx日志的配置：
　　

log_formatmain'$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for"';　　



[*]logstash中grok的正则（添加在logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-xxx/patterns/grok-patterns文件中）为：
　　

WZ ([^ ]*)　　
NGINXACCESS
%{IP:remote_ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{WZ:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} %{QS:xforward}　　

　　logstash的配置为：
　　

input {　　file {
　　path   
=> ["/var/log/nginx/access.log"]　　type   
=> "nginxlog"　　start_position
=> "beginning"　　}
　　
}
　　

　　
filter {
　　grok {
　　match
=> { "message" => "%{NGINXACCESS}" }　　}
　　
}
　　
output {
　　stdout {
　　codec   
=> rubydebug　　}
　　
}
　　

　　logstash的输出：
　　

{"message" => "192.168.154.2 - - \"GET /index.html HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\" \"-\"","@version" => "1","@timestamp" => "2017-03-30T08:27:09.539Z","path" => "/var/log/nginx/access.log","host" => "spark4","type" => "nginxlog","remote_ip" => "192.168.154.2","timestamp" => "30/Mar/2017:01:27:09 -0700","method" => "GET","request" => "/index.html","httpversion" => "1.1","status" => "304","bytes" => "0","referer" => "\"-\"","agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\"","xforward" => "\"-\""　　
}
　　

imissmylove 发表于 2018-1-26 12:36:54

Thanks for sharing!!!!!
页: [1]
查看完整版本: Logstash收集nginx日志之使用grok过滤插件解析日志