Lotgstash日志切割示例

4532132

                      

logstash的功能有一点是把 各种软件生成的各种格式的日志 转换成一个方便检索筛选的格式，本文演示了一个最简单的例子。

一  转换的效果

实例： rabbitmq-server 日志：

=INFO REPORT==== 16-Jan-2017::09:27:09 ===
Mirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>

转换后的格式为：

{

"year" => "2017",
"mounthday" => "16",
"logdata" => "Mirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>",
"message" => "=INFO REPORT==== 16-Jan-2017::09:27:09 ===\nMirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>",
"type" => "rabbit",
"tags" => [
[0] "multiline"
],
"path" => "/var/log/rabbitmq/rabbit@server-31.log",
"@timestamp" => 2017-01-16T01:27:09.718Z,
"loglevel" => "INFO",
"@version" => "1",
"host" => "server-31",
"time" => "09:27:09",
"mounth" => "Jan"
}

转换后的内容传入elasticsearch中，用户就可以按照时间、日志等级、主机等对汇总的日志进行筛选检索

二 转换的过程

还是以刚才那条日志为例

=INFO REPORT==== 16-Jan-2017::09:27:09 ===
Mirrored queue 'reply_963a14cce15f48e786240aad41817847' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2262.0>

=INFO REPORT==== 16-Jan-2017::09:27:09 ===
Mirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>

=INFO REPORT==== 16-Jan-2017::09:27:09 ===
Mirrored queue 'q-agent-notifier-network-update' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2270.0>

日志是多行，前后各有一行空行，日志行以=开头，

1、多行合并

首先是合并多行，

安装多行插件：

/usr/share/logstash/bin/logstash-plugin install  logstash-filter-multiline

在配置文件中配置多行合并

codec => multiline {

pattern => "^="
what => "previous"
negate => true
}

最终日志转换为 =INFO REPORT==== 16-Jan-2017::09:27:09 ===\nMirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>

2、分析日志的格式和规律

结合所有的rabbitmq的日志总结规律为

=“日志级别” REPORT====  "日期"::“时间” ===\n“日志内容”

注意不要忘记中间的空格

3、正则匹配

logstash内置了很多常规正则，参见

https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

本文都是采用内置的正则

=INFO REPORT==== 16-Jan-2017::09:27:09 ===\nMirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>

我最终的匹配的表达式为

^=%{LOGLEVEL:loglevel} REPORT=+ %{MONTHDAY:mounthday}-%{MONTH:mounth}-%{YEAR:year}::%{TIME:time} ===\n%{GREEDYDATA:logdata}$

%{LOGLEVEL:loglevel}表示这是一个变量，里面的内容要匹配logstash内置的LOGLEVE正则，并且里面的内容和loglevel这个key形成一对kv值："loglevel":"INFO"

其他一直类推

logstash提供了一个测试表达式的网址http://grokdebug.herokuapp.com/