通过ngx-lua来统计nginx上的虚拟主机性能数据

cundeng

介绍
　　以前我们为nginx做统计,都是通过对日志的分析来完成.比较麻烦,现在基于ngx_lua插件,开发了实时统计站点状态的脚本,解放生产力.
项目主页: https://github.com/skyeydemon/ngx-lua-stats
功能

• 支持分不同虚拟主机统计, 同一个虚拟主机下可以分不同的location统计.
  
• 可以统计与query-times request-time status-code speed 相关的数据.
  

环境依赖

• nginx + ngx_http_lua_module
  

安装
　　http://wiki.nginx.org/HttpLuaModule#Installation
使用方法
添加全局字典
　　在nginx的配置中添加dict的初始化, 类似如下
 

1 2	[color=teal !important]lua_shared_dict [color=#000000 !important]log_dict[color=#009999 !important]20M[color=#333333 !important]; [color=teal !important]lua_shared_dict [color=#000000 !important]result_dict[color=#009999 !important]20M[color=#333333 !important];

为特定的location添加统计
　　只需要添加一句即可~~
将lua脚本嵌套进nginx的配置中, 例如:
 

1 2 3 4 5 6 7 8 9 10 11 12

[color=teal !important]server[color=#333333 !important]{ [color=#006fe0 !important]    [color=#000000 !important]listen[color=#009999 !important]8080[color=#333333 !important]; [color=#006fe0 !important]    [color=teal !important]server_name [color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com[color=#333333 !important]; [color=#006fe0 !important]    [color=#000000 !important]access_log[color=#006fe0 !important]/[color=#000000 !important]home[color=#006fe0 !important]/[color=#000000 !important]work[color=#006fe0 !important]/[color=#000000 !important]nginx[color=#006fe0 !important]/[color=#000000 !important]logs[color=#006fe0 !important]/[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com[color=#333333 !important].[color=teal !important]log [color=#000000 !important]milog[color=#333333 !important]; [color=#006fe0 !important]    [color=teal !important]location[color=#006fe0 !important]/[color=#333333 !important]{ [color=#006fe0 !important]        [color=teal !important]proxy_set_header [color=#000000 !important]Host[color=#333333 !important]$[color=#000000 !important]host[color=#333333 !important]; [color=#006fe0 !important]        [color=#000000 !important]proxy_set_header[color=#000000 !important]X[color=#006fe0 !important]-[color=#000000 !important]Forwarded[color=#006fe0 !important]-[color=#000000 !important]For[color=#333333 !important]$[color=#000000 !important]remote_addr[color=#333333 !important]; [color=#006fe0 !important]        [color=teal !important]proxy_pass [color=#000000 !important]http[color=#006fe0 !important]:[color=#999999 !important]//weatherapi.market.xiaomi.com_backend; [color=#006fe0 !important]     [color=#006fe0 !important]        [color=#000000 !important]log_by_lua_file[color=#333333 !important].[color=#006fe0 !important]/[color=#000000 !important]site[color=#006fe0 !important]-[color=#000000 !important]enable[color=#006fe0 !important]/[color=#000000 !important]record[color=#333333 !important].[color=#002d7a !important]lua[color=#333333 !important]; [color=#006fe0 !important]    [color=#333333 !important]} [color=#333333 !important]}

输出结果
　　通过配置一个server, 使得可以通过curl获取到字典里的所有结果
 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

[color=teal !important]server[color=#333333 !important]{ [color=#006fe0 !important]    [color=#000000 !important]listen[color=#009999 !important]8080[color=#000000 !important]default[color=#333333 !important]; [color=#006fe0 !important]    [color=#000000 !important]server_name[color=#000000 !important]_[color=#333333 !important]; [color=#006fe0 !important]    [color=teal !important]location[color=#006fe0 !important]/[color=#333333 !important]{ [color=#006fe0 !important]        [color=#000000 !important]return[color=#009999 !important]404[color=#333333 !important]; [color=#006fe0 !important]    [color=#333333 !important]}   [color=#006fe0 !important]    [color=teal !important]location[color=#006fe0 !important]/[color=teal !important]status[color=#333333 !important]{ [color=#006fe0 !important]        [color=#000000 !important]content_by_lua_file[color=#333333 !important].[color=#006fe0 !important]/[color=#000000 !important]site[color=#006fe0 !important]-[color=#000000 !important]enable[color=#006fe0 !important]/[color=#000000 !important]output[color=#333333 !important].[color=#002d7a !important]lua[color=#333333 !important]; [color=#006fe0 !important]    [color=#333333 !important]}   [color=#006fe0 !important]    [color=teal !important]location[color=#006fe0 !important]/[color=teal !important]empty_dict[color=#333333 !important]{ [color=#006fe0 !important]        [color=#000000 !important]content_by_lua_file[color=#333333 !important].[color=#006fe0 !important]/[color=#000000 !important]site[color=#006fe0 !important]-[color=#000000 !important]enable[color=#006fe0 !important]/[color=#000000 !important]empty_dict[color=#333333 !important].[color=#002d7a !important]lua[color=#333333 !important]; [color=#006fe0 !important]    [color=#333333 !important]} [color=#333333 !important]}

　　可以通过如下命令获取
　　curl ip_addr:8080/status
清理字典
　　运行一段时间之后, 字典会变大. 可以通过如下接口清理
　　curl ip_addr:8080/empty_dict
支持的统计数据说明
　　目前支持统计以下数据,返回的原始数据类似于
 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

[color=#006fe0 !important]--[color=#006fe0 !important]--[color=#006fe0 !important]--[color=#006fe0 !important]--[color=#006fe0 !important]--[color=#006fe0 !important]--[color=#006fe0 !important]--[color=#006fe0 !important]--[color=#006fe0 !important]--[color=#006fe0 !important]--[color=#006fe0 !important]--[color=#006fe0 !important]--[color=#006fe0 !important]-- [color=#000000 !important]key[color=#006fe0 !important]:[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com__upstream_time_10[color=#333333 !important].[color=#009999 !important]0.3.32[color=#006fe0 !important]:[color=#009999 !important]8250_counter [color=#009999 !important]0.375 [color=#000000 !important]key[color=#006fe0 !important]:[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com__upstream_time_10[color=#333333 !important].[color=#009999 !important]0.3.32[color=#006fe0 !important]:[color=#009999 !important]8250_nb_counter [color=#009999 !important]124 [color=#000000 !important]key[color=#006fe0 !important]:[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com__upstream_time_10[color=#333333 !important].[color=#009999 !important]0.4.93[color=#006fe0 !important]:[color=#009999 !important]8250_counter [color=#009999 !important]0.131 [color=#000000 !important]key[color=#006fe0 !important]:[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com__upstream_time_10[color=#333333 !important].[color=#009999 !important]0.4.93[color=#006fe0 !important]:[color=#009999 !important]8250_nb_counter [color=#009999 !important]123 [color=#000000 !important]key[color=#006fe0 !important]:[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com__upstream_time_10[color=#333333 !important].[color=#009999 !important]20.12.49[color=#006fe0 !important]:[color=#009999 !important]8250_counter [color=#009999 !important]0.081 [color=#000000 !important]key[color=#006fe0 !important]:[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com__upstream_time_10[color=#333333 !important].[color=#009999 !important]20.12.49[color=#006fe0 !important]:[color=#009999 !important]8250_nb_counter [color=#009999 !important]127 [color=#000000 !important]key[color=#006fe0 !important]:[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com__query_counter [color=#009999 !important]500 [color=#000000 !important]key[color=#006fe0 !important]:[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com__request_time_counter [color=#009999 !important]0.683 [color=#000000 !important]key[color=#006fe0 !important]:[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com__upstream_time_counter [color=#009999 !important]0.683 [color=#000000 !important]key[color=#006fe0 !important]:[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com__upstream_time_10[color=#333333 !important].[color=#009999 !important]20.12.59[color=#006fe0 !important]:[color=#009999 !important]8250_counter [color=#009999 !important]0.096 [color=#000000 !important]key[color=#006fe0 !important]:[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com__upstream_time_10[color=#333333 !important].[color=#009999 !important]20.12.59[color=#006fe0 !important]:[color=#009999 !important]8250_nb_counter [color=#009999 !important]126 [color=#000000 !important]key[color=#006fe0 !important]:[color=#000000 !important]weatherapi[color=#333333 !important].[color=#002d7a !important]market[color=#333333 !important].[color=#002d7a !important]xiaomi[color=#333333 !important].[color=#002d7a !important]com__bytes_sent_counter [color=#009999 !important]81500

　　其中 __ 用来分割虚拟主机(包含prefix)与后面的数据项，便于数据处理.
counter表示此值一直在累加
nb表示次数
可以得到的数据包括: query次数 request_time bytes_sent upstream_time
其中 upstream_time_10.20.12.49:8250_counter 表示到某个特定后端的upstrea_time耗时累加
upstream_time_10.20.12.49:8250_nb_counter 表示到到某个特定后端的upstrea_time次数累加
如何处理数据
　　因为采集到的数据大多都是counter值,需要监控系统支持对于delta的处理.目前我们公司的perf-counter监控系统支持简单运算。所以这个处理起来比较简单，对于没有这种系统的同学来说，需要自己处理数据,得到delta值以及更复杂的数据，比如：
　　转自http://noops.me/?p=867