设为首页 收藏本站
云服务器等爆品抢先购,低至4.2元/月
查看: 1211|回复: 0

[经验分享] ELK 收集中断错误处理

[复制链接]
累计签到:1 天
连续签到:1 天
发表于 2016-8-4 09:07:22 | 显示全部楼层 |阅读模式
ELK收集中断,定位问题到redis发现redis内存不足,所以迁移到新的redis上
elasticsearch 删除索引语句:
1
curl -XDELETE http://localhost:9200/logstash-nginx.access-2016.05.31



错误1:Failed to send event to Redis
1
Failed to send event to Redis {:event=>#<LogStash::Event:0x4062cce3 @metadata_accessors=#<LogStash::Util::Accessors:0x7cbdd35c @store={"path"=>"/app/local/log/nginx/ckl_access.log"}, @lut={"[path]"=>[{"path"=>"/app/local/log/nginx/ckl_access.log"}, "path"]}>, @cancelled=false, @data={"message"=>"116.209.58.13 | 03/Aug/2016:16:01:01 +0800 | POST /api/v1 HTTP/1.1 | 200 | 69 | {\\x22os\\x22:\\x221\\x22,\\x22v\\x22:\\x222.0.2\\x22,\\x22m\\x22:\\x22user.isFollow\\x22,\\x22ver\\x22:\\x224\\x22,\\x22channel\\x22:\\x22OT_bdhn\\x22,\\x22p\\x22:{\\x22roomId\\x22:\\x221743331\\x22}} | 97 | - | Mozilla/5.0 (Linux; Android 4.4.4; C630Lw Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 QMTV/2.0.2 CHANNEL/OT_bdhn | - | 61.136.167.55 | 0.016 | 0.022 | -", "@version"=>"1", "@timestamp"=>"2016-08-03T08:52:11.442Z", "path"=>"/app/local/log/nginx/ckl_access.log", "host"=>"0.0.0.0", "type"=>"nginx.access", "host_name"=>"ckl_access_front-web15"}, @metadata={"path"=>"/app/local/log/nginx/ckl_access.log"}, @accessors=#<LogStash::Util::Accessors:0x64f18bac @store={"message"=>"116.209.58.13 | 03/Aug/2016:16:01:01 +0800 | POST /api/v1 HTTP/1.1 | 200 | 69 | {\\x22os\\x22:\\x221\\x22,\\x22v\\x22:\\x222.0.2\\x22,\\x22m\\x22:\\x22user.isFollow\\x22,\\x22ver\\x22:\\x224\\x22,\\x22channel\\x22:\\x22OT_bdhn\\x22,\\x22p\\x22:{\\x22roomId\\x22:\\x221743331\\x22}} | 97 | - | Mozilla/5.0 (Linux; Android 4.4.4; C630Lw Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 QMTV/2.0.2 CHANNEL/OT_bdhn | - | 61.136.167.55 | 0.016 | 0.022 | -", "@version"=>"1", "@timestamp"=>"2016-08-03T08:52:11.442Z", "path"=>"/app/local/log/nginx/ckl_access.log", "host"=>"0.0.0.0", "type"=>"nginx.access", "host_name"=>"ckl_access_front-web15"}, @lut={"path"=>[{"message"=>"116.209.58.13 | 03/Aug/2016:16:01:01 +0800 | POST /api/v1 HTTP/1.1 | 200 | 69 | {\\x22os\\x22:\\x221\\x22,\\x22v\\x22:\\x222.0.2\\x22,\\x22m\\x22:\\x22user.isFollow\\x22,\\x22ver\\x22:\\x224\\x22,\\x22channel\\x22:\\x22OT_bdhn\\x22,\\x22p\\x22:{\\x22roomId\\x22:\\x221743331\\x22}} | 97 | - | Mozilla/5.0 (Linux; Android 4.4.4; C630Lw Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 QMTV/2.0.2 CHANNEL/OT_bdhn | - | 61.136.167.55 | 0.016 | 0.022 | -", "@version"=>"1", "@timestamp"=>"2016-08-03T08:52:11.442Z", "path"=>"/app/local/log/nginx/ckl_access.log", "host"=>"0.0.0.0", "type"=>"nginx.access", "host_name"=>"ckl_access_front-web15"}, "path"], "host"=>[{"message"=>"116.209.58.13 | 03/Aug/2016:16:01:01 +0800 | POST /api/v1 HTTP/1.1 | 200 | 69 | {\\x22os\\x22:\\x221\\x22,\\x22v\\x22:\\x222.0.2\\x22,\\x22m\\x22:\\x22user.isFollow\\x22,\\x22ver\\x22:\\x224\\x22,\\x22channel\\x22:\\x22OT_bdhn\\x22,\\x22p\\x22:{\\x22roomId\\x22:\\x221743331\\x22}} | 97 | - | Mozilla/5.0 (Linux; Android 4.4.4; C630Lw Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 QMTV/2.0.2 CHANNEL/OT_bdhn | - | 61.136.167.55 | 0.016 | 0.022 | -", "@version"=>"1", "@timestamp"=>"2016-08-03T08:52:11.442Z", "path"=>"/app/local/log/nginx/ckl_access.log", "host"=>"0.0.0.0", "type"=>"nginx.access", "host_name"=>"ckl_access_front-web15"}, "host"], "type"=>[{"message"=>"116.209.58.13 | 03/Aug/2016:16:01:01 +0800 | POST /api/v1 HTTP/1.1 | 200 | 69 | {\\x22os\\x22:\\x221\\x22,\\x22v\\x22:\\x222.0.2\\x22,\\x22m\\x22:\\x22user.isFollow\\x22,\\x22ver\\x22:\\x224\\x22,\\x22channel\\x22:\\x22OT_bdhn\\x22,\\x22p\\x22:{\\x22roomId\\x22:\\x221743331\\x22}} | 97 | - | Mozilla/5.0 (Linux; Android 4.4.4; C630Lw Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 QMTV/2.0.2 CHANNEL/OT_bdhn | - | 61.136.167.55 | 0.016 | 0.022 | -", "@version"=>"1", "@timestamp"=>"2016-08-03T08:52:11.442Z", "path"=>"/app/local/log/nginx/ckl_access.log", "host"=>"0.0.0.0", "type"=>"nginx.access", "host_name"=>"ckl_access_front-web15"}, "type"], "host_name"=>[{"message"=>"116.209.58.13 | 03/Aug/2016:16:01:01 +0800 | POST /api/v1 HTTP/1.1 | 200 | 69 | {\\x22os\\x22:\\x221\\x22,\\x22v\\x22:\\x222.0.2\\x22,\\x22m\\x22:\\x22user.isFollow\\x22,\\x22ver\\x22:\\x224\\x22,\\x22channel\\x22:\\x22OT_bdhn\\x22,\\x22p\\x22:{\\x22roomId\\x22:\\x221743331\\x22}} | 97 | - | Mozilla/5.0 (Linux; Android 4.4.4; C630Lw Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 QMTV/2.0.2 CHANNEL/OT_bdhn | - | 61.136.167.55 | 0.016 | 0.022 | -", "@version"=>"1", "@timestamp"=>"2016-08-03T08:52:11.442Z", "path"=>"/app/local/log/nginx/ckl_access.log", "host"=>"0.0.0.0", "type"=>"nginx.access", "host_name"=>"ckl_access_front-web15"}, "host_name"], "[type]"=>[{"message"=>"116.209.58.13 | 03/Aug/2016:16:01:01 +0800 | POST /api/v1 HTTP/1.1 | 200 | 69 | {\\x22os\\x22:\\x221\\x22,\\x22v\\x22:\\x222.0.2\\x22,\\x22m\\x22:\\x22user.isFollow\\x22,\\x22ver\\x22:\\x224\\x22,\\x22channel\\x22:\\x22OT_bdhn\\x22,\\x22p\\x22:{\\x22roomId\\x22:\\x221743331\\x22}} | 97 | - | Mozilla/5.0 (Linux; Android 4.4.4; C630Lw Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36 QMTV/2.0.2 CHANNEL/OT_bdhn | - | 61.136.167.55 | 0.016 | 0.022 | -", "@version"=>"1", "@timestamp"=>"2016-08-03T08:52:11.442Z", "path"=>"/app/local/log/nginx/ckl_access.log", "host"=>"0.0.0.0", "type"=>"nginx.access", "host_name"=>"ckl_access_front-web15"}, "type"]}>>, :identity=>"default", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.3.0/lib/redis/client.rb:121:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.3.0/lib/redis.rb:1070:in `rpush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.3.0/lib/redis.rb:58:in `synchronize'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/monitor.rb:211:in `mon_synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.3.0/lib/redis.rb:58:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.3.0/lib/redis.rb:1069:in `rpush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-redis-2.0.5/lib/logstash/outputs/redis.rb:246:in `send_to_redis'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-json-2.1.4/lib/logstash/codecs/json.rb:42:in `encode'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-redis-2.0.5/lib/logstash/outputs/redis.rb:152:in `receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/outputs/base.rb:83:in `multi_receive'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/outputs/base.rb:83:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/output_delegator.rb:114:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:301:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:301:in `output_batch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:232:in `worker_loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:warn}




发现如下:
1
commanderror oom command not allowed when used memory




定位为redis问题
修改redis 配置文件,去掉max-memeory选项即可

错误2:elasticsearch` is obsolete and is no longer available
1
{:timestamp=>"2016-08-03T17:21:26.451000+0800", :message=>"Pipeline aborted due to error", :exception=>#<LogStash::ConfigurationError: The setting `host` in plugin `elasticsearch` is obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask.>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/config/mixin.rb:87:in `config_init'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/config/mixin.rb:71:in `config_init'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/outputs/base.rb:63:in `initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/output_delegator.rb:74:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:181:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:181:in `start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/agent.rb:465:in `start_pipeline'"], :level=>:error}



原配置文件:
1
2
3
4
5
6
7
8
9
10
    elasticsearch {
                hosts => ["10.11.11.12"]
                protocol => "http"
                index => "logstash-%{type}-%{+YYYY.MM.dd}"
                document_type => "%{type}"
                workers => 5
                flush_size => 3840
                idle_flush_time => 10
                template_overwrite => true
                }



                               
解决:
新的配置文件:
1
2
3
4
5
6
7
8
9
  elasticsearch {
                hosts => "10.11.11.12"
                index => "logstash-%{type}-%{+YYYY.MM.dd}"
                document_type => "%{type}"
                workers => 5
                flush_size => 3840
                idle_flush_time => 10
                template_overwrite => true
           }



错误3: Permission denied
1
2
3
4
5
Errno::EACCES: Permission denied - /tmp/logstash-log/log-2016.08.03.log
            initialize at org/jruby/RubyFile.java:370
                   new at org/jruby/RubyIO.java:853
                  open at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-2.2.5/lib/logstash/outputs/file.rb:264
           write_event at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-2.2.5/lib/logstash/outputs/file.rb:162




logstash 日志目录修改:
1
chown -R logstash.logstash logstash-log/



运维网声明 1、欢迎大家加入本站运维交流群:群②:261659950 群⑤:202807635 群⑦870801961 群⑧679858003
2、本站所有主题由该帖子作者发表,该帖子作者与运维网享有帖子相关版权
3、所有作品的著作权均归原作者享有,请您和我们一样尊重他人的著作权等合法权益。如果您对作品感到满意,请购买正版
4、禁止制作、复制、发布和传播具有反动、淫秽、色情、暴力、凶杀等内容的信息,一经发现立即删除。若您因此触犯法律,一切后果自负,我们对此不承担任何责任
5、所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其内容的准确性、可靠性、正当性、安全性、合法性等负责,亦不承担任何法律责任
6、所有作品仅供您个人学习、研究或欣赏,不得用于商业或者其他用途,否则,一切后果均由您自己承担,我们对此不承担任何法律责任
7、如涉及侵犯版权等问题,请您及时通知我们,我们将立即采取措施予以解决
8、联系人Email:admin@iyunv.com 网址:www.yunweiku.com

所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其承担任何法律责任,如涉及侵犯版权等问题,请您及时通知我们,我们将立即处理,联系人Email:kefu@iyunv.com,QQ:1061981298 本贴地址:https://www.yunweiku.com/thread-252531-1-1.html 上篇帖子: Elasticsearch RESTful API请求详解 下篇帖子: ELK安装部署
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

扫码加入运维网微信交流群X

扫码加入运维网微信交流群

扫描二维码加入运维网微信交流群,最新一手资源尽在官方微信交流群!快快加入我们吧...

扫描微信二维码查看详情

客服E-mail:kefu@iyunv.com 客服QQ:1061981298


QQ群⑦:运维网交流群⑦ QQ群⑧:运维网交流群⑧ k8s群:运维网kubernetes交流群


提醒:禁止发布任何违反国家法律、法规的言论与图片等内容;本站内容均来自个人观点与网络等信息,非本站认同之观点.


本站大部分资源是网友从网上搜集分享而来,其版权均归原作者及其网站所有,我们尊重他人的合法权益,如有内容侵犯您的合法权益,请及时与我们联系进行核实删除!



合作伙伴: 青云cloud

快速回复 返回顶部 返回列表