ElasticSearch配置文档(v1.7.2)

gteric

ElasticSearch
安装java环境

• sudo apt-get update
  
• java -version
  
• sudo apt-get install default-jre
  
• sudo apt-get install default-jdk
  
• （或者： yum -y install java-1.7.0-openjdk* ）
  

设置JAVA_HOME

• sudo update-alternatives --config java（返回YOUR_PATH）
  
• sudo nano /etc/environment
  
• JAVA_HOME="YOUR_PATH"
  
• source /etc/environment
  
• echo $JAVA_HOME
  

安装ElasticSearch（以1.7.2为例）

• wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.deb(yum 下载rpm包)
  
• sudo dpkg -i elasticsearch-1.7.2.deb  (sudo  rpm -ivh *.rpm)
  
• sudo update-rc.d elasticsearch defaults   (chkconfig –-add elasticsearch     ;     chkconfig elasticsearch on )
  

启动ElasticSearch

• sudo service elasticsearch start
  
• elasticsearch 结构目录位置： /var/lib/elasticsearch
  

配置ElasticSearch

• 配置文件地址：/etc/elasticsearch/elasticsearch.yml
  
• 配置文件详解：http://www.cnblogs.com/sunxucool/p/3799190.html
  

配置防火墙

• sudo ufw disable
  
• sudo ufw allow from 192.168.1.141
  
• sudo ufw enable
  

　　或者防火墙
 
#/usr/sbin/iptables -I INPUT -p tcp --dport 80 -j ACCEPT
#/usr/sbin/iptables -I INPUT -p tcp --dport 22 -j ACCEPT
#/usr/sbin/iptables -I INPUT -p tcp --dport 3306 -j ACCEPT
#/usr/sbin/iptables -I INPUT -p tcp --dport 8080 -j ACCEPT
 
/usr/sbin/iptables -I INPUT -p tcp --dport 9200 -j DROP
/usr/sbin/iptables -I INPUT -s 10.44.136.154 -p tcp --dport 9200 -j ACCEPT
/usr/sbin/iptables -I INPUT -s 某个ip -p tcp --dport 9200 -j ACCEPT
/usr/sbin/iptables -I INPUT -s 某个ip -p tcp --dport 9200 -j ACCEPT
 
 
/usr/sbin/service iptables save


创建快照

• elasticsearch.yml添加： path.repo: ["/mount/backups", "/mount/longterm_backups"]
  
• 创建目录： /mount/backups/my_backup,  /mount/longterm_backups
  
• 添加权限： sudo chmod -R 777 /mount
  
• 重启elasticsearch服务： sudo service elasticsearch restart
  
• 创建仓库，并指定仓库类型：
  

curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '
{ 
"type": "fs", 
"settings": { 
                "location": "/mount/backups/my_backup",
                "compress": true 
}
}'

• 创建第一个快照：curl -XPUT http://localhost:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true
  

• 查看所有快照： curl -XGET http://localhost:9200/_snapshot/my_backup/_all
  
• 删除一个快照： curl -XDELETE http://localhost:9200/_snapshot/my_backup/snapshot_1
  
• 使用快照： 
  

 

curl -XPOST http://localhost:9200/_snapshot/my_backup/snapshot_1/_restore -d '
{
"indices": "index_1,index_2", 
"ignore_unavailable": "true",
"include_global_state": false, 
"rename_pattern": "index_(.+)", 
"rename_replacement": "restored_index_$1"
}'

配置ik分词

• 版本参考
  

IK version
ES version

master
2.1.0 -> master


1.6.0
2.1.0


1.5.0
2.0.0


1.4.1
1.7.2


1.4.0
1.6.0


1.3.0
1.5.0


1.2.9
1.4.0


1.2.8
1.3.2


1.2.7
1.2.1


1.2.6
1.0.0


1.2.5
0.90.2


1.2.3
0.90.2


1.2.0
0.90.0


1.1.3
0.20.2


1.1.2
0.19.x


1.0.0
0.16.2 -> 0.19.0

• 克隆ik库： https://github.com/medcl/elasticsearch-analysis-ik.git
  

• checkout对应tag：  git checkout tags/v1.4.1 -b v1.4.1
  

• 安装maven： (略)
  

• 编译jar包： sudo mvn package
  

• copy ik文件夹到/etc/elasticsearch/: sudo cp config/ik /etc/elasticsearch/ 
  

• copy jar到/usr/share/elasticsearch/lib/: sudo cp target/*.jar /usr/share/elasticsearch/lib
  

• 配置elasticsearch.yml
  

index:
  analysis:
    analyzer:
      ik:
        alias: [ik_analyzer]
        type: org.elasticsearch.index.analysis.IkAnalyzerProvider
      ik_max_word:
        type: ik
        use_smart: false
      ik_smart:
        type: ik
        use_smart: true
index.analysis.analyzer.default.type : "ik"

• 创建index:  
  

curl -XPUT http://localhost:9200/index

• 创建mapping
  

curl -XPOST http://localhost:9200/index/fulltext/_mapping -d'
{
    "fulltext": {
             "_all": {
            "analyzer": "ik_max_word",
            "search_analyzer": "ik_max_word",
            "term_vector": "no",
            "store": "false"
        },
        "properties": {
            "content": {
                "type": "string",
                "store": "no",
                "term_vector": "with_positions_offsets",
                "analyzer": "ik_max_word",
                "search_analyzer": "ik_max_word",
                "include_in_all": "true",
                "boost": 8
            }
        }
    }
}'
  mapping:
 
curl -XPUT localhost:9200/feiliwu -d '{
"mappings":{
     "product":{
            "_timestamp":{
                    "enables": true
             }
      }
}
 
}'

• 录入数据
  

curl -XPOST http://localhost:9200/index/fulltext/1 -d'
{"content":"美国留给伊拉克的是个烂摊子吗"}
'
curl -XPOST http://localhost:9200/index/fulltext/2 -d'
{"content":"公安部：各地校车将享最高路权"}
'
curl -XPOST http://localhost:9200/index/fulltext/3 -d'
{"content":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}
'
curl -XPOST http://localhost:9200/index/fulltext/4 -d'
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}
'

• 查询
  

curl -XPOST http://localhost:9200/index/fulltext/_search  -d'
{
    "query" : { "term" : { "content" : "中国" }},
    "highlight" : {
        "pre_tags" : ["<tag1>", "<tag2>"],
        "post_tags" : ["</tag1>", "</tag2>"],
        "fields" : {
            "content" : {}
        }
    }
}
'
 


安装Marvel

• run sudo ./bin/plugin -i elasticsearch/marvel/latest
  
• 浏览器访问：http://localhost:9200/_plugin/marvel/
  

参考资料

• How To Install Java on Ubuntu with Apt-Get
  
• How To Install and Configure Elasticsearch on Ubuntu 14.04
  
• Setting up a Secure Single Node Elasticsearch server behind Nginx：
  
• Elasticsearch 权威指南（中文版）
  
• UFW防火墙简单设置
  
• Ufw使用指南
  
• Snapshot And Restore
  
• elasticsearch-analysis-ik
  
• Linux下安装maven
  
• ElasticSearch入门笔记
  
• ElasticSearch中文社区
  

      Logstash
下载安装

• wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.4-1_all.deb
  
• sudo dpkg -i logstash_1.5.4-1_all.deb
  
• sudo update-rc.d logstash defaults   
  

配置jdk

• sudo nano  /etc/profile
  

 

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar
export PATH=$PATH:$JAVA_HOME/bin
export LOGSTASH_HOME=/opt/logstash
export LOGSTASH=$LOGSTASH_HOME/bin
export PATH=$LOGSTASH:$PATH

 

• source /etc/profile
  

  配置logstash

• sudo nano /etc/logstash/conf.d/logstash-simple.conf
  

 
 
 
input {
 
  tcp {
    port => 5000
    type => syslog
  }
  udp {
    port => 5000
    type => syslog
  }
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
output {
    elasticsearch {
        protocol => "http"
        host => "localhost:9200"
    }
    stdout { }
}
 
 
 
 


启动logstash

• sudo service logstash status
  
• sudo service logstash start
  

查看log

• tail -f /var/log/logstash/logstash.log
  

参考资料

• ELKstack 中文指南
  
• logstash&node.js&elk&日志分析
  
• logstash download
  

Kibana
建立使用kibana的分组

• sudo groupadd -g 999 kibana 
  
• sudo useradd -u 999 -g 999 kibana
  
• 
  If those commands fail because the999GID or UID already exist, replace the number with IDs that are free
  

下载安装

• cd ~ 
  
• wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz
  
  
• tar xvf kibana-*.tar.gz
  

配置

• sudo nano ~/kibana-4*/config/kibana.yml（server.host: "localhost"）
  
• sudo mkdir -p /opt/kibana
  
• sudo cp -R ~/kibana-4*/* /opt/kibana/
  
  
• sudo chown -R kibana: /opt/kibana
  

run as a service

• cd /etc/init.d && sudo curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-init
  
• cd /etc/default && sudo curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-default
  
• sudo chmod +x /etc/init.d/kibana
  
• sudo update-rc.d kibana defaults 96 9
  
  
• sudo service kibana start
  
• visit localhost:5601
  

Install Nginx

• sudo apt-get install nginx apache2-utils(sudo yum install nginx httpd)
  
• sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin 
  
  
• sudo vi /etc/nginx/sites-available/default
  

 
 
 

server {
listen 80;
server_name localhost;
auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/htpasswd.users;
location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}

• sudo service nginx restart
  

Configure An Index Pattern

• 
  Go to Settings → Advanced.
  
• 
  Edit the metaFields and add "_timestamp". Hit save.
  
• 
  Now go back to Settings → Indices 
  

• create 
  

参考资料

• kibana download
  
• How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu 14.04
  
• configure an index pattern