LVS+Keepalived

231w

实验环境：
LVS01:192.168.0.149   #（外网）

          10.0.0.13   #（内网）

LVS02:192.168.0.150
           10.0.0.14

web01:10.0.0.15          #（web环境自行搭建）

web02:10.0.0.16          #（web环境自行搭建）
VIP:192.168.0.145

安装准备：
[iyunv@LVS01 ~]# cat /etc/redhat-release
CentOS release 6.7 (Final)
[iyunv@LVS01 ~]# uname -r
2.6.32-573.el6.x86_64
[iyunv@LVS01 ~]# lsmod|grep ip_vs    #查看是否安装LVS，或启用LVS
[iyunv@LVS01 ~]# ls -ld /usr/src/kernels/2.6.32-573.12.1.el6.x86_64
drwxr-xr-x 22 root root 4096 Dec 18 00:12 /usr/src/kernels/2.6.32-573.12.1.el6.x86_64
[iyunv@LVS01 ~]# ln -s /usr/src/kernels/2.6.32-573.12.1.el6.x86_64/  /usr/src/linux

[iyunv@LVS01 ~]# grep forward /etc/sysctl.conf    #开启内核转发
# Controls IP packet forwarding
net.ipv4.ip_forward = 1

1.安装LVS：
[iyunv@LVS01 ~]# wget http://www.linuxvirtualserver.or ... ipvsadm-1.26.tar.gz
[iyunv@LVS01 ~]# tar xf ipvsadm-1.26.tar.gz
[iyunv@LVS01 ~]# cd ipvsadm-1.26
[iyunv@LVS01 ipvsadm-1.26]# make
[iyunv@LVS01 ipvsadm-1.26]# make install
[iyunv@LVS01 ipvsadm-1.26]# lsmod|grep ip_vs
[iyunv@LVS01 ipvsadm-1.26]# which ipvsadm
/sbin/ipvsadm
[iyunv@LVS01 ipvsadm-1.26]# cd ..
[iyunv@LVS01 ~]# ipvsadm --version
ipvsadm v1.26 2008/5/15 (compiled with popt and IPVS v1.2.1)
[iyunv@LVS01 ~]# lsmod|grep ip_vs
ip_vs                 125694  0
libcrc32c               1246  1 ip_vs
ipv6                  334932  141 ip_vs
#出现以上三行结果，表示安装成功


2.安装Keepalived：
[iyunv@LVS01 tools]# ln -s /usr/src/kernels/2.6.32-573.12.1.el6.x86_64/  /usr/src/linux
[iyunv@LVS01 tools]# tar xf keepalived-1.2.7.tar.gz
[iyunv@LVS01 tools]# cd keepalived-1.2.7
[iyunv@LVS01 keepalived-1.2.7]# ./configure
............................................
Keepalived version       : 1.2.7
Compiler                 : gcc
Compiler flags           : -g -O2
Extra Lib                : -lpopt -lssl -lcrypto  -lnl
Use IPVS Framework       : Yes
IPVS sync daemon support : Yes
IPVS use libnl           : Yes
Use VRRP Framework       : Yes
Use VRRP VMAC            : Yes
SNMP support             : No
Use Debug flags          : No
#以上最后结果中，最好最少有3个Yes,如下：
Use IPVS Framework       : Yes    #ipvs框架
IPVS sync daemon support : Yes    #ipvs同步支持
Use VRRP Framework       : Yes    #VRRP框架
[iyunv@LVS01 keepalived-1.2.7]# make
[iyunv@LVS01 keepalived-1.2.7]# make install
[iyunv@LVS01 keepalived-1.2.7]# cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/
[iyunv@LVS01 keepalived-1.2.7]# cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/
[iyunv@LVS01 keepalived-1.2.7]#  mkdir /etc/keepalived -p
[iyunv@LVS01 keepalived-1.2.7]# cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
[iyunv@LVS01 keepalived-1.2.7]# cp /usr/local/sbin/keepalived /usr/sbin/
[iyunv@LVS01 keepalived-1.2.7]# /etc/init.d/keepalived start
Starting keepalived:                               [  OK  ]
[iyunv@LVS01 keepalived-1.2.7]# ps -ef|grep keepalived
root     14563     1  0 06:57 ?        00:00:00 keepalived -D
root     14565 14563  0 06:57 ?        00:00:00 keepalived -D
root     14566 14563  0 06:57 ?        00:00:00 keepalived -D
root     14570 13038  0 06:57 pts/0    00:00:00 grep keepalived
[iyunv@LVS01 keepalived-1.2.7]# /etc/init.d/keepalived stop
Stopping keepalived:                               [  OK  ]
[iyunv@LVS01 keepalived-1.2.7]# ps -ef|grep keepalived|grep -v  grep

3.配置Keepalived：
主(LVS01)：
[iyunv@LVS01 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     1729294227@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.0.200
   smtp_connect_timeout 30
   router_id LVS_1
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.0.145/24
    }
}

virtual_server 192.168.0.145 80 {
    delay_loop 6  
    lb_algo wrr                       #负载均衡算法
    lb_kind DR                        #负载均衡模式
    nat_mask 255.255.255.0   #子网掩码
    persistence_timeout 300  #会话保持
    protocol TCP                     #协议

    real_server 10.0.0.15 80 {
            weight 1
            TCP_CHECK {
            connect_timeout 8
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
            }
        }
    real_server 10.0.0.16 80 {
            weight 1
            TCP_CHECK {
            connect_timeout 8
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
            }
        }
    }
}

备(LVS02)：
[iyunv@LVS02 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     1729294227@qq.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.0.200
   smtp_connect_timeout 30
   router_id LVS_2
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.0.145/24
    }
}

virtual_server 192.168.0.145 80 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    nat_mask 255.255.255.0
    persistence_timeout 300
    protocol TCP

    real_server 10.0.0.15 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 8
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
            }
        }
    real_server 10.0.0.16 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 8
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
            }
        }
    }
}


4.手工在RS绑定VIP（两台web机器上）：
web01:

[iyunv@web01 ~]# curl -i localhost
HTTP/1.1 200 OK
Server: nginx/1.6.3
Date: Thu, 26 Nov 2015 12:09:12 GMT
Content-Type: text/html
Content-Length: 24
Last-Modified: Thu, 26 Nov 2015 09:19:01 GMT
Connection: keep-alive
ETag: "5656ce85-18"
Accept-Ranges: bytes

192.168.0.151:test1-web
[iyunv@web01 ~]# ifconfig lo:0 192.168.0.145/32 up
[iyunv@web01 ~]# ifconfig lo:0
lo:0      Link encap:Local Loopback  
          inet addr:192.168.0.145  Mask:0.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1

web02:
[iyunv@web02 ~]# curl -i localhost
HTTP/1.1 200 OK
Server: nginx/1.6.3
Date: Sat, 07 Nov 2015 16:18:31 GMT
Content-Type: text/html
Content-Length: 24
Last-Modified: Sat, 07 Nov 2015 13:27:39 GMT
Connection: keep-alive
ETag: "563dfc4b-18"
Accept-Ranges: bytes

192.168.0.160:test2-web
[iyunv@web02 ~]# ifconfig lo:0 192.168.0.145/32 up
[iyunv@web02 ~]# ifconfig lo:0
lo:0      Link encap:Local Loopback  
          inet addr:192.168.0.145  Mask:0.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
#以上可通过脚本实现

注意，做好以下三点保证IP不冲突：
1.绑定在回环接口上（lo)
2.绑定VIP地址
3.子网掩码是：255.255.255.255

5.手工在RS端(两台web机器上)抑制ARP响应：
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
#以上可通过脚本实现


6.最终测试：
[iyunv@LVS01 ~]# /etc/init.d/keepalived start
[iyunv@LVS01 ~]# ipvsadm -L -n                 
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.145:80 wrr persistent 300
  -> 10.0.0.15:80                 Route   1      0          0         
  -> 10.0.0.16:80                 Route   1      0          0     

[iyunv@LVS01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:a4:26:69 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.149/24 brd 192.168.0.255 scope global eth0
    inet 192.168.0.145/24 scope global secondary eth0
    inet6 fe80::20c:29ff:fea4:2669/64 scope link
       valid_lft forever preferred_lft forever
3: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:a4:26:73 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.13/24 brd 10.0.0.255 scope global eth3
    inet6 fe80::20c:29ff:fea4:2673/64 scope link
       valid_lft forever preferred_lft forever

[iyunv@LVS02 ~]# /etc/init.d/keepalived start

[iyunv@LVS02 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.145:80 wrr persistent 300
  -> 10.0.0.15:80                 Route   1      0          0         
  -> 10.0.0.16:80                 Route   1      0          0         

[iyunv@LVS02 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:6a:27:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.150/24 brd 192.168.0.255 scope global eth0
    inet6 fe80::20c:29ff:fe6a:27b4/64 scope link
       valid_lft forever preferred_lft forever
3: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:6a:27:be brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.14/24 brd 10.0.0.255 scope global eth3
    inet6 fe80::20c:29ff:fe6a:27be/64 scope link
       valid_lft forever preferred_lft forever

7.模拟“故障”测试高可用：
[iyunv@LVS01 ~]# /etc/init.d/keepalived stop
Stopping keepalived:                            [  OK  ]
[iyunv@LVS01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:a4:26:69 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.149/24 brd 192.168.0.255 scope global eth0
    inet6 fe80::20c:29ff:fea4:2669/64 scope link
       valid_lft forever preferred_lft forever
3: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:a4:26:73 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.13/24 brd 10.0.0.255 scope global eth3
    inet6 fe80::20c:29ff:fea4:2673/64 scope link
       valid_lft forever preferred_lft forever

[iyunv@LVS02 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:6a:27:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.150/24 brd 192.168.0.255 scope global eth0
    inet 192.168.0.145/24 scope global secondary eth0
    inet6 fe80::20c:29ff:fe6a:27b4/64 scope link
       valid_lft forever preferred_lft forever
3: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:6a:27:be brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.14/24 brd 10.0.0.255 scope global eth3
    inet6 fe80::20c:29ff:fe6a:27be/64 scope link
       valid_lft forever preferred_lft forever
嘿嘿，VIP飘移成功，看web界面吧！