在linux下搭建HA和LB集群（lvs&heartbeat群集）
案例应用拓扑图：案例应用实现详细步骤如下：
一．DNS服务器配置
1.1 在real-server-1上的DNS服务器配置：
# yum install bind bind-chroot caching-nameserver –y
# cd /var/named/chroot/etc/
# cp -p named.caching-nameserver.conf named.conf
# vim named.conf
15 listen-on port 53 { any; };
27 allow-query { any; };
28 allow-query-cache { any; };
37 match-clients { any; };
38 match-destinations { any; };
# vim named.rfc1912.zones
20 zone "japan.com" IN {
21 type master;
22 file "japan.com.db";
23 allow-update { none; };
24 };
37 zone "2.168.192.in-addr.arpa" IN {
38 type master;
39 file "192.168.2.db";
40 allow-update { none; };
41 };
# cd ../var/named/
# cp -p localhost.zone japan.com.db
# cp -p named.local 192.168.2.db
# vim japan.com.db
1 $TTL 86400
2 @ IN SOAns.japan.com. root (
3 43 ; serial (d. adams )
4 3H ; refresh
5 15M ; retry
6 1W ; expiry
7 1D ) ; minimum
8
9 @ IN NS ns.japan.com.
10 ns IN A 192.168.2.131
11 www IN A 192.168.2.133
12 director1 IN A 192.168.2.134
13 director2 IN A 192.168.2.135
# vim 192.168.2.db
1 $TTL 86400
2 @ IN SOA localhost. root.localhost.(
3 1997022700 ; Serial
4 28800 ; Refresh
5 14400 ; Retry
6 3600000 ; Expire
7 86400 ) ; Minimum
8 IN NS localhost.
9 133 IN PTR www.japan.com.
10 134 IN PTR director1.japan.com.
11 135 IN PTR director2.japan.com.
# service named restart
Stopping named:
Starting named:
# rndc reload
server reload successful
1.2 在real-server-2上的DNS服务器配置：
# yum install bind bind-chroot caching-nameserver –y
# cd /var/named/chroot/etc/
# cp -p named.caching-nameserver.conf named.conf
# vim named.conf
15 listen-on port 53 { any; };
27 allow-query { any; };
28 allow-query-cache { any; };
37 match-clients { any; };
38 match-destinations { any; };
# vim named.rfc1912.zones
20 zone "japan.com" IN {
21 type master;
22 file "japan.com.db";
23 allow-update { none; };
24 };
37 zone "2.168.192.in-addr.arpa" IN {
38 type master;
39 file "192.168.2.db";
40 allow-update { none; };
41 };
# cd ../var/named/
# cp -p localhost.zone japan.com.db
# cp -p named.local 192.168.2.db
# vim japan.com.db
1 $TTL 86400
2 @ IN SOAns.japan.com. root (
3 43 ; serial (d. adams )
4 3H ; refresh
5 15M ; retry
6 1W ; expiry
7 1D ) ; minimum
8
9 @ IN NS ns.japan.com.
10 ns IN A 192.168.2.131
11 www IN A 192.168.2.133
12 director1 IN A 192.168.2.134
13 director2 IN A 192.168.2.135
# vim 192.168.2.db
1 $TTL 86400
2 @ IN SOA localhost. root.localhost.(
3 1997022700 ; Serial
4 28800 ; Refresh
5 14400 ; Retry
6 3600000 ; Expire
7 86400 ) ; Minimum
8 IN NS localhost.
9 133 IN PTR www.japan.com.
10 134 IN PTR director1.japan.com.
11 135 IN PTR director2.japan.com.
# service named restart
Stopping named:
Starting named:
# rndc reload
server reload successful
二 . Director-1服务器配置
2.1Director-1服务器ip地址配置
# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=director1.japan.com
# setup
# service network restart
Shutting down interface eth0:
Shutting down interface eth1:
Shutting down loopback interface:
Bringing up loopback interface:
Bringing up interface eth0:
Bringing up interface eth1:
# ifconfig eth0
eth0 Link encap:EthernetHWaddr 00:0C:29:66:E1:DA
inet addr:192.168.2.134Bcast:192.168.2.143Mask:255.255.255.240
# ifconfig eth0:0
eth0:0 Link encap:EthernetHWaddr 00:0C:29:66:E1:DA
inet addr:192.168.2.133Bcast:192.168.2.133Mask:255.255.255.255
# ifconfig eth1
eth1 Link encap:EthernetHWaddr 00:0C:29:66:E1:E4
inet addr:192.168.0.1Bcast:192.168.0.255Mask:255.255.255.0
2.2为director-1添加路由
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.2.128 0.0.0.0 255.255.255.240 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
0.0.0.0 192.168.2.133 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 192.168.2.142 0.0.0.0 UG 0 0 0 eth0
# route add -host 192.168.2.133 dev eth0:0
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.2.133 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
192.168.2.128 0.0.0.0 255.255.255.240 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
0.0.0.0 192.168.2.133 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 192.168.2.142 0.0.0.0 UG 0 0 0 eth0
2.3配置本地yum服务器：
# vim /etc/yum.repos.d/server.repo
name=Red Hat Enterprise Linux server
baseurl=file:///mnt/cdrom/Server/
enabled=1
gpgcheck=1
gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release
name=Red Hat Enterprise Linux cluster
baseurl=file:///mnt/cdrom/Cluster/
enabled=1
gpgcheck=1
gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release
#mkdir /mnt/cdrom
# mount /dev/cdrom /mnt/cdrom/
mount: block device /dev/cdrom is write-protected, mounting read-only
#yum list all
2.4安装配置dircetor-1服务器：
# yum install -y ipvsadm
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
# ipvsadm -A -t 192.168.2.133:80 -s rr
# ipvsadm -a -t 192.168.2.133:80 -r 192.168.2.131 -g
# ipvsadm -a -t 192.168.2.133:80 -r 192.168.2.132 -g
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.133:80 rr
-> 192.168.2.132:80 Route 1 0 0
-> 192.168.2.131:80 Route 1 0 0
# service ipvsadm save
Saving IPVS table to /etc/sysconfig/ipvsadm:
# service ipvsadm restart
Clearing the current IPVS table:
Applying IPVS configuration:
# service ipvsadm stop
Clearing the current IPVS table:
三 . Director-2服务器配置
3.1Director服务器ip地址配置
# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=director2.japan.com
# setup
# service network restart
Shutting down interface eth0:
Shutting down interface eth1:
Shutting down loopback interface:
Bringing up loopback interface:
Bringing up interface eth0:
Bringing up interface eth1:
# ifconfig eth0
eth0 Link encap:EthernetHWaddr 00:0C:29:79:F8:F7
inet addr:192.168.2.135Bcast:192.168.2.143Mask:255.255.255.240
# ifconfig eth0:0
eth0:0 Link encap:EthernetHWaddr 00:0C:29:79:F8:F7
inet addr:192.168.2.133Bcast:192.168.2.133Mask:255.255.255.255
# ifconfig eth1
eth1 Link encap:EthernetHWaddr 00:0C:29:79:F8:01
inet addr:192.168.0.2Bcast:192.168.0.255Mask:255.255.255.0
3．2为director-2添加路由
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.2.128 0.0.0.0 255.255.255.240 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
0.0.0.0 192.168.2.133 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 192.168.2.142 0.0.0.0 UG 0 0 0 eth0
# route add -host 192.168.2.133 dev eth0:0
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.2.133 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
192.168.2.128 0.0.0.0 255.255.255.240 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
0.0.0.0 192.168.2.133 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 192.168.2.142 0.0.0.0 UG 0 0 0 eth0
3.3配置本地yum服务器：
# vim /etc/yum.repos.d/server.repo
name=Red Hat Enterprise Linux server
baseurl=file:///mnt/cdrom/Server/
enabled=1
gpgcheck=1
gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release
name=Red Hat Enterprise Linux cluster
baseurl=file:///mnt/cdrom/Cluster/
enabled=1
gpgcheck=1
gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release
#mkdir /mnt/cdrom
# mount /dev/cdrom /mnt/cdrom/
mount: block device /dev/cdrom is write-protected, mounting read-only
#yum list all
3.4安装配置dircetor-2服务器：
# yum install -y ipvsadm
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
# ipvsadm -A -t 192.168.2.133:80 -s rr
# ipvsadm -a -t 192.168.2.133:80 -r 192.168.2.131 -g
# ipvsadm -a -t 192.168.2.133:80 -r 192.168.2.132 -g
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.133:80 rr
-> 192.168.2.132:80 Route 1 0 0
-> 192.168.2.131:80 Route 1 0 0
# service ipvsadm save
Saving IPVS table to /etc/sysconfig/ipvsadm:
# service ipvsadm restart
Clearing the current IPVS table:
Applying IPVS configuration:
四．配置real-server-1的web服务器：
4.1 解决arp问题:
# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=server1.japan.com
# echo "net.ipv4.conf.all.arp_announce = 2" >> /etc/sysctl.conf
# echo "net.ipv4.conf.lo.arp_announce = 2" >> /etc/sysctl.conf
# echo "net.ipv4.conf.all.arp_ignore = 1" >> /etc/sysctl.conf
# echo "net.ipv4.conf.lo.arp_ignore = 1" >> /etc/sysctl.conf
#sysctl -p
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
4.2配置ip地址和路由
# setup
# service network restart
Shutting down interface eth0:
Shutting down loopback interface:
Bringing up loopback interface:
Bringing up interface eth0:
# ifconfig eth0
eth0 Link encap:EthernetHWaddr 00:0C:29:1B:F1:BA
inet addr:192.168.2.131Bcast:192.168.2.143Mask:255.255.255.240
# ifconfig lo:0
lo:0 Link encap:Local Loopback
inet addr:192.168.2.133Mask:255.255.255.255
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.2.128 0.0.0.0 255.255.255.240 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 192.168.2.142 0.0.0.0 UG 0 0 0 eth0
# route add -host 192.168.2.133 dev lo:0
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.2.133 0.0.0.0 255.255.255.255 UH 0 0 0 lo
192.168.2.128 0.0.0.0 255.255.255.240 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 192.168.2.142 0.0.0.0 UG 0 0 0 eth0
4.3配置Real-server-1的Web服务器:
#mkdir /mnt/cdrom
# mount /dev/cdrom /mnt/cdrom/
mount: block device /dev/cdrom is write-protected, mounting read-only
# rpm -ivh /mnt/cdrom/Server/httpd-2.2.3-31.el5.i386.rpm
warning: /mnt/cdrom/Server/httpd-2.2.3-31.el5.i386.rpm: Header V3 DSA signature: NOKEY, key ID 37017186
Preparing... ####################################
1:httpd ###########################################
# echo "web1-- real-server-1" > /var/www/html/index.html
# service httpd start
Starting httpd: httpd: apr_sockaddr_info_get() failed for r1.junjie.com
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
4.4客户端配置信息
4.5 客户端访问real-server-1的web服务：（桥接）
五．配置real-server2的web服务器：
5.1解决arp问题:
# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=server2.japan.com
# echo "net.ipv4.conf.all.arp_announce = 2" >> /etc/sysctl.conf
# echo "net.ipv4.conf.lo.arp_announce = 2" >> /etc/sysctl.conf
# echo "net.ipv4.conf.all.arp_ignore = 1" >> /etc/sysctl.conf
# echo "net.ipv4.conf.lo.arp_ignore = 1" >> /etc/sysctl.conf
# sysctl -p
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
5.2配置ip地址和路由
# setup
# service network restart
Shutting down interface eth0:
Shutting down loopback interface:
Bringing up loopback interface:
Bringing up interface eth0:
# ifconfig eth0
eth0 Link encap:EthernetHWaddr 00:0C:29:AE:83:D1
inet addr:192.168.2.132Bcast:192.168.2.143Mask:255.255.255.240
# ifconfig lo:0
lo:0 Link encap:Local Loopback
inet addr:192.168.2.133Mask:255.255.255.255
UP LOOPBACK RUNNINGMTU:16436Metric:1
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.2.128 0.0.0.0 255.255.255.240 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 192.168.2.142 0.0.0.0 UG 0 0 0 eth0
# route add -host 192.168.2.133 dev lo:0
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.2.133 0.0.0.0 255.255.255.255 UH 0 0 0 lo
192.168.2.128 0.0.0.0 255.255.255.240 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 192.168.2.142 0.0.0.0 UG 0 0 0 eth0
5.3配置Real-server-2的Web服务器：
#mkdir /mnt/cdrom
# mount /dev/cdrom /mnt/cdrom/
mount: block device /dev/cdrom is write-protected, mounting read-only
# rpm -ivh /mnt/cdrom/Server/httpd-2.2.3-31.el5.i386.rpm
warning: /mnt/cdrom/Server/httpd-2.2.3-31.el5.i386.rpm: Header V3 DSA signature: NOKEY, key ID 37017186
Preparing... ####################################
1:httpd ###########################################
#echo "web2-- real-server-2" > /var/www/html/index.html l
# service httpd start
Starting httpd: httpd: apr_sockaddr_info_get() failed for r2.junjie.vom
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
5.4客户端访问real-server-2的web服务：（桥接）
六．客户端测试lvs-DR模型：
6.1测试1
关闭director-1的ipvsadm服务，确保以下信息
# service ipvsadm stop
Clearing the current IPVS table:
# service ipvsadm status
ipvsadm is stopped
开启director-2的ipvsadm服务，确保以下信息
# service ipvsadm restart
Clearing the current IPVS table:
Applying IPVS configuration:
# service ipvsadm status
ipvsadm dead but subsys locked
客户端访问director-2的群集服务服务：（网卡使用桥接模式）
客户端开始不断刷新，发现web2和web1交替出现，比率为1:1，说明依次轮询模式为RR
在director-2上查看信息如下：轮询调度比几乎为1:1；
说明lvs调度方法是用的是RR模式
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.133:80 rr
-> 192.168.2.131:80 Route 1 0 50
-> 192.168.2.132:80 Route 1 0 50
6.2测试2
关闭director-2的ipvsadm服务，确保以下信息
# service ipvsadm stop
Clearing the current IPVS table:
# service ipvsadm status
ipvsadm is stopped
开启director-1的ipvsadm服务，确保以下信息
# service ipvsadm start
Clearing the current IPVS table:
Applying IPVS configuration:
# service ipvsadm restart
Clearing the current IPVS table:
Applying IPVS configuration:
# service ipvsadm status
ipvsadm dead but subsys locked
客户端访问director-1的群集服务服务：（网卡使用桥接模式）
客户端开始不断刷新，发现web2和web1交替出现，比率为1:1，说明依次轮询模式为RR
在director-1上查看信息如下：轮询调度比几乎为1:1；
说明lvs调度方法是用的是RR模式
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.133:80 rr
-> 192.168.2.131:80 Route 1 0 25
-> 192.168.2.132:80 Route 1 0 25
七：heartbeat服务搭建
7．1首先停止ipvsadm服务：
# service ipvsadm stop
Clearing the current IPVS table:
# service ipvsadm status
ipvsadm is stopped
# service ipvsadm stop
Clearing the current IPVS table:
# service ipvsadm status
ipvsadm is stopped
7.2 下载heartbeat有关的软件包，这里我放在/root/HA目录下了，并安装
# cd /root/HA/
# ls
heartbeat-2.1.4-9.el5.i386.rpm
heartbeat-pils-2.1.4-10.el5.i386.rpm
heartbeat-stonith-2.1.4-10.el5.i386.rpm
libnet-1.1.4-3.el5.i386.rpm
perl-MailTools-1.77-1.el5.noarch.rpm
# yum localinstall -y heartbeat-2.1.4-9.el5.i386.rpm heartbeat-pils-2.1.4-10.el5.i386.rpm heartbeat-stonith-2.1.4-10.el5.i386.rpm libnet-1.1.4-3.el5.i386.rpm perl-MailTools-1.77-1.el5.noarch.rpm --nogpgcheck
# cd /root/HA/
# ls
heartbeat-2.1.4-9.el5.i386.rpm
heartbeat-pils-2.1.4-10.el5.i386.rpm
heartbeat-stonith-2.1.4-10.el5.i386.rpm
libnet-1.1.4-3.el5.i386.rpm
perl-MailTools-1.77-1.el5.noarch.rpm
# yum localinstall -y heartbeat-2.1.4-9.el5.i386.rpm heartbeat-pils-2.1.4-10.el5.i386.rpm heartbeat-stonith-2.1.4-10.el5.i386.rpm libnet-1.1.4-3.el5.i386.rpm perl-MailTools-1.77-1.el5.noarch.rpm --nogpgcheck
7.3 配置director-1的heartbeat服务：
# cd /etc/ha.d/
# cp /usr/share/doc/heartbeat-2.1.4/ha.cf ./
# cp /usr/share/doc/heartbeat-2.1.4/authkeys ./
# cp /usr/share/doc/heartbeat-2.1.4/haresources ./
# cp /etc/init.d/ipvsadm resource.d/
# vim ha.cf
#添加以下几行
95 bcast eth1
214 node director1.japan.com
215 node director2.japan.com
# vim haresources
45 director1.japan.com 192.168.2.133/28/eth0/192.168.2.143 ipvsadm
# dd if=/dev/random bs=512 count=1 |openssl md5
0+1 records in
0+1 records out
128 bytes (128 B) copied, 0.000474241 seconds, 270 kB/s
007d644a23b1e0e6361fcce286268582
# vim authkeys
#在末尾添加以下几行：
auth 1
1 md5 007d644a23b1e0e6361fcce286268582
# chmod 600 authkeys
启动heartbeat服务：
# service heartbeat status
heartbeat is stopped. No process
# service heartbeat start
Starting High-Availability services:
2012/04/02_13:34:38 INFO:Running OK
2012/04/02_13:34:38 CRITICAL: Resource 192.168.2.133/27/eth0/192.168.2.143 is active, and should not be!
2012/04/02_13:34:38 CRITICAL: Non-idle resources can affect data integrity!
2012/04/02_13:34:38 info: If you don't know what this means, then get help!
2012/04/02_13:34:38 info: Read the docs and/or source to /usr/share/heartbeat/ResourceManager for more details.
CRITICAL: Resource 192.168.2.133/27/eth0/192.168.2.143 is active, and should not be!
CRITICAL: Non-idle resources can affect data integrity!
info: If you don't know what this means, then get help!
info: Read the docs and/or the source to /usr/share/heartbeat/ResourceManager for more details.
2012/04/02_13:34:38 CRITICAL: Non-idle resources will affect resource takeback!
2012/04/02_13:34:38 CRITICAL: Non-idle resources may affect data integrity!
# service heartbeat status
heartbeat OK is running on director1.japan.com ...
7.4 配置director-2的heartbeat服务：
# cd /etc/ha.d/
# cp /usr/share/doc/heartbeat-2.1.4/ha.cf ./
# cp /usr/share/doc/heartbeat-2.1.4/authkeys ./
# cp /usr/share/doc/heartbeat-2.1.4/haresources ./
# cp /etc/init.d/ipvsadm resource.d/
# vim ha.cf
#添加以下几行
95 bcast eth1
214 node director1.japan.com
215 node director2.japan.com
# vim haresources
45 director1.japan.com 192.168.2.133/28/eth0/192.168.2.143 ipvsadm
# vim authkeys
#在末尾添加以下几行：
auth 1
1 md5 007d644a23b1e0e6361fcce286268582
# chmod 600 authkeys
# service heartbeat status
heartbeat is stopped. No process
# service heartbeat start
Starting High-Availability services:
2012/04/02_13:42:27 INFO:Running OK
2012/04/02_13:42:27 CRITICAL: Resource 192.168.2.133/27/eth0/192.168.2.143 is active, and should not be!
2012/04/02_13:42:27 CRITICAL: Non-idle resources can affect data integrity!
2012/04/02_13:42:27 info: If you don't know what this means, then get help!
2012/04/02_13:42:27 info: Read the docs and/or source to /usr/share/heartbeat/ResourceManager for more details.
CRITICAL: Resource 192.168.2.133/27/eth0/192.168.2.143 is active, and should not be!
CRITICAL: Non-idle resources can affect data integrity!
info: If you don't know what this means, then get help!
info: Read the docs and/or the source to /usr/share/heartbeat/ResourceManager for more details.
2012/04/02_13:42:27 CRITICAL: Non-idle resources will affect resource takeback!
2012/04/02_13:42:27 CRITICAL: Non-idle resources may affect data integrity!
# service heartbeat status
heartbeat OK is running on director2.japan.com ...
# vim /etc/resolv.conf
1 ; generated by /sbin/dhclient-script
2 nameserver 192.168.2.131
3 nameserver 192.168.2.132
# vim /etc/resolv.conf
1 ; generated by /sbin/dhclient-script
2 nameserver 192.168.2.131
3 nameserver 192.168.2.132
八．测试：
8.1 使用ip地址测试：
# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCPwww.japan.com:http rr
-> 192.168.2.131:http Route 1 0 7
-> 192.168.2.132:http Route 1 0 7
# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCPwww.japan.com:http rr
-> 192.168.2.131:http Route 1 0 0
-> 192.168.2.132:http Route 1 0 0
# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
nameserver 192.168.2.131
nameserver 192.168.2.132
# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
nameserver 192.168.2.131
nameserver 192.168.2.132
# ipvsadm -A -t 192.168.2.133:53 -s rr
# ipvsadm -a -t 192.168.2.133:53 -r 192.168.2.131 -g
# ipvsadm -a -t 192.168.2.133:53 -r 192.168.2.132 -g
# ipvsadm -A -u 192.168.2.133:53 -s rr
# ipvsadm -a -u 192.168.2.133:53 -r 192.168.2.131 -g
# ipvsadm -a -u 192.168.2.133:53 -r 192.168.2.132 -g
# service ipvsadm save
Saving IPVS table to /etc/sysconfig/ipvsadm:
# cat /etc/sysconfig/ipvsadm
-A -u 192.168.2.133:53 -s rr
-a -u 192.168.2.133:53 -r 192.168.2.132:53 -g -w 1
-a -u 192.168.2.133:53 -r 192.168.2.131:53 -g -w 1
-A -t 192.168.2.133:53 -s rr
-a -t 192.168.2.133:53 -r 192.168.2.132:53 -g -w 1
-a -t 192.168.2.133:53 -r 192.168.2.131:53 -g -w 1
-A -t 192.168.2.133:80 -s rr
-a -t 192.168.2.133:80 -r 192.168.2.131:80 -g -w 1
-a -t 192.168.2.133:80 -r 192.168.2.132:80 -g -w 1
# ipvsadm -A -t 192.168.2.133:53 -s rr
# ipvsadm -a -t 192.168.2.133:53 -r 192.168.2.131 -g
# ipvsadm -a -t 192.168.2.133:53 -r 192.168.2.132 -g
# ipvsadm -A -u 192.168.2.133:53 -s rr
# ipvsadm -a -u 192.168.2.133:53 -r 192.168.2.131 -g
# ipvsadm -a -u 192.168.2.133:53 -r 192.168.2.132 -g
# service ipvsadm save
Saving IPVS table to /etc/sysconfig/ipvsadm:
#
# cat /etc/sysconfig/ipvsadm
-A -u 192.168.2.133:53 -s rr
-a -u 192.168.2.133:53 -r 192.168.2.132:53 -g -w 1
-a -u 192.168.2.133:53 -r 192.168.2.131:53 -g -w 1
-A -t 192.168.2.133:53 -s rr
-a -t 192.168.2.133:53 -r 192.168.2.132:53 -g -w 1
-a -t 192.168.2.133:53 -r 192.168.2.131:53 -g -w 1
-A -t 192.168.2.133:80 -s rr
-a -t 192.168.2.133:80 -r 192.168.2.131:80 -g -w 1
-a -t 192.168.2.133:80 -r 192.168.2.132:80 -g -w 1
8.2：使用域名http://www.japan.com/访问，
并不断刷新网页，以下网页交替出现
在director1上查看信息：
# ifconfig eth0:0
eth0:0 Link encap:EthernetHWaddr 00:0C:29:66:E1:DA
inet addr:192.168.2.133Bcast:192.168.2.143Mask:255.255.255.240
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
Interrupt:19 Base address:0x2000
# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDPwww.japan.com:domain rr
-> 192.168.2.131:domain Route 1 0 51
-> 192.168.2.132:domain Route 1 0 49
TCPwww.japan.com:domain rr
-> 192.168.2.131:domain Route 1 0 0
-> 192.168.2.132:domain Route 1 0 0
TCPwww.japan.com:http rr
-> 192.168.2.132:http Route 1 0 31
-> 192.168.2.131:http Route 1 0 30
在director2上查看信息：
root@director2 ha.d]# ifconfig eth0:0
eth0:0 Link encap:EthernetHWaddr 00:0C:29:79:F8:F7
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
Interrupt:19 Base address:0x2000
# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
可以看出此时director1为主要的调度器，director2为standby状态！！
九：模拟director1服务器故障情况，并测试
9.1模拟失效情况：
# cd /usr/lib/heartbeat/
# ls
# ./hb_standby # (模拟失效)
2012/04/02_17:00:35 Going standby .
在director1上查看信息：
# ifconfig eth0:0
eth0:0 Link encap:EthernetHWaddr 00:0C:29:66:E1:DA
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
Interrupt:19 Base address:0x2000
# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
在director2上查看信息：
# ifconfig eth0:0
eth0:0 Link encap:EthernetHWaddr 00:0C:29:79:F8:F7
inet addr:192.168.2.133Bcast:192.168.2.143Mask:255.255.255.240
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
Interrupt:19 Base address:0x2000
# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDPwww.japan.com:domain rr
-> 192.168.2.131:domain Route 1 0 9
-> 192.168.2.132:domain Route 1 0 9
TCPwww.japan.com:domain rr
-> 192.168.2.131:domain Route 1 0 0
-> 192.168.2.132:domain Route 1 0 0
TCPwww.japan.com:http rr
-> 192.168.2.132:http Route 1 0 0
-> 192.168.2.131:http Route 1 0 0
9.2使用域名http://www.japan.com/访问，
并不断刷新网页，以下网页交替出现
# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDPwww.japan.com:domain rr
-> 192.168.2.131:domain Route 1 0 13
-> 192.168.2.132:domain Route 1 0 12
TCPwww.japan.com:domain rr
-> 192.168.2.131:domain Route 1 0 0
-> 192.168.2.132:domain Route 1 0 0
TCPwww.japan.com:http rr
-> 192.168.2.132:http Route 1 0 30
-> 192.168.2.131:http Route 1 0 30
9.3 模拟故障恢复：
# ./hb_takeover
在director1上查看信息：
# ifconfig eth0:0
eth0:0 Link encap:EthernetHWaddr 00:0C:29:66:E1:DA
inet addr:192.168.2.133Bcast:192.168.2.143Mask:255.255.255.240
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
Interrupt:19 Base address:0x2000
# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDPwww.japan.com:domain rr
-> 192.168.2.131:domain Route 1 0 8
-> 192.168.2.132:domain Route 1 0 8
TCPwww.japan.com:domain rr
-> 192.168.2.131:domain Route 1 0 0
-> 192.168.2.132:domain Route 1 0 0
TCPwww.japan.com:http rr
-> 192.168.2.132:http Route 1 0 0
-> 192.168.2.131:http Route 1 0 0
在director2上查看信息：
# ifconfig eth0:0
eth0:0 Link encap:EthernetHWaddr 00:0C:29:79:F8:F7
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
Interrupt:19 Base address:0x2000
# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
9.4使用域名访问，并不断刷新网页，以下网页交替出现
在director1上查看信息：
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP192.168.2.133:53 rr
-> 192.168.2.131:53 Route 1 0 22
-> 192.168.2.132:53 Route 1 0 22
TCP192.168.2.133:53 rr
-> 192.168.2.131:53 Route 1 0 0
-> 192.168.2.132:53 Route 1 0 0
TCP192.168.2.133:80 rr
-> 192.168.2.132:80 Route 1 0 25
-> 192.168.2.131:80 Route 1 0 24
至此在linux下搭建HA和LB集群成功！！！
十．模拟web服务器故障情况：
10.1 故障测试
10.1.1查看director1上的HA集群信息：如下：
（可以看出director1为主控制器，而且显示的HA显示了2个real-server的信息）
# ifconfig eth0:0
eth0:0 Link encap:EthernetHWaddr 00:0C:29:66:E1:DA
inet addr:192.168.2.133Bcast:192.168.2.143Mask:255.255.255.240
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
Interrupt:19 Base address:0x2000
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP192.168.2.133:53 rr
-> 192.168.2.131:53 Route 1 0 97
-> 192.168.2.132:53 Route 1 0 97
TCP192.168.2.133:53 rr
-> 192.168.2.131:53 Route 1 0 0
-> 192.168.2.132:53 Route 1 0 0
TCP192.168.2.133:80 rr
-> 192.168.2.132:80 Route 1 0 0
-> 192.168.2.131:80 Route 1 0 0
10.1.2在real-server-1上的停止httpd和named服务，模拟real-server-1服务器故障，如下：
# service httpd stop
Stopping httpd:
# service named stop
Stopping named:
#
10.1.3再次查看director1上的HA集群信息，如下：
（可以看出在director1显示了错误的的HA的信息，此时real-server-1服务器也不能正常工作，但是在director1上无法发现）
# ifconfig eth0:0
eth0:0 Link encap:EthernetHWaddr 00:0C:29:66:E1:DA
inet addr:192.168.2.133Bcast:192.168.2.143Mask:255.255.255.240
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
Interrupt:19 Base address:0x2000
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP192.168.2.133:53 rr
-> 192.168.2.131:53 Route 1 0 97
-> 192.168.2.132:53 Route 1 0 97
TCP192.168.2.133:53 rr
-> 192.168.2.131:53 Route 1 0 0
-> 192.168.2.132:53 Route 1 0 0
TCP192.168.2.133:80 rr
-> 192.168.2.132:80 Route 1 0 0
-> 192.168.2.131:80 Route 1 0 0
10.1.4客户端不断刷新网页，而且只显示如下网页，且反应缓慢：
（说明：real-server-1服务器已经出现故障）
10.1.5再次查看director1上的HA集群信息，如下：
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP192.168.2.133:53 rr
-> 192.168.2.131:53 Route 1 0 113
-> 192.168.2.132:53 Route 1 0 114
TCP192.168.2.133:53 rr
-> 192.168.2.131:53 Route 1 0 0
-> 192.168.2.132:53 Route 1 0 0
TCP192.168.2.133:80 rr
-> 192.168.2.132:80 Route 1 0 16
-> 192.168.2.131:80 Route 1 0 17
（说明：director1依然认为real-server-1正常工作，并且不断调度请求给real-server-1，此时已经出现大问题了）
此时的解决方法是：使director能够知道real-server的工作情况！
10.2 解决web故障问题
10.2.1在director1上安装配置heartbeat-ldirectord
# cd HA/
# ls
heartbeat-2.1.4-9.el5.i386.rpm
heartbeat-ldirectord-2.1.4-9.el5.i386.rpm
heartbeat-pils-2.1.4-10.el5.i386.rpm
heartbeat-stonith-2.1.4-10.el5.i386.rpm
libnet-1.1.4-3.el5.i386.rpm
perl-MailTools-1.77-1.el5.noarch.rpm
# yum localinstall heartbeat-ldirectord-2.1.4-9.el5.i386.rpm –nogpgcheck –y
# cp /usr/share/doc/heartbeat-ldirectord-2.1.4/ldirectord.cf /etc/ha.d
# cd /etc/ha.d/
# vim ldirectord.cf
21 quiescent=yes
24 virtual=192.168.2.133:80
25 real=192.168.2.131:80 gate
26 real=192.168.2.132:80 gate
27 service=http
28 request=".test.html"
29 receive="ok"
30 virtualhost=www.japan.com
31 scheduler=rr
34 protocol=tcp
# vim haresources
46 director1.japan.com 192.168.2.133 ldirectord::ldirectord.cf
# service heartbeat restart
Stopping High-Availability services:
Waiting to allow resource takeover to complete:
Starting High-Availability services:
2012/04/04_11:25:46 INFO:Resource is stopped
10.2.2在director2上安装配置heartbeat-ldirectord
# cd HA/
# ls
heartbeat-2.1.4-9.el5.i386.rpm
heartbeat-ldirectord-2.1.4-9.el5.i386.rpm
heartbeat-pils-2.1.4-10.el5.i386.rpm
heartbeat-stonith-2.1.4-10.el5.i386.rpm
libnet-1.1.4-3.el5.i386.rpm
perl-MailTools-1.77-1.el5.noarch.rpm
# yum localinstall heartbeat-ldirectord-2.1.4-9.el5.i386.rpm –nogpgcheck –y
# cp /usr/share/doc/heartbeat-ldirectord-2.1.4/ldirectord.cf /etc/ha.d
# cd /etc/ha.d/
# vim ldirectord.cf
21 quiescent=yes
24 virtual=192.168.2.133:80
25 real=192.168.2.131:80 gate
26 real=192.168.2.132:80 gate
27 service=http
28 request=".test.html"
29 receive="ok"
30 virtualhost=www.japan.com
31 scheduler=rr
34 protocol=tcp
# vim haresources
46 director1.japan.com 192.168.2.133 ldirectord::ldirectord.cf
# service heartbeat restart
Stopping High-Availability services:
Waiting to allow resource takeover to complete:
Starting High-Availability services:
2012/04/04_11:25:53 INFO:Resource is stopped
10.2.3此时，查看director1上的HA集群信息，如下：
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.133:80 rr
-> 192.168.2.132:80 Route 0 0 0
-> 192.168.2.131:80 Route 0 0 0
（由于在/etc/ha.d/ldirectord.cf中21行存在 quiescent=yes，故此处http的权重为0，即此时不提供服务）
#修改/etc/ha.d/ldirectord.cf中21行为 quiescent=no，会自动加载
# vim ldirectord.cf
21 quiescent=no
# vim ldirectord.cf
21 quiescent=no
#并再次查看director1上的HA集群信息
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.133:80 rr
#
（由于在/etc/ha.d/ldirectord.cf中21行存在 quiescent=no，故此处http的记录为空）
10.2.4此时，real-server-1上添加以下信息：
# echo "ok" >> /var/www/html/.test.html
查看director1上的HA集群信息，如下：
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.133:80 rr
-> 192.168.2.131:80 Route 1 0 0
#
此时，real-server-2上添加以下信息：
# echo "ok" >> /var/www/html/.test.html
查看director1上的HA集群信息，如下：
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.133:80 rr
-> 192.168.2.132:80 Route 1 0 0
-> 192.168.2.131:80 Route 1 0 0
#
10.2.5此时，在real-server-1上停止httpd服务，出现以下信息：
# service httpd stop
Stopping httpd:
查看director1上的HA集群信息，如下
#
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.133:80 rr
-> 192.168.2.132:80 Route 1 0 0
在real-server-1上和real-server-2上停止httpd服务，出现以下信息：
# service httpd stop
Stopping httpd:
#
# service httpd stop
Stopping httpd:
#
查看director1上的HA集群信息，如下
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.133:80 rr
10.2.6 在real-server-1上和real-server-2上启动httpd服务，恢复正常情况
# service httpd start
Starting httpd: httpd: apr_sockaddr_info_get() failed for server1.japan.com
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
#
# service httpd start
Starting httpd: httpd: apr_sockaddr_info_get() failed for server2.japan.com
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
#
查看director1上的HA集群信息，如下
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.133:80 rr
-> 192.168.2.131:80 Route 1 0 0
-> 192.168.2.132:80 Route 1 0 0
至此在linux下搭建HA和LB集群成功！！！
附录：stonith简单测试(群集分裂解决方案)
安装相关软件包：（我在前面已经安装了，所以这里就不安装了）
# cd HA/
# ls
heartbeat-pils-2.1.4-10.el5.i386.rpm
heartbeat-stonith-2.1.4-10.el5.i386.rpm
# cd HA/
# ls
heartbeat-pils-2.1.4-10.el5.i386.rpm
heartbeat-stonith-2.1.4-10.el5.i386.rpm
#查看当前支持Stonith设备清单的命令：
# /usr/sbin/stonith -L
apcmaster
apcmastersnmp
apcsmart
baytech
bladehpi
cyclades
…
ibmhmc
meatware
null
nw_rpc100s
rcd_serial
rps10
ssh
suicide
wti_nps
# /usr/sbin/stonith -L
apcmaster
apcmastersnmp
apcsmart
baytech
bladehpi
cyclades
…
ibmhmc
meatware
null
nw_rpc100s
rcd_serial
rps10
ssh
suicide
wti_nps
如果尚未购买STONITH功能所支援的硬件，但是想测试STONITH功能的话 ，可以使用虚拟的STONITH装置进行试验。可以使用文件编辑器打开主服务器上的/etc/ha.d/ha.cf
# cd /etc/ha.d/
199 stonith_host director1.japan.com null director2.japan.com
# cd /etc/ha.d/
199 stonith_host director2.japan.com null director1.japan.com
编辑完成后重启heartbeat，以使新设定生效。
# service heartbeat restart
Stopping High-Availability services:
Waiting to allow resource takeover to complete:
Starting High-Availability services:
2012/04/04_14:31:41 INFO:Resource is stopped
#
# service heartbeat restart
Stopping High-Availability services:
Waiting to allow resource takeover to complete:
Starting High-Availability services:
2012/04/04_14:31:41 INFO:Resource is stopped
#
# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDPwww.japan.com:domain rr
-> 192.168.2.131:domain Route 1 0 6
-> 192.168.2.132:domain Route 1 0 6
TCPwww.japan.com:domain rr
-> 192.168.2.131:domain Route 1 0 0
-> 192.168.2.132:domain Route 1 0 0
TCPwww.japan.com:http rr
-> 192.168.2.132:http Route 1 0 0
-> 192.168.2.131:http Route 1 0 0
然后在备份服务器上执行下面命令关闭网络：
最后开启主要服务器上的/var/log/ha-log记录，搜寻STONITH字串，观察STONITH功能的运作情形:会发现内有：
# tail -f /var/log/messages
Apr4 14:23:05 director1 heartbeat: : WARN: node director2.japan.com: is dead
Apr4 14:23:05 director1 heartbeat: : info: Link director2.japan.com:eth1 dead.
Apr4 14:23:05 director1 heartbeat: : info: Resetting node director2.japan.com with //使用STONITH装置将备份服务器关机
Apr4 14:23:05 director1 heartbeat: : info: glib: Host null-reset: director2.japan.com
Apr4 14:23:05 director1 heartbeat: : info: node director2.japan.com now reset.
//备份服务器已经关机
Apr4 14:23:05 director1 heartbeat: : info: Managed STONITH director2.japan.com process 16873 exited with return code 0.
{:6_390:} {:6_404:} {:6_394:}{:6_394:}{:6_394:} 爱护环境，人人有病。 人生自古谁无死，啊个拉屎不用纸! 打破老婆终身制，实行小姨股份制。引入小姐竞争制，推广情人合同制。
页:
[1]