Heartbeat实现Nginx高可用性(style 1.x)

踏雪寻梅

一、准备工作

1. 系统：两台CentOS 5.4虚拟机
2. Hostname：HA1，HA2
3. IP地址：HA1   eth0：192.168.2.10   eth1：192.168.10.1
HA2   eth0：192.168.2.20   eth1：192.168.10.2
4. VIP：192.168.2.100   （Failover转移用的IP）

二、安装

1. Nginx编译安装
tar xzvf pcre-7.9.tar.gz
cd pcre-7.9
./configure
make
make install
cd ..

tar xzvf nginx-0.7.63.tar.gz
cd nginx-0.7.63
./configure –user=nobody –group=nobody –prefix=/usr/local/nginx –with-http_stub_status_module –with-http_ssl_module
make
make install

Nginx具体配置略。

2. Heartbeat编译安装

tar xzvf libnet-1.1.2.1.tar.gz
cd libnet
./configure
make
make install
cd ..

创建用户和用户组

heartbeat需要haclient用户组和hacluster用户两个节点做同样的操作，并保证haclient和hacluster的ID一样。

groupadd -g 500 haclient

useradd -u 500 -g haclient hacluster

tar jxvf STABLE-2.1.4.tar.bz2
cd Heartbeat-STABLE-2-1-STABLE-2.1.4/
./ConfigureMe configure
make
make install
# 拷贝配置文件到相应目录
cp doc/ha.cf /etc/ha.d/
cp doc/haresources /etc/ha.d/
cp doc/authkeys /etc/ha.d/
cd !$   # 跳转到/etc/ha.d/目录

三、配置Heartbeat

在/etc/ha.d/目录下进行配置：
1. vi authkeys   # 节点认证方式，这里使用第一种crc
auth 1
1 crc
＃ 修改authkeys权限为600
chmode 600 authkeys

2. 编辑/etc/ha.d/ha.cf：
[iyunv@HA1 ha.d]# cat ha.cf |sed ‘/^#/d’
# 开启HA的debug日志，建议调试完后关闭此日志
debugfile /var/log/ha-debug
# 开启HA日志
logfile    /var/log/ha-log
# 设置日志打印级别
logfacility    local0
# 多长时间建材一次心跳
keepalive 2
# 连续多长时间检测失败示对方挂掉，单位秒
deadtime 30
# 连续多长时间检测失败开始警告提示，单位秒
warntime 10
# 为服务重启预留一段时间，在这段时间不进行心跳检测
initdead 120
# 默认端口是UDP 694，我改为了695，如果在局域网还有人在玩Heartbeat，并且他用广播，你最好改个端口
# 否则可能会导致认证失败
udpport    695
# 使用单播通信，在HA2上修改为ucast    eth1 192.168.10.1
ucast    eth1 192.168.10.2
# 主节点恢复正常后是否再切换回来
auto_failback on
# 设置看门狗
# Watchdog在实现上可以是硬件电路也可以是软件定时器，能够在系统出现故障时自动重新启动系统。
# 在Linux 内核下,
watchdog的基本工作原理是：当watchdog启动后(即/dev/watchdog
设备被打开后)，
# 如果在某一设定的时间间隔内/dev/watchdog没有被执行写操作,
# 硬件watchdog电路或软件定时器就会重新启动系统。
watchdog /dev/watchdog
# 节点列表，主节点在前，不要写反了
node    HA1
node    HA2

3. [iyunv@HA1 ha.d]# cat haresources

# 每一行代表一个资源组,资源组启动顺序是从左往右，关闭的顺序是从右往左。
＃ 一个资源组里面不同资源之间以空格分隔，不同的资源组之间没有必然关系
# 资源组的第一列是我们在ha.cf配置文件中列出的节点之一，而且应该是准备作为主节点的那一个节点。
# 每个资源都是一个脚本，可以放在/etc/init.d目录下面，也可以在/usr/local/etc/ha.d/resource.d目录下。
# 这些脚本必须要支持start和stop参数。
# 脚本的参数通过::来分隔。
# 主节点   VIP      资源名
HA1    192.168.2.100    nginxd

4. 编写nginxd资源脚本，放到/etc/rc.d/init.d/和/etc/ha.d/resource.d/下

#!/bin/sh

# source function library
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0

RETVAL=0
prog="nginx"

nginxDir=/usr/local/nginx
nginxd=$nginxDir/sbin/nginx
nginxConf=$nginxDir/conf/nginx.conf
nginxPid=$nginxDir/nginx.pid

nginx_check()
{
    if [[ -e $nginxPid ]]; then
        ps aux |grep -v grep |grep -q nginx
        if (( $? == 0 )); then
            echo "$prog already running..."
            exit 1
        else
            rm -rf $nginxPid &> /dev/null
        fi
    fi
}

start()
{
    nginx_check
    if (( $? != 0 )); then
        true
    else
        echo -n $"Starting $prog:"
        daemon $nginxd -c $nginxConf
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch /var/lock/subsys/nginx
        return $RETVAL
    fi
}

stop()
{
    echo -n $"Stopping $prog:"
    killproc $nginxd
    RETVAL=$?
    echo
    [ $RETVAL = 0 ] && rm -f /var/lock/subsys/nginx $nginxPid
}

reload()
{
    echo -n $"Reloading $prog:"
    killproc $nginxd -HUP
    RETVAL=$?
    echo
}

case "$1" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        restart)
                stop
                start
                ;;
        reload)
                reload
                ;;
        status)
                status $prog
                RETVAL=$?
                ;;
        *)
                echo $"Usage: $0 {start|stop|restart|reload|status}"
                RETVAL=1
esac
exit $RETVAL

设置hosts
[iyunv@HA1 ha.d]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1        vpc localhost.localdomain localhost
::1        localhost6.localdomain6 localhost6
192.168.10.1    HA1
192.168.10.2    HA2

注：在HA1和HA2上进行二、三步（安装、配置heartbeat）操作

6. 启动heartbeat
注意：主服务器和备份服务器的时间同步，如果相差太多heartbeat可能发生故障。

service heartbeat restart
查看heartbeat的日志启动信息（日志对于排错很有帮助）
tail -100 /var/log/ha-log
heartbeat[13821]: 2009/11/07_19:41:27 info: Configuration validated. Starting heartbeat 2.1.4
heartbeat[13822]: 2009/11/07_19:41:27 info: heartbeat: version 2.1.4
heartbeat[13822]: 2009/11/07_19:41:27 info: Heartbeat generation: 1257517561
heartbeat[13822]: 2009/11/07_19:41:27 info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth1
heartbeat[13822]: 2009/11/07_19:41:27 info: glib: ucast: bound send socket to device: eth1
heartbeat[13822]: 2009/11/07_19:41:27 info: glib: ucast: bound receive socket to device: eth1
heartbeat[13822]: 2009/11/07_19:41:27 info: glib: ucast: started on port 695 interface eth1 to 192.168.10.2
heartbeat[13822]: 2009/11/07_19:41:27 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[13822]: 2009/11/07_19:41:27 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[13822]: 2009/11/07_19:41:27 notice: Using watchdog device: /dev/watchdog
heartbeat[13822]: 2009/11/07_19:41:27 info: G_main_add_SignalHandler: Added signal handler for signal 17
heartbeat[13822]: 2009/11/07_19:41:27 info: Local status now set to: ‘up’
heartbeat[13822]: 2009/11/07_19:41:29 info: Link ha2:eth1 up.
heartbeat[13822]: 2009/11/07_19:41:29 info: Status update for node ha2: status up
harc[13828]:    2009/11/07_19:41:29 info: Running /etc/ha.d/rc.d/status status
heartbeat[13822]: 2009/11/07_19:41:30 info: Comm_now_up(): updating status to active
heartbeat[13822]: 2009/11/07_19:41:30 info: Local status now set to: ‘active’
heartbeat[13822]: 2009/11/07_19:41:30 info: Status update for node ha2: status active
harc[13845]:    2009/11/07_19:41:30 info: Running /etc/ha.d/rc.d/status status
heartbeat[13822]: 2009/11/07_19:41:45 info: local resource transition completed.
heartbeat[13822]: 2009/11/07_19:41:45 info: Initial resource acquisition complete (T_RESOURCES(us))
IPaddr[13900]:    2009/11/07_19:41:45 INFO:  Resource is stopped
heartbeat[13864]: 2009/11/07_19:41:45 info: Local Resource acquisition completed.
harc[13939]:    2009/11/07_19:41:45 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp
ip-request-resp[13939]:    2009/11/07_19:41:45 received ip-request-resp 192.168.2.100 OK yes
ResourceManager[13960]:    2009/11/07_19:41:45 info: Acquiring resource group: ha1 192.168.2.100 nginxd
IPaddr[13987]:    2009/11/07_19:41:45 INFO:  Resource is stopped
ResourceManager[13960]:    2009/11/07_19:41:45 info: Running /etc/ha.d/resource.d/IPaddr 192.168.2.100 start
IPaddr[14063]:    2009/11/07_19:41:46 INFO: Using calculated nic for 192.168.2.100: eth0
IPaddr[14063]:    2009/11/07_19:41:46 INFO: Using calculated netmask for 192.168.2.100: 255.255.255.0
IPaddr[14063]:    2009/11/07_19:41:46 INFO: eval ifconfig eth0:0 192.168.2.100 netmask 255.255.255.0 broadcast 192.168.2.255
IPaddr[14046]:    2009/11/07_19:41:46 INFO:  Success
heartbeat[13822]: 2009/11/07_19:41:46 info: remote resource transition completed.

查看网卡配置情况，VIP已配置到HA1上。
eth0:0    Link encap:Ethernet  HWaddr 00:0C:29:35:6F:D0
inet addr:192.168.2.100  Bcast:192.168.2.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
Interrupt:67 Base address:0×2000
查看nginx已经启动。

如果看到下面日志，可能是同网段中有人在UDP 694端口运行广播的heartbeat，换个端口试试可能能解决问题。

heartbeat[9966]: 2009/11/07_00:18:53 info: Configuration validated. Starting heartbeat 2.1.4
heartbeat[9967]: 2009/11/07_00:18:53 info: heartbeat: version 2.1.4
heartbeat[9967]: 2009/11/07_00:18:53 info: Heartbeat generation: 1257517538
heartbeat[9967]: 2009/11/07_00:18:53 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1
heartbeat[9967]: 2009/11/07_00:18:53 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 – Status: 1
heartbeat[9967]: 2009/11/07_00:18:53 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[9967]: 2009/11/07_00:18:53 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[9967]: 2009/11/07_00:18:53 info: G_main_add_SignalHandler: Added signal handler for signal 17
heartbeat[9967]: 2009/11/07_00:18:53 info: Local status now set to: ‘up’
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: process_status_message: bad node [master] in message
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG: Dumping message with 12 fields
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG[0] : [t=status]
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG[1] : [st=active]
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG[2] : [dt=7530]
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG[3] : [protocol=1]
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG[4] : [src=master]
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG[5] : [(1)srcuuid=0x9696e70(36 27)]
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG[6] : [seq=1fed7]
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG[7] : [hg=4aee4ce7]
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG[8] : [ts=4af3a3d5]
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG[9] : [ld=0.11 0.03 0.01 1/107 30681]
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG[10] : [ttl=3]
heartbeat[9967]: 2009/11/07_00:18:55 ERROR: MSG[11] : [auth=1 ba81b6cc]
heartbeat[9967]: 2009/11/07_00:18:55 info: Link ha1:eth1 up.
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: process_status_message: bad node [slave] in message
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG: Dumping message with 12 fields
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG[0] : [t=status]
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG[1] : [st=active]
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG[2] : [dt=7530]
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG[3] : [protocol=1]
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG[4] : [src=slave]
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG[5] : [(1)srcuuid=0x9696dc8(36 27)]
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG[6] : [seq=1f94b]
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG[7] : [hg=4aee4cf3]
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG[8] : [ts=4af3a3d6]
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG[9] : [ld=0.00 0.00 0.00 1/105 870]
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG[10] : [ttl=3]
heartbeat[9967]: 2009/11/07_00:18:56 ERROR: MSG[11] : [auth=1 bcd3be0a]

四、测试
1. 手动切换是否正常
在HA1上执行/usr/share/heartbeat/hb_standby看VIP是否能够转移到HA2
查看heartbeat的日志信息
tail -100 /var/log/ha-log
heartbeat[13822]: 2009/11/07_19:44:33 info: ha1 wants to go standby [all]
heartbeat[13822]: 2009/11/07_19:44:33 info: standby: ha2 can take our all resources
heartbeat[14194]: 2009/11/07_19:44:33 info: give up all HA resources (standby).
ResourceManager[14207]:    2009/11/07_19:44:34 info: Releasing resource group: ha1 192.168.2.100 nginxd
ResourceManager[14207]:    2009/11/07_19:44:34 info: Running /etc/ha.d/resource.d/nginxd  stop
ResourceManager[14207]:    2009/11/07_19:44:34 info: Running /etc/ha.d/resource.d/IPaddr 192.168.2.100 stop
IPaddr[14295]:    2009/11/07_19:44:34 INFO: ifconfig eth0:0 down
IPaddr[14278]:    2009/11/07_19:44:34 INFO:  Success
heartbeat[14194]: 2009/11/07_19:44:34 info: all HA resource release completed (standby).
heartbeat[13822]: 2009/11/07_19:44:34 info: Local standby process completed [all].
heartbeat[13822]: 2009/11/07_19:44:36 WARN: 1 lost packet(s) for [ha2] [83:85]
heartbeat[13822]: 2009/11/07_19:44:36 info: remote resource transition completed.
heartbeat[13822]: 2009/11/07_19:44:36 info: No pkts missing from ha2!
heartbeat[13822]: 2009/11/07_19:44:36 info: Other node completed standby takeover of all resources.
查看HA2上VIP已经配置上，nginx也已启动。

2. 切断主节点和备份节点的心跳线看是VIP否能够转移
Down掉HA1的eth1网卡，在HA2上查看heartbeat日志
[iyunv@HA2 ~]# tail -100 /var/log/ha-log
heartbeat[3753]: 2009/11/07_19:59:36 WARN: node ha1: is dead
heartbeat[3753]: 2009/11/07_19:59:36 WARN: No STONITH device configured.
heartbeat[3753]: 2009/11/07_19:59:36 WARN: Shared disks are not protected.
heartbeat[3753]: 2009/11/07_19:59:36 info: Resources being acquired from ha1.
heartbeat[3753]: 2009/11/07_19:59:36 info: Link ha1:eth1 dead.
harc[4255]:    2009/11/07_19:59:36 info: Running /etc/ha.d/rc.d/status status
heartbeat[4256]: 2009/11/07_19:59:36 info: No local resources [/usr/share/heartbeat/ResourceManager listkeys ha2] to acquire.
mach_down[4276]:    2009/11/07_19:59:36 info: Taking over resource group 192.168.2.100
ResourceManager[4310]:    2009/11/07_19:59:36 info: Acquiring resource group: ha1 192.168.2.100 nginxd
IPaddr[4337]:    2009/11/07_19:59:37 INFO:  Resource is stopped
ResourceManager[4310]:    2009/11/07_19:59:37 info: Running /etc/ha.d/resource.d/IPaddr 192.168.2.100 start
IPaddr[4413]:    2009/11/07_19:59:37 INFO: Using calculated nic for 192.168.2.100: eth0
IPaddr[4413]:    2009/11/07_19:59:37 INFO: Using calculated netmask for 192.168.2.100: 255.255.255.0
IPaddr[4413]:    2009/11/07_19:59:37 INFO: eval ifconfig eth0:0 192.168.2.100 netmask 255.255.255.0 broadcast 192.168.2.255
IPaddr[4396]:    2009/11/07_19:59:37 INFO:  Success
ResourceManager[4310]:    2009/11/07_19:59:37 info: Running /etc/ha.d/resource.d/nginxd  start
mach_down[4276]:    2009/11/07_19:59:38 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down[4276]:    2009/11/07_19:59:38 info: mach_down takeover complete for node ha1.
heartbeat[3753]: 2009/11/07_19:59:38 info: mach_down takeover complete.
资源从HA1转移到了HA2。

启动HA1的eth1网卡，可以看到资源从HA2上自动转移到HA1上。

3. 停掉HA1或是停掉HA1上的heartbeat，看VIP是否能够转移到HA2
资源从HA1转移到了HA2。

五、HA管理

启动/停止heartbeat：
service heartbeat start/stop

查看heartbeat状态：
[iyunv@HA2 ~]# service heartbeat status
heartbeat OK [pid 4724 et al] is running on ha2 [ha2]…

手工切换（将本地资源转移到远程主机）：
[iyunv@HA1 ～]# /usr/share/heartbeat/hb_standby
2009/11/07_20:11:03 Going standby [all].

手动接管（将资源接管到本地）：
[iyunv@HA2 ~]# /usr/share/heartbeat/hb_takeover

总结：通过上面的配置可以达到当其中一个节点Down掉后有另一个节点接管资源目的，但是当nginx本身Down掉后并不能自动故障转移，要想达到此目的必须配置heartbeat style 2.x，请参考《Heartbeat实现Nginx高可用性(style 2.x)》