MooseFS灾备演练实录
昨天晚上去机房为数据库服务器做磁盘扩容，顺带为目前线上的存储系统MooseFS做了一次灾难演练。故此，今天准备把昨天的灾难演练的详情总结一下，分享给大家。如果大家正在使用MooseFS，那么就可以有所参考了。 MooseFS是一个分布式的文件系统，有关它的具体信息，我这里就不多做介绍了，大家可以去参考我之前写过的三篇博文：分布式文件系统之MooseFS----介绍
分布式文件系统之MooseFS----部署
分布式文件系统之MooseFS----管理优化
这里简单先介绍一下，目前我们这套存储的架构设计：
服务器的配置情况：
mfsmaster主和mfsmaster备
品牌：Dell PowerEdge R720
CPU：Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz
内存：8 * 8G
磁盘：300G * 6
RAID级别：10
mfschunkserver
品牌：Dell PowerEdge R720CPU：Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz内存：4 * 8G磁盘：2T * 6RAID级别：10
架构介绍：
整个MooseFS的架构，是以两台mfsmaster，一台主，一台备，中间heartbeat+drbd技术来做该服务的高可用，后端放置3台存储节点，负责提供数据存储服务。
灾难演练计划：
本次灾难演练主要从几个会触发故障的因素出发，采用人工模拟故障的方式来真实触发故障，然后记录故障发生前后相关服务的状态，相关服务的日志记录和故障发生前后客户端的服务使用情况等，然后通过分析记录的信息来判断服务的高可用性是否OK。
几个记录点的目标：
通过记录故障发生前后的服务状态来判断服务切换是否正常
通过记录发生故障时相关服务的日志记录来分析故障发生时，高可用软件的决策和动作
通过记录发生故障前后客户端的服务使用情况，来判断故障对客户端的影响程度
模拟如下几个灾难：
1、heartbeat服务崩溃
2、mfsmaster主对外服务的网络中断
3、mfsmaster主的drbd同步网络中断
备注：
在测试之前，我会在 mfs 客户端放一个持续输入脚本，它会以一秒的间隔向挂载的mfs目录中的某个文件进行文字输入，以此用来判断 mfs 的恢复时间。
1
# for i in {1..20000};do echo `date` $i >> /mfsdata/testxxxx;sleep 1;done
故障1：heartbeat服务崩溃
模拟故障：
1
2
# /etc/init.d/heartbeat stop
Stopping High-Availability services: Done.
故障发生之前的mfsmaster主备服务器状态：
mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
mfs 48931 10 20:03 ? 00:00:00 /usr/local/mfs/sbin/mfsmaster start
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.27/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master01-rj.btr, 2015-06-25 17:20:34
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:10824 nr:3912 dw:12716 dr:16539 al:11 bm:14 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfsmaster备：
1
2
3
4
5
6
7
8
9
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:3912 nr:10896 dw:106643992 dr:16616 al:10 bm:6414 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
故障发生时，mfsmaster主备服务器的日志信息：
mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
Jun 25 20:05:29 mfs-master01-rj.btr heartbeat: : info: Heartbeat shutdown in progress. (48425)
Jun 25 20:05:29 mfs-master01-rj.btr heartbeat: : info: Giving up all HA resources.
ResourceManager(default): 2015/06/25_20:05:29 info: Releasing resource group: mfs-master01-rj.btr IPaddr::10.1.1.26/24/em4 drbddisk::drbd Filesystem::/dev/drbd0::/usr/local/mfs::ext4 mfsmaster
ResourceManager(default): 2015/06/25_20:05:29 info: Running /etc/ha.d/resource.d/mfsmasterstop
ResourceManager(default): 2015/06/25_20:05:31 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /usr/local/mfs ext4 stop
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:05:31 INFO: Running stop for /dev/drbd0 on /usr/local/mfs
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:05:31 INFO: Trying to unmount /usr/local/mfs
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:05:31 INFO: unmounted /usr/local/mfs successfully
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:05:31 INFO:Success
ResourceManager(default): 2015/06/25_20:05:31 info: Running /etc/ha.d/resource.d/drbddisk drbd stop
ResourceManager(default): 2015/06/25_20:05:31 info: Running /etc/ha.d/resource.d/IPaddr 10.1.1.26/24/em4 stop
IPaddr(IPaddr_10.1.1.26):2015/06/25_20:05:31 INFO: IP status = ok, IP_CIP=
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_20:05:31 INFO:Success
Jun 25 20:05:31 mfs-master01-rj.btr heartbeat: : info: All HA resources relinquished.
Jun 25 20:05:32 mfs-master01-rj.btr heartbeat: : WARN: 1 lost packet(s) for
Jun 25 20:05:32 mfs-master01-rj.btr heartbeat: : info: No pkts missing from mfs-master02-rj.btr!
Jun 25 20:05:32 mfs-master01-rj.btr heartbeat: : info: killing /usr/lib64/heartbeat/ipfail process group 48439 with signal 15
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: killing HBFIFO process 48428 with signal 15
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: killing HBWRITE process 48429 with signal 15
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: killing HBREAD process 48430 with signal 15
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: killing HBWRITE process 48431 with signal 15
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: killing HBREAD process 48432 with signal 15
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: killing HBWRITE process 48433 with signal 15
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: killing HBREAD process 48434 with signal 15
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: killing HBWRITE process 48435 with signal 15
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: killing HBREAD process 48436 with signal 15
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: Core process 48430 exited. 9 remaining
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: Core process 48433 exited. 8 remaining
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: Core process 48431 exited. 7 remaining
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: Core process 48428 exited. 6 remaining
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: Core process 48432 exited. 5 remaining
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: Core process 48434 exited. 4 remaining
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: Core process 48435 exited. 3 remaining
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: Core process 48436 exited. 2 remaining
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: Core process 48429 exited. 1 remaining
Jun 25 20:05:33 mfs-master01-rj.btr heartbeat: : info: mfs-master01-rj.btr Heartbeat shutdown complete.
mfsmaster备：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
Jun 25 20:05:31 mfs-master02-rj.btr heartbeat: : info: Received shutdown notice from 'mfs-master01-rj.btr'.
Jun 25 20:05:31 mfs-master02-rj.btr heartbeat: : info: Resources being acquired from mfs-master01-rj.btr.
Jun 25 20:05:31 mfs-master02-rj.btr heartbeat: : info: acquire local HA resources (standby).
Jun 25 20:05:31 mfs-master02-rj.btr heartbeat: : info: No local resources to acquire.
Jun 25 20:05:31 mfs-master02-rj.btr heartbeat: : info: local HA resource acquisition completed (standby).
Jun 25 20:05:31 mfs-master02-rj.btr heartbeat: : info: Standby resource acquisition done .
harc(default): 2015/06/25_20:05:31 info: Running /etc/ha.d//rc.d/status status
mach_down(default): 2015/06/25_20:05:31 info: Taking over resource group IPaddr::10.1.1.26/24/em4
ResourceManager(default): 2015/06/25_20:05:31 info: Acquiring resource group: mfs-master01-rj.btr IPaddr::10.1.1.26/24/em4 drbddisk::drbd Filesystem::/dev/drbd0::/usr/local/mfs::ext4 mfsmaster
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_20:05:31 INFO:Resource is stopped
ResourceManager(default): 2015/06/25_20:05:31 info: Running /etc/ha.d/resource.d/IPaddr 10.1.1.26/24/em4 start
IPaddr(IPaddr_10.1.1.26): 2015/06/25_20:05:31 INFO: Adding inet address 10.1.1.26/24 with broadcast address 10.1.1.255 to device em4
IPaddr(IPaddr_10.1.1.26): 2015/06/25_20:05:31 INFO: Bringing device em4 up
IPaddr(IPaddr_10.1.1.26): 2015/06/25_20:05:31 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.1.1.26 em4 10.1.1.26 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_20:05:31 INFO:Success
ResourceManager(default): 2015/06/25_20:05:31 info: Running /etc/ha.d/resource.d/drbddisk drbd start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:05:32 INFO:Resource is stopped
ResourceManager(default): 2015/06/25_20:05:32 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /usr/local/mfs ext4 start
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:05:32 INFO: Running start for /dev/drbd0 on /usr/local/mfs
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:05:32 INFO:Success
ResourceManager(default): 2015/06/25_20:05:32 info: Running /etc/ha.d/resource.d/mfsmasterstart
mach_down(default): 2015/06/25_20:05:32 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default): 2015/06/25_20:05:32 info: mach_down takeover complete for node mfs-master01-rj.btr.
Jun 25 20:05:32 mfs-master02-rj.btr heartbeat: : info: mach_down takeover complete.
Jun 25 20:05:42 mfs-master02-rj.btr heartbeat: : WARN: node mfs-master01-rj.btr: is dead
Jun 25 20:05:42 mfs-master02-rj.btr heartbeat: : info: Dead node mfs-master01-rj.btr gave up resources.
Jun 25 20:05:42 mfs-master02-rj.btr ipfail: : info: Status update: Node mfs-master01-rj.btr now has status dead
Jun 25 20:05:42 mfs-master02-rj.btr heartbeat: : info: Link mfs-master01-rj.btr:em2 dead.
Jun 25 20:05:44 mfs-master02-rj.btr ipfail: : info: NS: We are still alive!
Jun 25 20:05:44 mfs-master02-rj.btr ipfail: : info: Link Status update: Link mfs-master01-rj.btr/em2 now has status dead
Jun 25 20:05:45 mfs-master02-rj.btr ipfail: : info: Asking other side for ping node count.
Jun 25 20:05:45 mfs-master02-rj.btr ipfail: : info: Checking remote count of ping nodes.
故障发生后，mfsmaster主备服务器的状态：
mfsmaster主：
1
2
3
4
5
6
7
8
9
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.27/24 brd 10.1.1.255 scope global em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master01-rj.btr, 2015-06-25 17:20:34
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:11876 nr:4536 dw:14392 dr:16551 al:12 bm:14 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfsmaster备：
1
2
3
4
5
6
7
8
9
10
11
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
mfs 48197 10 20:05 ? 00:00:00 /usr/local/mfs/sbin/mfsmaster start
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:4604 nr:11864 dw:106645652 dr:18533 al:12 bm:6414 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
客户端的服务中断情况：
1
2
3
4
5
6
7
Thu Jun 25 18:52:31 CST 2015 86
Thu Jun 25 18:52:32 CST 2015 87
Thu Jun 25 18:52:33 CST 2015 88
Thu Jun 25 18:52:34 CST 2015 89 ########
Thu Jun 25 18:52:47 CST 2015 90 ########
Thu Jun 25 18:52:48 CST 2015 91
Thu Jun 25 18:52:49 CST 2015 92
恢复故障1：mfsmaster 的heartbeat服务恢复之后
恢复heartbeat服务：
1
2
3
# /etc/init.d/heartbeat start
Starting High-Availability services: INFO:Resource is stopped
Done.
故障恢复时，mfsmaster主备服务器的日志信息：
mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: Pacemaker support: false
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : WARN: Logging daemon is disabled --enabling logging daemon is recommended
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: **************************
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: Configuration validated. Starting heartbeat 3.0.4
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: heartbeat: version 3.0.4
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: Heartbeat generation: 1435221812
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: glib: UDP multicast heartbeat started for group 225.0.0.192 port 694 interface em2 (ttl=1 loop=0)
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: glib: ping heartbeat started.
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: glib: ping heartbeat started.
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: glib: ping heartbeat started.
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: G_main_add_TriggerHandler: Added signal manual handler
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: G_main_add_TriggerHandler: Added signal manual handler
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: G_main_add_SignalHandler: Added signal handler for signal 17
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: Local status now set to: 'up'
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: Status update for node 10.1.1.27: status ping
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: Link 10.1.1.27:10.1.1.27 up.
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: Status update for node 10.1.1.28: status ping
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: Link 10.1.1.28:10.1.1.28 up.
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: Link mfs-master02-rj.btr:em2 up.
Jun 25 20:09:29 mfs-master01-rj.btr heartbeat: : info: Status update for node mfs-master02-rj.btr: status active
harc(default): 2015/06/25_20:09:29 info: Running /etc/ha.d//rc.d/status status
Jun 25 20:09:30 mfs-master01-rj.btr heartbeat: : info: Link 10.1.1.1:10.1.1.1 up.
Jun 25 20:09:30 mfs-master01-rj.btr heartbeat: : info: Status update for node 10.1.1.1: status ping
Jun 25 20:09:30 mfs-master01-rj.btr heartbeat: : info: Comm_now_up(): updating status to active
Jun 25 20:09:30 mfs-master01-rj.btr heartbeat: : info: Local status now set to: 'active'
Jun 25 20:09:30 mfs-master01-rj.btr heartbeat: : info: Starting child client "/usr/lib64/heartbeat/ipfail" (497,496)
Jun 25 20:09:30 mfs-master01-rj.btr heartbeat: : info: Starting "/usr/lib64/heartbeat/ipfail" as uid 497gid 496 (pid 685)
Jun 25 20:09:30 mfs-master01-rj.btr heartbeat: : info: remote resource transition completed.
Jun 25 20:09:30 mfs-master01-rj.btr heartbeat: : info: remote resource transition completed.
Jun 25 20:09:30 mfs-master01-rj.btr heartbeat: : info: Local Resource acquisition completed. (none)
Jun 25 20:09:31 mfs-master01-rj.btr heartbeat: : info: mfs-master02-rj.btr wants to go standby
Jun 25 20:09:33 mfs-master01-rj.btr heartbeat: : info: standby: acquire resources from mfs-master02-rj.btr
Jun 25 20:09:33 mfs-master01-rj.btr heartbeat: : info: acquire local HA resources (standby).
ResourceManager(default):2015/06/25_20:09:34 info: Acquiring resource group: mfs-master01-rj.btr IPaddr::10.1.1.26/24/em4 drbddisk::drbd Filesystem::/dev/drbd0::/usr/local/mfs::ext4 mfsmaster
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_20:09:34 INFO:Resource is stopped
ResourceManager(default):2015/06/25_20:09:34 info: Running /etc/ha.d/resource.d/IPaddr 10.1.1.26/24/em4 start
IPaddr(IPaddr_10.1.1.26):2015/06/25_20:09:34 INFO: Adding inet address 10.1.1.26/24 with broadcast address 10.1.1.255 to device em4
IPaddr(IPaddr_10.1.1.26):2015/06/25_20:09:34 INFO: Bringing device em4 up
IPaddr(IPaddr_10.1.1.26):2015/06/25_20:09:34 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.1.1.26 em4 10.1.1.26 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_20:09:34 INFO:Success
ResourceManager(default):2015/06/25_20:09:34 info: Running /etc/ha.d/resource.d/drbddisk drbd start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:09:34 INFO:Resource is stopped
ResourceManager(default):2015/06/25_20:09:34 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /usr/local/mfs ext4 start
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:09:34 INFO: Running start for /dev/drbd0 on /usr/local/mfs
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:09:34 INFO:Success
ResourceManager(default):2015/06/25_20:09:34 info: Running /etc/ha.d/resource.d/mfsmasterstart
Jun 25 20:09:34 mfs-master01-rj.btr heartbeat: : info: local HA resource acquisition completed (standby).
Jun 25 20:09:34 mfs-master01-rj.btr heartbeat: : info: Standby resource acquisition done .
Jun 25 20:09:34 mfs-master01-rj.btr heartbeat: : info: Initial resource acquisition complete (auto_failback)
Jun 25 20:09:35 mfs-master01-rj.btr heartbeat: : info: remote resource transition completed.
Jun 25 20:09:42 mfs-master01-rj.btr ipfail: : info: Ping node count is balanced.
Jun 25 20:09:42 mfs-master01-rj.btr ipfail: : info: Giving up foreign resources (auto_failback).
Jun 25 20:09:42 mfs-master01-rj.btr ipfail: : info: Delayed giveup in 4 seconds.
Jun 25 20:09:46 mfs-master01-rj.btr ipfail: : info: giveup() called (timeout worked)
Jun 25 20:09:47 mfs-master01-rj.btr heartbeat: : info: mfs-master01-rj.btr wants to go standby
Jun 25 20:09:47 mfs-master01-rj.btr heartbeat: : info: standby: mfs-master02-rj.btr can take our foreign resources
Jun 25 20:09:47 mfs-master01-rj.btr heartbeat: : info: give up foreign HA resources (standby).
Jun 25 20:09:48 mfs-master01-rj.btr heartbeat: : info: foreign HA resource release completed (standby).
Jun 25 20:09:48 mfs-master01-rj.btr heartbeat: : info: Local standby process completed .
Jun 25 20:09:48 mfs-master01-rj.btr heartbeat: : WARN: 1 lost packet(s) for
Jun 25 20:09:48 mfs-master01-rj.btr heartbeat: : info: remote resource transition completed.
Jun 25 20:09:48 mfs-master01-rj.btr heartbeat: : info: No pkts missing from mfs-master02-rj.btr!
Jun 25 20:09:48 mfs-master01-rj.btr heartbeat: : info: Other node completed standby takeover of foreign resources.
mfsmaster备：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
Jun 25 20:09:29 mfs-master02-rj.btr heartbeat: : info: Heartbeat restart on node mfs-master01-rj.btr
Jun 25 20:09:29 mfs-master02-rj.btr heartbeat: : info: Link mfs-master01-rj.btr:em2 up.
Jun 25 20:09:29 mfs-master02-rj.btr heartbeat: : info: Status update for node mfs-master01-rj.btr: status init
Jun 25 20:09:29 mfs-master02-rj.btr ipfail: : info: Link Status update: Link mfs-master01-rj.btr/em2 now has status up
Jun 25 20:09:29 mfs-master02-rj.btr ipfail: : info: Status update: Node mfs-master01-rj.btr now has status init
Jun 25 20:09:29 mfs-master02-rj.btr heartbeat: : info: Status update for node mfs-master01-rj.btr: status up
Jun 25 20:09:29 mfs-master02-rj.btr ipfail: : info: Status update: Node mfs-master01-rj.btr now has status up
harc(default): 2015/06/25_20:09:29 info: Running /etc/ha.d//rc.d/status status
harc(default): 2015/06/25_20:09:29 info: Running /etc/ha.d//rc.d/status status
Jun 25 20:09:30 mfs-master02-rj.btr heartbeat: : info: Status update for node mfs-master01-rj.btr: status active
Jun 25 20:09:30 mfs-master02-rj.btr ipfail: : info: Status update: Node mfs-master01-rj.btr now has status active
harc(default): 2015/06/25_20:09:30 info: Running /etc/ha.d//rc.d/status status
Jun 25 20:09:30 mfs-master02-rj.btr heartbeat: : info: remote resource transition completed.
Jun 25 20:09:30 mfs-master02-rj.btr heartbeat: : info: mfs-master02-rj.btr wants to go standby
Jun 25 20:09:31 mfs-master02-rj.btr heartbeat: : info: standby: mfs-master01-rj.btr can take our foreign resources
Jun 25 20:09:31 mfs-master02-rj.btr heartbeat: : info: give up foreign HA resources (standby).
ResourceManager(default): 2015/06/25_20:09:31 info: Releasing resource group: mfs-master01-rj.btr IPaddr::10.1.1.26/24/em4 drbddisk::drbd Filesystem::/dev/drbd0::/usr/local/mfs::ext4 mfsmaster
ResourceManager(default): 2015/06/25_20:09:31 info: Running /etc/ha.d/resource.d/mfsmasterstop
Jun 25 20:09:32 mfs-master02-rj.btr ipfail: : info: Asking other side for ping node count.
ResourceManager(default): 2015/06/25_20:09:33 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /usr/local/mfs ext4 stop
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:09:33 INFO: Running stop for /dev/drbd0 on /usr/local/mfs
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:09:33 INFO: Trying to unmount /usr/local/mfs
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:09:33 INFO: unmounted /usr/local/mfs successfully
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_20:09:33 INFO:Success
ResourceManager(default): 2015/06/25_20:09:33 info: Running /etc/ha.d/resource.d/drbddisk drbd stop
ResourceManager(default): 2015/06/25_20:09:33 info: Running /etc/ha.d/resource.d/IPaddr 10.1.1.26/24/em4 stop
IPaddr(IPaddr_10.1.1.26): 2015/06/25_20:09:33 INFO: IP status = ok, IP_CIP=
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_20:09:33 INFO:Success
Jun 25 20:09:33 mfs-master02-rj.btr heartbeat: : info: foreign HA resource release completed (standby).
Jun 25 20:09:33 mfs-master02-rj.btr heartbeat: : info: Local standby process completed .
Jun 25 20:09:34 mfs-master02-rj.btr heartbeat: : WARN: 1 lost packet(s) for
Jun 25 20:09:34 mfs-master02-rj.btr heartbeat: : info: remote resource transition completed.
Jun 25 20:09:34 mfs-master02-rj.btr heartbeat: : info: No pkts missing from mfs-master01-rj.btr!
Jun 25 20:09:34 mfs-master02-rj.btr heartbeat: : info: Other node completed standby takeover of foreign resources.
Jun 25 20:09:42 mfs-master02-rj.btr ipfail: : info: No giveup timer to abort.
Jun 25 20:09:47 mfs-master02-rj.btr heartbeat: : info: mfs-master01-rj.btr wants to go standby
Jun 25 20:09:47 mfs-master02-rj.btr heartbeat: : info: standby: acquire resources from mfs-master01-rj.btr
Jun 25 20:09:47 mfs-master02-rj.btr heartbeat: : info: acquire local HA resources (standby).
Jun 25 20:09:47 mfs-master02-rj.btr heartbeat: : info: local HA resource acquisition completed (standby).
Jun 25 20:09:47 mfs-master02-rj.btr heartbeat: : info: Standby resource acquisition done .
Jun 25 20:09:48 mfs-master02-rj.btr heartbeat: : info: remote resource transition completed.
故障恢复后，mfsmaster主备服务器的状态：
mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
mfs 1165 10 20:09 ? 00:00:00 /usr/local/mfs/sbin/mfsmaster start
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.27/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master01-rj.btr, 2015-06-25 17:20:34
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:12288 nr:5600 dw:15868 dr:18468 al:13 bm:14 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfsmaster备：
1
2
3
4
5
6
7
8
9
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:5600 nr:12324 dw:106647108 dr:18541 al:12 bm:6414 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfs客户端数据恢复信息：
1
2
3
4
5
6
7
8
Thu Jun 25 18:56:33 CST 2015 314
Thu Jun 25 18:56:34 CST 2015 315
Thu Jun 25 18:56:35 CST 2015 316
Thu Jun 25 18:56:36 CST 2015 317 #######
Thu Jun 25 18:56:49 CST 2015 318 #######
Thu Jun 25 18:56:50 CST 2015 319
Thu Jun 25 18:56:51 CST 2015 320
Thu Jun 25 18:56:52 CST 2015 321
故障2：mfsmaster主对外服务的网络中断
模拟故障：
拔掉mfsmaster的网线
故障发生之前的mfsmaster主备服务器状态：mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
mfs 1165 10 20:09 ? 00:00:00 /usr/local/mfs/sbin/mfsmaster start
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.27/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master01-rj.btr, 2015-06-25 17:20:34
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:12692 nr:5600 dw:16272 dr:18468 al:13 bm:14 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfsmaster备：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:5600 nr:12816 dw:106647600 dr:18541 al:12 bm:6414 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:6040 nr:13816 dw:106649040 dr:20458 al:13 bm:6414 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
故障发生时，mfsmaster主备服务器的日志信息：mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
Jun 25 21:17:19 mfs-master01-rj.btr heartbeat: : WARN: node 10.1.1.1: is dead
Jun 25 21:17:19 mfs-master01-rj.btr ipfail: : info: Status update: Node 10.1.1.1 now has status dead
Jun 25 21:17:19 mfs-master01-rj.btr heartbeat: : WARN: node 10.1.1.28: is dead
Jun 25 21:17:19 mfs-master01-rj.btr heartbeat: : info: Link 10.1.1.1:10.1.1.1 dead.
Jun 25 21:17:19 mfs-master01-rj.btr heartbeat: : info: Link 10.1.1.28:10.1.1.28 dead.
harc(default): 2015/06/25_21:17:19 info: Running /etc/ha.d//rc.d/status status
harc(default): 2015/06/25_21:17:19 info: Running /etc/ha.d//rc.d/status status
Jun 25 21:17:21 mfs-master01-rj.btr ipfail: : info: NS: We are still alive!
Jun 25 21:17:21 mfs-master01-rj.btr ipfail: : info: Status update: Node 10.1.1.28 now has status dead
Jun 25 21:17:23 mfs-master01-rj.btr ipfail: : info: NS: We are still alive!
Jun 25 21:17:23 mfs-master01-rj.btr ipfail: : info: Link Status update: Link 10.1.1.1/10.1.1.1 now has status dead
Jun 25 21:17:25 mfs-master01-rj.btr ipfail: : info: Asking other side for ping node count.
Jun 25 21:17:25 mfs-master01-rj.btr ipfail: : info: Checking remote count of ping nodes.
Jun 25 21:17:25 mfs-master01-rj.btr ipfail: : info: Link Status update: Link 10.1.1.28/10.1.1.28 now has status dead
Jun 25 21:17:27 mfs-master01-rj.btr ipfail: : info: Asking other side for ping node count.
Jun 25 21:17:27 mfs-master01-rj.btr ipfail: : info: Checking remote count of ping nodes.
Jun 25 21:17:29 mfs-master01-rj.btr ipfail: : info: Giving up because we have less visible ping nodes.
Jun 25 21:17:29 mfs-master01-rj.btr ipfail: : info: Delayed giveup in 4 seconds.
Jun 25 21:17:29 mfs-master01-rj.btr ipfail: : info: Giving up because we were told that we have less ping nodes.
Jun 25 21:17:29 mfs-master01-rj.btr ipfail: : info: Delayed giveup in 4 seconds.
Jun 25 21:17:30 mfs-master01-rj.btr ipfail: : info: Giving up because we were told that we have less ping nodes.
Jun 25 21:17:30 mfs-master01-rj.btr ipfail: : info: Delayed giveup in 4 seconds.
Jun 25 21:17:34 mfs-master01-rj.btr ipfail: : info: giveup() called (timeout worked)
Jun 25 21:17:35 mfs-master01-rj.btr heartbeat: : info: mfs-master01-rj.btr wants to go standby
Jun 25 21:17:35 mfs-master01-rj.btr heartbeat: : info: standby: mfs-master02-rj.btr can take our all resources
Jun 25 21:17:35 mfs-master01-rj.btr heartbeat: : info: give up all HA resources (standby).
ResourceManager(default): 2015/06/25_21:17:35 info: Releasing resource group: mfs-master01-rj.btr IPaddr::10.1.1.26/24/em4 drbddisk::drbd Filesystem::/dev/drbd0::/usr/local/mfs::ext4 mfsmaster
ResourceManager(default): 2015/06/25_21:17:35 info: Running /etc/ha.d/resource.d/mfsmasterstop
ResourceManager(default): 2015/06/25_21:17:37 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /usr/local/mfs ext4 stop
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:17:37 INFO: Running stop for /dev/drbd0 on /usr/local/mfs
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:17:37 INFO: Trying to unmount /usr/local/mfs
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:17:37 INFO: unmounted /usr/local/mfs successfully
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:17:37 INFO:Success
ResourceManager(default): 2015/06/25_21:17:38 info: Running /etc/ha.d/resource.d/drbddisk drbd stop
ResourceManager(default): 2015/06/25_21:17:38 info: Running /etc/ha.d/resource.d/IPaddr 10.1.1.26/24/em4 stop
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:17:38 INFO: IP status = ok, IP_CIP=
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:17:38 INFO:Success
Jun 25 21:17:38 mfs-master01-rj.btr heartbeat: : info: all HA resource release completed (standby).
Jun 25 21:17:38 mfs-master01-rj.btr heartbeat: : info: Local standby process completed .
Jun 25 21:17:39 mfs-master01-rj.btr heartbeat: : WARN: 1 lost packet(s) for
Jun 25 21:17:39 mfs-master01-rj.btr heartbeat: : info: remote resource transition completed.
Jun 25 21:17:39 mfs-master01-rj.btr heartbeat: : info: No pkts missing from mfs-master02-rj.btr!
Jun 25 21:17:39 mfs-master01-rj.btr heartbeat: : info: Other node completed standby takeover of all resources.
mfsmaster备：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Jun 25 21:17:18 mfs-master02-rj.btr heartbeat: : WARN: node 10.1.1.27: is dead
Jun 25 21:17:18 mfs-master02-rj.btr ipfail: : info: Status update: Node 10.1.1.27 now has status dead
Jun 25 21:17:18 mfs-master02-rj.btr heartbeat: : info: Link 10.1.1.27:10.1.1.27 dead.
harc(default): 2015/06/25_21:17:18 info: Running /etc/ha.d//rc.d/status status
Jun 25 21:17:19 mfs-master02-rj.btr ipfail: : info: NS: We are still alive!
Jun 25 21:17:19 mfs-master02-rj.btr ipfail: : info: Link Status update: Link 10.1.1.27/10.1.1.27 now has status dead
Jun 25 21:17:21 mfs-master02-rj.btr ipfail: : info: Asking other side for ping node count.
Jun 25 21:17:21 mfs-master02-rj.btr ipfail: : info: Checking remote count of ping nodes.
Jun 25 21:17:27 mfs-master02-rj.btr ipfail: : info: Telling other node that we have more visible ping nodes.
Jun 25 21:17:29 mfs-master02-rj.btr ipfail: : info: Telling other node that we have more visible ping nodes.
Jun 25 21:17:35 mfs-master02-rj.btr heartbeat: : info: mfs-master01-rj.btr wants to go standby
Jun 25 21:17:38 mfs-master02-rj.btr heartbeat: : info: standby: acquire resources from mfs-master01-rj.btr
Jun 25 21:17:38 mfs-master02-rj.btr heartbeat: : info: acquire all HA resources (standby).
ResourceManager(default): 2015/06/25_21:17:38 info: Acquiring resource group: mfs-master01-rj.btr IPaddr::10.1.1.26/24/em4 drbddisk::drbd Filesystem::/dev/drbd0::/usr/local/mfs::ext4 mfsmaster
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:17:38 INFO:Resource is stopped
ResourceManager(default): 2015/06/25_21:17:38 info: Running /etc/ha.d/resource.d/IPaddr 10.1.1.26/24/em4 start
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:17:38 INFO: Adding inet address 10.1.1.26/24 with broadcast address 10.1.1.255 to device em4
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:17:38 INFO: Bringing device em4 up
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:17:38 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.1.1.26 em4 10.1.1.26 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:17:38 INFO:Success
ResourceManager(default): 2015/06/25_21:17:38 info: Running /etc/ha.d/resource.d/drbddisk drbd start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:17:38 INFO:Resource is stopped
ResourceManager(default): 2015/06/25_21:17:38 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /usr/local/mfs ext4 start
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:17:38 INFO: Running start for /dev/drbd0 on /usr/local/mfs
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:17:38 INFO:Success
ResourceManager(default): 2015/06/25_21:17:38 info: Running /etc/ha.d/resource.d/mfsmasterstart
Jun 25 21:17:38 mfs-master02-rj.btr heartbeat: : info: all HA resource acquisition completed (standby).
Jun 25 21:17:38 mfs-master02-rj.btr heartbeat: : info: Standby resource acquisition done .
Jun 25 21:17:39 mfs-master02-rj.btr heartbeat: : info: remote resource transition completed.
故障发生后，mfsmaster主备服务器的状态：mfsmaster主：
1
2
3
4
5
6
7
8
9
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.27/24 brd 10.1.1.255 scope global em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master01-rj.btr, 2015-06-25 17:20:34
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:7556 nr:1264 dw:2380 dr:9530 al:6 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfsmaster备：
1
2
3
4
5
6
7
8
9
10
11
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
mfs 5345 10 21:17 ? 00:00:00 /usr/local/mfs/sbin/mfsmaster start
# ip a|grep em4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:1336 nr:7556 dw:106668492 dr:24076 al:16 bm:6421 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
客户端的服务中断情况：
1
2
3
4
5
6
7
8
9
Thu Jun 25 19:01:50 CST 2015 616
Thu Jun 25 19:01:51 CST 2015 617
Thu Jun 25 19:01:52 CST 2015 618
Thu Jun 25 19:01:53 CST 2015 619
Thu Jun 25 19:01:54 CST 2015 620 ################
Thu Jun 25 19:02:56 CST 2015 621 ################
Thu Jun 25 19:02:57 CST 2015 622
Thu Jun 25 19:02:58 CST 2015 623
Thu Jun 25 19:02:59 CST 2015 624
恢复故障2：mfsmaster主对外服务的网络中断
恢复网络服务：插上网线
故障恢复时，mfsmaster主备服务器的日志信息：mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Jun 25 21:21:44 mfs-master01-rj.btr heartbeat: : WARN: Late heartbeat: Node 10.1.1.28: interval 275010 ms
Jun 25 21:21:44 mfs-master01-rj.btr heartbeat: : info: Status update for node 10.1.1.28: status ping
Jun 25 21:21:44 mfs-master01-rj.btr heartbeat: : info: Link 10.1.1.28:10.1.1.28 up.
Jun 25 21:21:44 mfs-master01-rj.btr ipfail: : info: Status update: Node 10.1.1.28 now has status ping
Jun 25 21:21:44 mfs-master01-rj.btr ipfail: : info: A ping node just came up.
Jun 25 21:21:45 mfs-master01-rj.btr heartbeat: : info: Link 10.1.1.1:10.1.1.1 up.
Jun 25 21:21:45 mfs-master01-rj.btr heartbeat: : WARN: Late heartbeat: Node 10.1.1.1: interval 276070 ms
Jun 25 21:21:45 mfs-master01-rj.btr heartbeat: : info: Status update for node 10.1.1.1: status ping
Jun 25 21:21:45 mfs-master01-rj.btr ipfail: : info: Asking other side for ping node count.
Jun 25 21:21:45 mfs-master01-rj.btr ipfail: : info: Link Status update: Link 10.1.1.28/10.1.1.28 now has status up
Jun 25 21:21:45 mfs-master01-rj.btr ipfail: : info: Link Status update: Link 10.1.1.1/10.1.1.1 now has status up
Jun 25 21:21:45 mfs-master01-rj.btr ipfail: : info: Status update: Node 10.1.1.1 now has status ping
Jun 25 21:21:45 mfs-master01-rj.btr ipfail: : info: A ping node just came up.
Jun 25 21:21:47 mfs-master01-rj.btr ipfail: : info: Asking other side for ping node count.
Jun 25 21:21:49 mfs-master01-rj.btr ipfail: : info: Ping node count is balanced.
Jun 25 21:21:50 mfs-master01-rj.btr ipfail: : info: Giving up foreign resources (auto_failback).
Jun 25 21:21:50 mfs-master01-rj.btr ipfail: : info: Delayed giveup in 4 seconds.
Jun 25 21:21:50 mfs-master01-rj.btr ipfail: : info: Giving up because we were told that we have less ping nodes.
Jun 25 21:21:50 mfs-master01-rj.btr ipfail: : info: Delayed giveup in 4 seconds.
Jun 25 21:21:50 mfs-master01-rj.btr ipfail: : info: Aborted delayed giveup (8)
mfsmaster备：
1
2
3
4
5
6
7
8
9
10
11
12
Jun 25 21:21:44 mfs-master02-rj.btr heartbeat: : WARN: Late heartbeat: Node 10.1.1.27: interval 276050 ms
Jun 25 21:21:44 mfs-master02-rj.btr heartbeat: : info: Status update for node 10.1.1.27: status ping
Jun 25 21:21:44 mfs-master02-rj.btr heartbeat: : info: Link 10.1.1.27:10.1.1.27 up.
Jun 25 21:21:44 mfs-master02-rj.btr ipfail: : info: Status update: Node 10.1.1.27 now has status ping
Jun 25 21:21:44 mfs-master02-rj.btr ipfail: : info: A ping node just came up.
Jun 25 21:21:46 mfs-master02-rj.btr ipfail: : info: Asking other side for ping node count.
Jun 25 21:21:46 mfs-master02-rj.btr ipfail: : info: Link Status update: Link 10.1.1.27/10.1.1.27 now has status up
Jun 25 21:21:48 mfs-master02-rj.btr ipfail: : info: Telling other node that we have more visible ping nodes.
Jun 25 21:21:50 mfs-master02-rj.btr ipfail: : info: Ping node count is balanced.
Jun 25 21:21:50 mfs-master02-rj.btr ipfail: : info: Giving up foreign resources (auto_failback).
Jun 25 21:21:50 mfs-master02-rj.btr ipfail: : info: Delayed giveup in 4 seconds.
Jun 25 21:21:50 mfs-master02-rj.btr ipfail: : info: Aborted delayed giveup (4)
故障恢复后，mfsmaster主备服务器的状态：
mfsmaster主：
1
2
3
4
5
6
7
8
9
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.27/24 brd 10.1.1.255 scope global em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master01-rj.btr, 2015-06-25 17:20:34
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:7556 nr:1264 dw:2380 dr:9530 al:6 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfsmaster备：
1
2
3
4
5
6
7
8
9
10
11
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
mfs 5345 10 21:17 ? 00:00:00 /usr/local/mfs/sbin/mfsmaster start
# ip a|grep em4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:1336 nr:7556 dw:106668492 dr:24076 al:16 bm:6421 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
状态说明：服务没有发生回切状态！
mfs客户端数据恢复信息：
客户端服务不受任何影响
故障2：关闭mfsmaster的heartbeat心跳中断模拟故障：拔掉heartbeat的心跳线em2故障发生之前的mfsmaster主备服务器状态：mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
mfs 14129 10 21:25 ? 00:00:00 /usr/local/mfs/sbin/mfsmaster start
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.27/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master01-rj.btr, 2015-06-25 17:20:34
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:8312 nr:2556 dw:4428 dr:11447 al:8 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfsmaster备：
1
2
3
4
5
6
7
8
9
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:2556 nr:8400 dw:106670556 dr:24084 al:16 bm:6421 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
故障发生时，mfsmaster主备服务器的日志信息：mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Jun 25 21:30:09 mfs-master01-rj.btr heartbeat: : WARN: node mfs-master02-rj.btr: is dead
Jun 25 21:30:09 mfs-master01-rj.btr heartbeat: : WARN: No STONITH device configured.
Jun 25 21:30:09 mfs-master01-rj.btr ipfail: : info: Status update: Node mfs-master02-rj.btr now has status dead
Jun 25 21:30:09 mfs-master01-rj.btr heartbeat: : WARN: Shared disks are not protected.
Jun 25 21:30:09 mfs-master01-rj.btr heartbeat: : info: Resources being acquired from mfs-master02-rj.btr.
Jun 25 21:30:09 mfs-master01-rj.btr heartbeat: : info: Link mfs-master02-rj.btr:em2 dead.
harc(default): 2015/06/25_21:30:09 info: Running /etc/ha.d//rc.d/status status
mach_down(default): 2015/06/25_21:30:09 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default): 2015/06/25_21:30:09 info: mach_down takeover complete for node mfs-master02-rj.btr.
Jun 25 21:30:09 mfs-master01-rj.btr heartbeat: : info: mach_down takeover complete.
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:30:09 INFO:Running OK
Jun 25 21:30:09 mfs-master01-rj.btr heartbeat: : info: Local Resource acquisition completed.
Jun 25 21:30:10 mfs-master01-rj.btr ipfail: : info: NS: We are still alive!
Jun 25 21:30:10 mfs-master01-rj.btr ipfail: : info: Link Status update: Link mfs-master02-rj.btr/em2 now has status dead
Jun 25 21:30:13 mfs-master01-rj.btr ipfail: : info: Asking other side for ping node count.
Jun 25 21:30:13 mfs-master01-rj.btr ipfail: : info: Checking remote count of ping nodes.
mfsmaster备：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
Jun 25 21:30:09 mfs-master02-rj.btr heartbeat: : WARN: node mfs-master01-rj.btr: is dead
Jun 25 21:30:09 mfs-master02-rj.btr heartbeat: : WARN: No STONITH device configured.
Jun 25 21:30:09 mfs-master02-rj.btr ipfail: : info: Status update: Node mfs-master01-rj.btr now has status dead
Jun 25 21:30:09 mfs-master02-rj.btr ipfail: : info: Status update: Node mfs-master01-rj.btr now has status dead
Jun 25 21:30:09 mfs-master02-rj.btr heartbeat: : WARN: Shared disks are not protected.
Jun 25 21:30:09 mfs-master02-rj.btr heartbeat: : info: Resources being acquired from mfs-master01-rj.btr.
Jun 25 21:30:09 mfs-master02-rj.btr heartbeat: : info: Link mfs-master01-rj.btr:em2 dead.
harc(default): 2015/06/25_21:30:09 info: Running /etc/ha.d//rc.d/status status
Jun 25 21:30:09 mfs-master02-rj.btr heartbeat: : info: No local resources to acquire.
mach_down(default): 2015/06/25_21:30:09 info: Taking over resource group IPaddr::10.1.1.26/24/em4
ResourceManager(default): 2015/06/25_21:30:09 info: Acquiring resource group: mfs-master01-rj.btr IPaddr::10.1.1.26/24/em4 drbddisk::drbd Filesystem::/dev/drbd0::/usr/local/mfs::ext4 mfsmaster
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:30:10 INFO:Resource is stopped
ResourceManager(default): 2015/06/25_21:30:10 info: Running /etc/ha.d/resource.d/IPaddr 10.1.1.26/24/em4 start
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:30:10 INFO: Adding inet address 10.1.1.26/24 with broadcast address 10.1.1.255 to device em4
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:30:10 INFO: Bringing device em4 up
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:30:10 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.1.1.26 em4 10.1.1.26 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:30:10 INFO:Success
ResourceManager(default): 2015/06/25_21:30:10 info: Running /etc/ha.d/resource.d/drbddisk drbd start
Jun 25 21:30:10 mfs-master02-rj.btr ipfail: : info: NS: We are still alive!
Jun 25 21:30:10 mfs-master02-rj.btr ipfail: : info: Link Status update: Link mfs-master01-rj.btr/em2 now has status dead
Jun 25 21:30:12 mfs-master02-rj.btr ipfail: : info: Asking other side for ping node count.
Jun 25 21:30:12 mfs-master02-rj.btr ipfail: : info: Checking remote count of ping nodes.
ResourceManager(default): 2015/06/25_21:30:22 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk
ResourceManager(default): 2015/06/25_21:30:22 CRIT: Giving up resources due to failure of drbddisk::drbd
ResourceManager(default): 2015/06/25_21:30:22 info: Releasing resource group: mfs-master01-rj.btr IPaddr::10.1.1.26/24/em4 drbddisk::drbd Filesystem::/dev/drbd0::/usr/local/mfs::ext4 mfsmaster
ResourceManager(default): 2015/06/25_21:30:22 info: Running /etc/ha.d/resource.d/mfsmasterstop
ResourceManager(default): 2015/06/25_21:30:22 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /usr/local/mfs ext4 stop
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:30:22 INFO: Running stop for /dev/drbd0 on /usr/local/mfs
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:30:22 INFO:Success
ResourceManager(default): 2015/06/25_21:30:22 info: Running /etc/ha.d/resource.d/drbddisk drbd stop
ResourceManager(default): 2015/06/25_21:30:22 info: Running /etc/ha.d/resource.d/IPaddr 10.1.1.26/24/em4 stop
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:30:22 INFO: IP status = ok, IP_CIP=
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:30:22 INFO:Success
mach_down(default): 2015/06/25_21:30:22 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default): 2015/06/25_21:30:22 info: mach_down takeover complete for node mfs-master01-rj.btr.
Jun 25 21:30:22 mfs-master02-rj.btr heartbeat: : info: mach_down takeover complete.
hb_standby(default): 2015/06/25_21:30:52 Going standby .
Jun 25 21:30:53 mfs-master02-rj.btr heartbeat: : info: mfs-master02-rj.btr wants to go standby
Jun 25 21:31:04 mfs-master02-rj.btr heartbeat: : WARN: No reply to standby request.Standby request cancelled.
故障发生后，mfsmaster主备服务器的状态：mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
mfs 14129 10 21:25 ? 00:00:00 /usr/local/mfs/sbin/mfsmaster start
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.27/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master01-rj.btr, 2015-06-25 17:20:34
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:8564 nr:2556 dw:4680 dr:11447 al:8 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfsmaster备：
1
2
3
4
5
6
7
8
9
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:2556 nr:8564 dw:106670720 dr:24084 al:16 bm:6421 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfs客户端数据恢复信息：
1
2
3
4
5
6
7
8
9
Thu Jun 25 20:17:10 CST 2015 4015
Thu Jun 25 20:17:11 CST 2015 4016
Thu Jun 25 20:17:12 CST 2015 4017
Thu Jun 25 20:17:13 CST 2015 4018
Thu Jun 25 20:17:14 CST 2015 4019
Thu Jun 25 20:17:15 CST 2015 4020 ##########
Thu Jun 25 20:18:33 CST 2015 4021 ##########
Thu Jun 25 20:18:34 CST 2015 4022
Thu Jun 25 20:18:35 CST 2015 4023
恢复故障2：关闭mfsmaster的heartbeat心跳中断恢复heartbeat心跳：插上mfsmaster主的em2网线故障恢复时，mfsmaster主备服务器的日志信息：mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
Jun 25 21:32:43 mfs-master01-rj.btr heartbeat: : CRIT: Cluster node mfs-master02-rj.btr returning after partition.
Jun 25 21:32:43 mfs-master01-rj.btr heartbeat: : info: For information on cluster partitions, See URL: http://linux-ha.org/wiki/Split_Brain
Jun 25 21:32:43 mfs-master01-rj.btr heartbeat: : WARN: Deadtime value may be too small.
Jun 25 21:32:43 mfs-master01-rj.btr heartbeat: : info: See FAQ for information on tuning deadtime.
Jun 25 21:32:43 mfs-master01-rj.btr heartbeat: : info: URL: http://linux-ha.org/wiki/FAQ#Heavy_Load
Jun 25 21:32:43 mfs-master01-rj.btr heartbeat: : info: Link mfs-master02-rj.btr:em2 up.
Jun 25 21:32:43 mfs-master01-rj.btr heartbeat: : WARN: Late heartbeat: Node mfs-master02-rj.btr: interval 164510 ms
Jun 25 21:32:43 mfs-master01-rj.btr heartbeat: : info: Status update for node mfs-master02-rj.btr: status active
Jun 25 21:32:43 mfs-master01-rj.btr ipfail: : info: Link Status update: Link mfs-master02-rj.btr/em2 now has status up
Jun 25 21:32:43 mfs-master01-rj.btr ipfail: : info: Status update: Node mfs-master02-rj.btr now has status active
harc(default): 2015/06/25_21:32:43 info: Running /etc/ha.d//rc.d/status status
Jun 25 21:32:45 mfs-master01-rj.btr heartbeat: : WARN: Shutdown delayed until current resource activity finishes.
Jun 25 21:32:46 mfs-master01-rj.btr heartbeat: : info: Heartbeat shutdown in progress. (12255)
Jun 25 21:32:46 mfs-master01-rj.btr heartbeat: : info: Received shutdown notice from 'mfs-master02-rj.btr'.
Jun 25 21:32:46 mfs-master01-rj.btr heartbeat: : info: Resource takeover cancelled - shutdown in progress.
Jun 25 21:32:46 mfs-master01-rj.btr heartbeat: : info: Giving up all HA resources.
ResourceManager(default): 2015/06/25_21:32:46 info: Releasing resource group: mfs-master01-rj.btr IPaddr::10.1.1.26/24/em4 drbddisk::drbd Filesystem::/dev/drbd0::/usr/local/mfs::ext4 mfsmaster
ResourceManager(default): 2015/06/25_21:32:46 info: Running /etc/ha.d/resource.d/mfsmasterstop
ResourceManager(default): 2015/06/25_21:32:48 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /usr/local/mfs ext4 stop
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:32:48 INFO: Running stop for /dev/drbd0 on /usr/local/mfs
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:32:48 INFO: Trying to unmount /usr/local/mfs
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:32:48 INFO: unmounted /usr/local/mfs successfully
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:32:48 INFO:Success
ResourceManager(default): 2015/06/25_21:32:48 info: Running /etc/ha.d/resource.d/drbddisk drbd stop
ResourceManager(default): 2015/06/25_21:32:48 info: Running /etc/ha.d/resource.d/IPaddr 10.1.1.26/24/em4 stop
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:32:48 INFO: IP status = ok, IP_CIP=
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:32:48 INFO:Success
Jun 25 21:32:48 mfs-master01-rj.btr heartbeat: : info: All HA resources relinquished.
Jun 25 21:32:49 mfs-master01-rj.btr heartbeat: : info: killing /usr/lib64/heartbeat/ipfail process group 12269 with signal 15
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: killing HBFIFO process 12258 with signal 15
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: killing HBWRITE process 12259 with signal 15
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: killing HBREAD process 12260 with signal 15
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: killing HBWRITE process 12261 with signal 15
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: killing HBREAD process 12262 with signal 15
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: killing HBWRITE process 12263 with signal 15
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: killing HBREAD process 12264 with signal 15
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: killing HBWRITE process 12265 with signal 15
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: killing HBREAD process 12266 with signal 15
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: Core process 12260 exited. 9 remaining
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: Core process 12261 exited. 8 remaining
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: Core process 12265 exited. 7 remaining
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: Core process 12262 exited. 6 remaining
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: Core process 12258 exited. 5 remaining
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: Core process 12263 exited. 4 remaining
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: Core process 12264 exited. 3 remaining
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: Core process 12266 exited. 2 remaining
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: Core process 12259 exited. 1 remaining
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: mfs-master01-rj.btr Heartbeat shutdown complete.
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: Heartbeat restart triggered.
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: Restarting heartbeat.
Jun 25 21:32:51 mfs-master01-rj.btr heartbeat: : info: Performing heartbeat restart exec.
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: Pacemaker support: false
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : WARN: Logging daemon is disabled --enabling logging daemon is recommended
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: **************************
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: Configuration validated. Starting heartbeat 3.0.4
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: heartbeat: version 3.0.4
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: Heartbeat generation: 1435221816
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: glib: UDP multicast heartbeat started for group 225.0.0.192 port 694 interface em2 (ttl=1 loop=0)
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: glib: ping heartbeat started.
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: glib: ping heartbeat started.
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: glib: ping heartbeat started.
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: G_main_add_TriggerHandler: Added signal manual handler
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: G_main_add_TriggerHandler: Added signal manual handler
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: G_main_add_SignalHandler: Added signal handler for signal 17
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: Local status now set to: 'up'
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: Status update for node 10.1.1.27: status ping
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: Link 10.1.1.27:10.1.1.27 up.
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: Link 10.1.1.1:10.1.1.1 up.
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: Status update for node 10.1.1.1: status ping
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: Status update for node 10.1.1.28: status ping
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: Link 10.1.1.28:10.1.1.28 up.
Jun 25 21:33:02 mfs-master01-rj.btr heartbeat: : info: Link mfs-master02-rj.btr:em2 up.
Jun 25 21:33:03 mfs-master01-rj.btr heartbeat: : info: Status update for node mfs-master02-rj.btr: status up
Jun 25 21:33:03 mfs-master01-rj.btr heartbeat: : info: Status update for node mfs-master02-rj.btr: status active
Jun 25 21:33:03 mfs-master01-rj.btr heartbeat: : info: Comm_now_up(): updating status to active
Jun 25 21:33:03 mfs-master01-rj.btr heartbeat: : info: Local status now set to: 'active'
Jun 25 21:33:03 mfs-master01-rj.btr heartbeat: : info: Starting child client "/usr/lib64/heartbeat/ipfail" (497,496)
Jun 25 21:33:03 mfs-master01-rj.btr heartbeat: : info: Starting "/usr/lib64/heartbeat/ipfail" as uid 497gid 496 (pid 15369)
harc(default): 2015/06/25_21:33:03 info: Running /etc/ha.d//rc.d/status status
harc(default): 2015/06/25_21:33:03 info: Running /etc/ha.d//rc.d/status status
Jun 25 21:33:14 mfs-master01-rj.btr heartbeat: : info: local resource transition completed.
Jun 25 21:33:14 mfs-master01-rj.btr heartbeat: : info: Initial resource acquisition complete (T_RESOURCES(us))
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:33:14 INFO:Resource is stopped
Jun 25 21:33:14 mfs-master01-rj.btr heartbeat: : info: Local Resource acquisition completed.
harc(default): 2015/06/25_21:33:14 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
ip-request-resp(default): 2015/06/25_21:33:14 received ip-request-resp IPaddr::10.1.1.26/24/em4 OK yes
ResourceManager(default): 2015/06/25_21:33:14 info: Acquiring resource group: mfs-master01-rj.btr IPaddr::10.1.1.26/24/em4 drbddisk::drbd Filesystem::/dev/drbd0::/usr/local/mfs::ext4 mfsmaster
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:33:14 INFO:Resource is stopped
ResourceManager(default): 2015/06/25_21:33:14 info: Running /etc/ha.d/resource.d/IPaddr 10.1.1.26/24/em4 start
Jun 25 21:33:14 mfs-master01-rj.btr heartbeat: : info: remote resource transition completed.
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:33:14 INFO: Adding inet address 10.1.1.26/24 with broadcast address 10.1.1.255 to device em4
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:33:14 INFO: Bringing device em4 up
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:33:15 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.1.1.26 em4 10.1.1.26 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:33:15 INFO:Success
ResourceManager(default): 2015/06/25_21:33:15 info: Running /etc/ha.d/resource.d/drbddisk drbd start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:33:15 INFO:Resource is stopped
ResourceManager(default): 2015/06/25_21:33:15 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /usr/local/mfs ext4 start
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:33:15 INFO: Running start for /dev/drbd0 on /usr/local/mfs
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:33:15 INFO:Success
ResourceManager(default): 2015/06/25_21:33:15 info: Running /etc/ha.d/resource.d/mfsmasterstart
Jun 25 21:33:17 mfs-master01-rj.btr ipfail: : info: Ping node count is balanced.
Jun 25 21:33:18 mfs-master01-rj.btr ipfail: : info: Giving up foreign resources (auto_failback).
Jun 25 21:33:18 mfs-master01-rj.btr ipfail: : info: Delayed giveup in 4 seconds.
Jun 25 21:33:22 mfs-master01-rj.btr ipfail: : info: giveup() called (timeout worked)
Jun 25 21:33:22 mfs-master01-rj.btr heartbeat: : info: mfs-master01-rj.btr wants to go standby
Jun 25 21:33:23 mfs-master01-rj.btr heartbeat: : info: standby: mfs-master02-rj.btr can take our foreign resources
Jun 25 21:33:23 mfs-master01-rj.btr heartbeat: : info: give up foreign HA resources (standby).
Jun 25 21:33:23 mfs-master01-rj.btr heartbeat: : info: foreign HA resource release completed (standby).
Jun 25 21:33:23 mfs-master01-rj.btr heartbeat: : info: Local standby process completed .
Jun 25 21:33:23 mfs-master01-rj.btr heartbeat: : WARN: 1 lost packet(s) for
Jun 25 21:33:23 mfs-master01-rj.btr heartbeat: : info: remote resource transition completed.
Jun 25 21:33:23 mfs-master01-rj.btr heartbeat: : info: No pkts missing from mfs-master02-rj.btr!
Jun 25 21:33:23 mfs-master01-rj.btr heartbeat: : info: Other node completed standby takeover of foreign resources.
mfsmaster备：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
Jun 25 21:32:43 mfs-master02-rj.btr heartbeat: : CRIT: Cluster node mfs-master01-rj.btr returning after partition.
Jun 25 21:32:43 mfs-master02-rj.btr heartbeat: : info: For information on cluster partitions, See URL: http://linux-ha.org/wiki/Split_Brain
Jun 25 21:32:43 mfs-master02-rj.btr heartbeat: : WARN: Deadtime value may be too small.
Jun 25 21:32:43 mfs-master02-rj.btr heartbeat: : info: See FAQ for information on tuning deadtime.
Jun 25 21:32:43 mfs-master02-rj.btr heartbeat: : info: URL: http://linux-ha.org/wiki/FAQ#Heavy_Load
Jun 25 21:32:43 mfs-master02-rj.btr heartbeat: : info: Link mfs-master01-rj.btr:em2 up.
Jun 25 21:32:43 mfs-master02-rj.btr heartbeat: : WARN: Late heartbeat: Node mfs-master01-rj.btr: interval 164060 ms
Jun 25 21:32:43 mfs-master02-rj.btr heartbeat: : info: Status update for node mfs-master01-rj.btr: status active
Jun 25 21:32:43 mfs-master02-rj.btr ipfail: : info: Link Status update: Link mfs-master01-rj.btr/em2 now has status up
Jun 25 21:32:43 mfs-master02-rj.btr ipfail: : info: Status update: Node mfs-master01-rj.btr now has status active
harc(default): 2015/06/25_21:32:43 info: Running /etc/ha.d//rc.d/status status
Jun 25 21:32:45 mfs-master02-rj.btr heartbeat: : info: Heartbeat shutdown in progress. (6134)
Jun 25 21:32:45 mfs-master02-rj.btr heartbeat: : info: Giving up all HA resources.
ResourceManager(default): 2015/06/25_21:32:45 info: Releasing resource group: mfs-master01-rj.btr IPaddr::10.1.1.26/24/em4 drbddisk::drbd Filesystem::/dev/drbd0::/usr/local/mfs::ext4 mfsmaster
ResourceManager(default): 2015/06/25_21:32:45 info: Running /etc/ha.d/resource.d/mfsmasterstop
ResourceManager(default): 2015/06/25_21:32:45 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /usr/local/mfs ext4 stop
Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:32:45 INFO: Running stop for /dev/drbd0 on /usr/local/mfs
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2015/06/25_21:32:45 INFO:Success
ResourceManager(default): 2015/06/25_21:32:45 info: Running /etc/ha.d/resource.d/drbddisk drbd stop
ResourceManager(default): 2015/06/25_21:32:45 info: Running /etc/ha.d/resource.d/IPaddr 10.1.1.26/24/em4 stop
IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:32:45 INFO: IP status = no, IP_CIP=
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.1.1.26): 2015/06/25_21:32:45 INFO:Success
Jun 25 21:32:45 mfs-master02-rj.btr heartbeat: : info: All HA resources relinquished.
Jun 25 21:32:46 mfs-master02-rj.btr heartbeat: : info: killing /usr/lib64/heartbeat/ipfail process group 6173 with signal 15
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: killing HBWRITE process 6138 with signal 15
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: killing HBREAD process 6139 with signal 15
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: killing HBWRITE process 6140 with signal 15
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: killing HBREAD process 6141 with signal 15
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: killing HBWRITE process 6142 with signal 15
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: killing HBREAD process 6143 with signal 15
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: killing HBWRITE process 6144 with signal 15
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: killing HBREAD process 6145 with signal 15
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: killing HBFIFO process 6137 with signal 15
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: Core process 6139 exited. 9 remaining
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: Core process 6143 exited. 8 remaining
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: Core process 6144 exited. 7 remaining
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: Core process 6141 exited. 6 remaining
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: Core process 6137 exited. 5 remaining
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: Core process 6145 exited. 4 remaining
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: Core process 6140 exited. 3 remaining
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: Core process 6142 exited. 2 remaining
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: Core process 6138 exited. 1 remaining
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: mfs-master02-rj.btr Heartbeat shutdown complete.
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: Heartbeat restart triggered.
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: Restarting heartbeat.
Jun 25 21:32:47 mfs-master02-rj.btr heartbeat: : info: Performing heartbeat restart exec.
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: Pacemaker support: false
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : WARN: Logging daemon is disabled --enabling logging daemon is recommended
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: **************************
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: Configuration validated. Starting heartbeat 3.0.4
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: heartbeat: version 3.0.4
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: Heartbeat generation: 1435221806
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: glib: UDP multicast heartbeat started for group 225.0.0.192 port 694 interface em2 (ttl=1 loop=0)
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: glib: ping heartbeat started.
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: glib: ping heartbeat started.
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: glib: ping heartbeat started.
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: G_main_add_TriggerHandler: Added signal manual handler
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: G_main_add_TriggerHandler: Added signal manual handler
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: G_main_add_SignalHandler: Added signal handler for signal 17
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: Local status now set to: 'up'
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: Status update for node 10.1.1.28: status ping
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: Link 10.1.1.28:10.1.1.28 up.
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: Status update for node 10.1.1.27: status ping
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: Link 10.1.1.27:10.1.1.27 up.
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: Link 10.1.1.1:10.1.1.1 up.
Jun 25 21:32:58 mfs-master02-rj.btr heartbeat: : info: Status update for node 10.1.1.1: status ping
Jun 25 21:33:02 mfs-master02-rj.btr heartbeat: : info: Link mfs-master01-rj.btr:em2 up.
Jun 25 21:33:02 mfs-master02-rj.btr heartbeat: : info: Status update for node mfs-master01-rj.btr: status up
harc(default): 2015/06/25_21:33:02 info: Running /etc/ha.d//rc.d/status status
Jun 25 21:33:02 mfs-master02-rj.btr heartbeat: : info: Comm_now_up(): updating status to active
Jun 25 21:33:02 mfs-master02-rj.btr heartbeat: : info: Local status now set to: 'active'
Jun 25 21:33:02 mfs-master02-rj.btr heartbeat: : info: Starting child client "/usr/lib64/heartbeat/ipfail" (497,496)
Jun 25 21:33:02 mfs-master02-rj.btr heartbeat: : info: Starting "/usr/lib64/heartbeat/ipfail" as uid 497gid 496 (pid 7461)
Jun 25 21:33:03 mfs-master02-rj.btr heartbeat: : info: Status update for node mfs-master01-rj.btr: status active
harc(default): 2015/06/25_21:33:03 info: Running /etc/ha.d//rc.d/status status
Jun 25 21:33:10 mfs-master02-rj.btr ipfail: : info: Status update: Node mfs-master01-rj.btr now has status active
Jun 25 21:33:14 mfs-master02-rj.btr heartbeat: : info: remote resource transition completed.
Jun 25 21:33:14 mfs-master02-rj.btr heartbeat: : info: remote resource transition completed.
Jun 25 21:33:14 mfs-master02-rj.btr heartbeat: : info: Initial resource acquisition complete (T_RESOURCES(us))
Jun 25 21:33:14 mfs-master02-rj.btr heartbeat: : info: No local resources to acquire.
Jun 25 21:33:14 mfs-master02-rj.btr ipfail: : info: Asking other side for ping node count.
Jun 25 21:33:18 mfs-master02-rj.btr ipfail: : info: No giveup timer to abort.
Jun 25 21:33:22 mfs-master02-rj.btr heartbeat: : info: mfs-master01-rj.btr wants to go standby
Jun 25 21:33:23 mfs-master02-rj.btr heartbeat: : info: standby: acquire resources from mfs-master01-rj.btr
Jun 25 21:33:23 mfs-master02-rj.btr heartbeat: : info: acquire local HA resources (standby).
Jun 25 21:33:23 mfs-master02-rj.btr heartbeat: : info: local HA resource acquisition completed (standby).
Jun 25 21:33:23 mfs-master02-rj.btr heartbeat: : info: Standby resource acquisition done .
Jun 25 21:33:24 mfs-master02-rj.btr heartbeat: : info: remote resource transition completed.
故障恢复后，mfsmaster主备服务器的状态：mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
mfs 16004 10 21:33 ? 00:00:00 /usr/local/mfs/sbin/mfsmaster start
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.27/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master01-rj.btr, 2015-06-25 17:20:34
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:9932 nr:2556 dw:6048 dr:13372 al:9 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfsmaster备：
1
2
3
4
5
6
7
8
9
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:2556 nr:10040 dw:106672196 dr:24084 al:16 bm:6421 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfs客户端数据恢复信息：
1
2
3
4
5
6
7
8
Thu Jun 25 20:19:48 CST 2015 4095
Thu Jun 25 20:19:49 CST 2015 4096
Thu Jun 25 20:19:50 CST 2015 4097
Thu Jun 25 20:19:51 CST 2015 4098 ###############
Thu Jun 25 20:20:30 CST 2015 4099 ###############
Thu Jun 25 20:20:31 CST 2015 4100
Thu Jun 25 20:20:32 CST 2015 4101
Thu Jun 25 20:20:33 CST 2015 4102
故障3：mfsmaster主的drbd同步网络断开模拟故障：拔掉mfsmaster主的drbd同步的网卡em3故障发生之前的mfsmaster主备服务器状态：mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
mfs 16004 10 21:33 ? 00:00:00 /usr/local/mfs/sbin/mfsmaster start
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.27/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master01-rj.btr, 2015-06-25 17:20:34
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:10348 nr:2556 dw:6464 dr:13372 al:9 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfsmaster备：
1
2
3
4
5
6
7
8
9
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:2556 nr:10376 dw:106672532 dr:24084 al:16 bm:6421 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
故障发生时，mfsmaster主备服务器的日志信息：
mfsmaster主：无日志
mfsmaster主：
无日志
故障发生后，mfsmaster主备服务器的状态：mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
mfs 16004 10 21:33 ? 00:00:00 /usr/local/mfs/sbin/mfsmaster start
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.27/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master01-rj.btr, 2015-06-25 17:20:34
0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
ns:10508 nr:2556 dw:7004 dr:13372 al:9 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:308
mfsmaster备：
1
2
3
4
5
6
7
8
9
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----
ns:2556 nr:10500 dw:106672656 dr:24084 al:16 bm:6421 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfs客户端数据恢复信息：
1
2
3
4
5
6
7
Thu Jun 25 20:24:15 CST 2015 4321
Thu Jun 25 20:24:16 CST 2015 4322
Thu Jun 25 20:24:17 CST 2015 4323 ######
Thu Jun 25 20:24:23 CST 2015 4324 ######
Thu Jun 25 20:24:24 CST 2015 4325
Thu Jun 25 20:24:25 CST 2015 4326
Thu Jun 25 20:24:26 CST 2015 4327
备注：
当mfsmaster的drbd同步网络断开之后，在系统的中断上会报如下信息：
恢复故障3：mfsmaster主的drbd同步网络断开模拟故障：插上mfsmaster主的drbd同步的网卡em3故障恢复时，mfsmaster主备服务器的日志信息：mfsmaster主：无日志mfsmaster备：无日志故障恢复后，mfsmaster主备服务器的状态：mfsmaster主：
1
2
3
4
5
6
7
8
9
10
11
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
mfs 16004 10 21:33 ? 00:00:00 /usr/local/mfs/sbin/mfsmaster start
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.27/24 brd 10.1.1.255 scope global em4
inet 10.1.1.26/24 brd 10.1.1.255 scope global secondary em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master01-rj.btr, 2015-06-25 17:20:34
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:9932 nr:2556 dw:6048 dr:13372 al:9 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfsmaster备：
1
2
3
4
5
6
7
8
9
# ps -ef|awk '$0!~"awk"&&$0~"mfs"{print}'
# ip a|grepem4
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
inet 10.1.1.28/24 brd 10.1.1.255 scope global em4
# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@mfs-master02-rj.btr, 2015-06-25 17:20:33
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:2556 nr:10040 dw:106672196 dr:24084 al:16 bm:6421 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
mfs客户端数据恢复信息：
客户端无任何干扰
经过本次测试，目前线上的MooseFS运行稳定，灾难测试很成功，服务的高可用性得到了验证。
页:
[1]