Heartbeat+DRBD+NFS构建高可用文件共享存储
Heartbeat+DRBD+NFS构建高可用文件共享存储
一．试验简介
本实验部署DRBD + HEARDBEAT + NFS 环境，建立一个高可用(HA)的文件服务器集群。在方案中，通
过DRBD保证了服务器数据的完整性和一致性。DRBD类似于一个网络RAID-1功能。当你将数据写入本
地文件系统时，数据还将会被发送到网络中另一台主机上，以相同的形式记录在一个另文件系统中。主节
点与备节点的数据可以保证实时相互同步。当本地主服务器出现故障时，备份服务器上还会保留有一份相
同的数据，可以继续使用。在高可用(HA)中使用DRBD功能，可以代替使用一个共享盘阵。因为数据同时
存在于本地主服务器和备份服务器上。切换时，远程主机只要使用它上面的那份备份数据，就可以继续提
供主服务器上相同的服务，并且client用户对主服务器的故障无感知。
二．实验环境
虚拟机操作系统：Red Hat Enterprise Linux Server release 5.4
A:master计算机名：Server1 etho：192.168.20.101
B：slave计算机名：Server2eth0:192.168.20.102
Heartbeat 虚拟IP：192.168.20.188
两台服务器将/dev/sda4互为镜像
两台服务器/etc/export配置相同
同步服务器时间
# hwclock –s
两个主机的hosts文件里包含2台主机的ip地址和主机名称
# echo "192.168.20.101 server1" >>/etc/hosts
# echo "192.168.20.102 server2" >>/etc/hosts
# echo "192.168.20.101 server1" >>/etc/hosts
# echo "192.168.20.102 server2" >>/etc/hosts
在server1和server2做一下操作
配置yum服务
# mkdir /mnt/cdrom
# mount /dev/cdrom /mnt/cdrom/
mount: block device /dev/cdrom is write-protected, mounting read-only
# vim /etc/yum.repos.d/rhel-debuginfo.repo
name=Red Hat Enterprise Linux server
baseurl=file:///mnt/cdrom/Server
enabled=1
gpgcheck=1
gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release
name=Red Hat Enterprise Linux cluster
baseurl=file:///mnt/cdrom/Cluster
enabled=1
gpgcheck=1
gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release
name=Red Hat Enterprise Linux clusterstorage
baseurl=file:///mnt/cdrom/ClusterStorage
enabled=1
gpgcheck=1
gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release
三．磁盘分区
两主机分区大小一致
# fdisk –l
Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks IdSystem
/dev/sda1 * 1 13 104391 83Linux
/dev/sda2 14 1288 10241437+83Linux
/dev/sda3 1289 1415 1020127+82Linux swap / Solaris
# fdisk /dev/sda
The number of cylinders for this disk is set to 2610.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Selected partition 4
First cylinder (1416-2610, default 1416):
Using default value 1416
Last cylinder or +size or +sizeM or +sizeK (1416-2610, default 2610): +1G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
# partprobe /dev/sda
# cat /proc/partitions
major minor#blocksname
8 0 20971520 sda
8 1 104391 sda1
8 2 10241437 sda2
8 3 1020127 sda3
8 4 987997 sda4
四．DRDB配置
Server1和server2 做一下相同操作
4.1.安装 drbd包
# ll
total 424
drwxr-xr-x 2 root root 4096 Mar2 14:26 Desktop
-rw------- 1 root root 1300 Mar2 20:34 anaconda-ks.cfg
-rw-r--r-- 1 root root 221868 May7 16:15 drbd83-8.3.8-1.el5.centos.i386.rpm
-rw-r--r-- 1 root root35768 Mar2 20:33 install.log
-rw-r--r-- 1 root root 4713 Mar2 20:33 install.log.syslog
-rw-r--r-- 1 root root 125974 May7 16:15 kmod-drbd83-8.3.8-1.el5.centos.i686.rpm
-rw-r--r-- 1 root root 240 Mar2 12:38 scsrun.log
# yum localinstall *.rpm --nogpgcheck –y
加载DRBD模块
# modprobe drbd
查看模块加载
# lsmod |grep drbd
drbd 2285284
4.2.修改配置文件
# cat /etc/drbd.conf
#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd83/drbd.conf
#
# cp /usr/share/doc/drbd83-8.3.8/drbd.conf /etc/
cp: overwrite `/etc/drbd.conf'? y
# vim /etc/drbd.conf
# cd /etc/drbd.d/
# ll
total 4
-rwxr-xr-x 1 root root 1418 Jun42010 global_common.conf
# cp global_common.conf global_common.conf.bak
# vim global_common.conf
1 global {
2 usage-count no;
3 #minor-count dialog-refresh disable-ip-verification
4 }
5
6 common {
7 protocol C;
8
9 startup{
10 wfc-timeout 120;
11 degr-wfc-timeout 120;
12 }
13 disk {
14 on-io-error detach;
15 fencing resource-only;
16
17 }
18 net {
19 cram-hmac-alg "sha1"
20 shared-secret "mydrbdlab”;
21 }
22 syncer {
23 rate 100M;
24 }
25 }
# vim web.res
resource web{
on server1{
device/dev/drdb0;
disk /dev/sda4;
address 192.168.20.101:7789;
meta-disk internal;
}
on server2{
device /dev/drbd0;
disk /dev/sda4;
address 192.168.20.102:7789;
meta-disk internal;
}
}
4.3.检测配置文件
# dd if=/dev/zero bs=1M count=1 of=/dev/sda4;sync
# drbdadm create-md web
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
启动服务时两主机服务同时启动
# service drbd start
Starting DRBD resources: [
web
Found valid meta data in the expected location, 1011703808 bytes into /dev/sda4.
d(web) s(web) n(web) ]outdated-wfc-timeout has to be shorter than degr-wfc-timeout
outdated-wfc-timeout implicitly set to degr-wfc-timeout (120s)
# drbd-overview
0:webConnected Secondary/Secondary Inconsistent/Inconsistent C r----
#
# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:987928
# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:987928
此时发现两server没有主服务
创建文件夹
# mkdir /data
4.4.只在主节点上执行
把server1 作为主节点
# cd /etc/drbd.d/
# drbdadm -- --overwrite-data-of-peer primary web
# drbd-overview
0:webSyncSource Primary/Secondary UpToDate/Inconsistent C r----
[=================>..] sync'ed: 91.8% (84248/987928)K delay_probe: 70
格式化
# mkfs -t ext3 -L drbdweb /dev/drbd0
# mkdir /mnt/1
# mount /dev/drbd0 /mnt/1
# df -h
Filesystem SizeUsed Avail Use% Mounted on
/dev/sda2 9.5G2.6G6.4G29% /
/dev/sda1 99M 12M 83M12% /boot
tmpfs 97M 0 97M 0% /dev/shm
/dev/hdc 2.8G2.8G 0 100% /media/RHEL_5.4 i386 DVD
/dev/hdc 2.8G2.8G 0 100% /mnt/cdrom
/dev/drbd0 950M 18M885M 2% /mnt/1
查看状态
# service drbd status
drbd driver loaded OK; device status:
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
m:rescs ro ds pmountedfstype
0:webConnected Primary/SecondaryUpToDate/UpToDateC/mnt/1 ext
# service drbd status
drbd driver loaded OK; device status:
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
m:rescs ro ds pmountedfstype
0:webConnected Secondary/PrimaryUpToDate/UpToDateC
此时发现server1为主服务，server2为辅助服务
五．NFS配置
两台服务器都修改nfs配置文件如下
# vim /etc/exports
/data *(rw,sync,insecure,no_root_squash,no_wdelay)
# service portmap start
Starting portmap:
# chkconfig portmap on
# service nfs start
# chkconfig nfs on
两台服务器都修改nfs启动脚本。
将/etc/init.d/nfs 脚本中的stop部分中的killproc nfsd -2修改为 -9
# vim /etc/init.d/nfs
stop)
# Stop daemons.
echo -n $"Shutting down NFS mountd: "
killproc rpc.mountd
echo
echo -n $"Shutting down NFS daemon: "
killproc nfsd -9
echo
if [ -n "$RQUOTAD" -a "$RQUOTAD" != "no" ]; then
echo -n $"Shutting down NFS quotas: "
killproc rpc.rquotad
RETVAL=$?
echo
修改Heartbeat配置
在server1和server2做以下操作
安装Heartbeat套件
# yum localinstall heartbeat-2.1.4-9.el5.i386.rpm heartbeat-pils-2.1.4-10.el5.i386.rpm heartbeat-stonith-2.1.4-10.el5.i386.rpm libnet-1.1.4-3.el5.i386.rpm perl-MailTools-1.77-1.el5.noarch.rpm --nogpgcheck –y
# cd /usr/share/doc/heartbeat-2.1.4/
Heartbeat配置
# cp authkeys haresources ha.cf /etc/ha.d/
Ha.cf配置，2台机子不一样的地方标红，填写对方服务器IP地址
# vim ha.cf
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 10
udpport 694
ucast eth0 192.168.20.102
ping 192.168.20.1
auto_failback off
node server1
node server2
haresources配置
2台机子相同
# echo "server1 IPaddr::192.168.20.188/24/eth0 drbddisk::web Filesystem::/dev/drbd0::/data::ext3 killnfsd" >> haresources
authkeys 配置相同
auth 1
1 crc
#2 sha1 HI!
#3 md5 Hello!
Killnfsd配置相同（手动编写）
# echo "killall -9 nfsd; /etc/init.d/nfs restart; exit 0 " >> resource.d/killnfsd
设置文档权限
# chmod 600 /etc/ha.d/authkeys
# chmod 755 /etc/ha.d/resource.d/killnfsd
开启Heartbeat服务
# service heartbeat start
Starting High-Availability services:
2012/05/08_01:28:21 INFO:Resource is stopped
# chkconfig heartbeat on
六．测试
1.在测试机上将192.168.10.188：/data挂载到本地/mnt/nfs
# mkdir /mnt/nfs
# mount 192.168.20.188:/data /mnt/nfs/
2.在测试机上创建shell，二秒一个
# vim /mnt/test.sh
while true
do
echo ---\> trying touch x : `date`
touch x
echo \ trying touch x : Tue May 8 13:14:54 CST 2012
trying touch x : Tue May 8 13:14:56 CST 2012
trying touch x : Tue May 8 13:14:58 CST 2012
trying touch x : Tue May 8 13:15:00 CST 2012
touch: cannot touch `x': Stale NFS file handle
trying touch x : Tue May 8 13:15:07 CST 2012
trying touch x : Tue May 8 13:15:09 CST 2012
trying touch x : Tue May 8 13:15:11 CST 2012
页:
[1]