moosefs 解决 mfsmaster单点故障keepalived 辅助
一 服务器IP及应用角色二 原理及思路
三 moosefs安装 配置 管理
四metadata.mfs.bak的定时备份
五keepalived 安装 配置 管理
六 故障切换测试
七 参考资料联系方式
八 本教程相关资料下载
一 服务器IP及应用角色
192.168.20.237 keepalived MASTERmfsmastermfschuckservermfsclient
192.168.20.38 keepalived Slave mfsmetaloggermfschuckservermfsclient mfsmaster(备份)
192.168.20.233 虚拟IP
二 原理及思路
1 mfsmaster的故障恢复在1.6.5版本后可以由mfsmetalogger产生的日志文件 changelog_ml.*.mfs和metadata.mfs.back由命令mfsmetarestore恢复
2 定时从mfsmaster 获取 metadata.mfs.back 文件用于master恢复
3 192.168.11.237 keepalived MASTER run检测到 mfsmaster 故障的时候 停止keepalived
4 192.168.20.38 keepalived 状态从backup转到 master 并触发 notify_master事件执行恢复并启动mfsmaster脚本
5 整个切换在2~5秒内完成 根据检测时间间隔。
6nagios 监控mfs状态可见参考资料
三 moosefs安装 配置 管理
mfsmaster mfsmetaloggermfschuckserver mfsclient 四个角色的安装 如下 只是使用的配置文件不同
[*] useradd -s /sbin/nologin mfs
[*] mkdir -p /opt/software
[*] cd /opt/software
[*] wget http://sourceforge.net/projects/fuse/files/fuse-2.X/2.8.5/fuse-2.8.5.tar.gz
[*] wget http://sourceforge.net/projects/moosefs/files/moosefs/1.6.20/mfs-1.6.20-2.tar.gz
[*]
[*] tar zxvf fuse-2.8.5.tar.gz
[*] cd fuse-2.8.5
[*] ./configure --prefix=/usr
[*] make && make install
[*]
[*] cd /opt/software
[*] tar zxvf mfs-1.6.20-2.tar.gz
[*] cd mfs-1.6.20-2
[*] ./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs
[*] make && make install
[*] cd ..
复制代码
MFS相关的配置文件有mfsmaster.cfg mfsexports.cfg
#192.168.20.237和192.168.20.38 mfsmaster 配置文件如下
vi/usr/local/mfs/etc/mfsmaster.cfg
[*] # WORKING_USER = mfs
复制代码
#192.168.20.237192.168.20.38mfschunkserver 使用相同的mfschunkserver.cfg mfshdd.cfg配置文件如下
vi /usr/local/mfs/etc/mfschunkserver.cfg
[*] # WORKING_USER = mfs
[*] # WORKING_GROUP = mfs
[*] # SYSLOG_IDENT = mfschunkserver
[*] # LOCK_MEMORY = 0
[*] # NICE_LEVEL = -19
[*] # DATA_PATH = /usr/local/mfs/var/mfs
[*] # MASTER_RECONNECTION_DELAY = 5
[*] # BIND_HOST = *
[*] MASTER_HOST = 192.168.20.233
[*] # MASTER_PORT = 9420
[*] # MASTER_TIMEOUT = 60
[*] # CSSERV_LISTEN_HOST = *
[*] # CSSERV_LISTEN_PORT = 9422
[*] # HDD_CONF_FILENAME = /usr/local/mfs/etc/mfshdd.cfg
[*] # HDD_TEST_FREQ = 10
[*] # deprecated, to be removed in MooseFS 1.7
[*] # LOCK_FILE = /var/run/mfs/mfschunkserver.lock
[*] # BACK_LOGS = 50
[*] # CSSERV_TIMEOUT = 5
复制代码
我这边mfschunk存储数据的目录是 /data/9ai9.net
vim/usr/local/mfs/etc/mfshdd.cfg
[*] /data/9ai9.net
复制代码
mfs各角色的管理命令如下 更详细的说明参见参考资料
/usr/local/mfs/sbin/mfsmaster start|stop|restart|reload
/usr/local/mfs/sbin/mfsmetalogger start|stop|restart|reload
/usr/local/mfs/sbin/mfschunkserver start|stop|restart|reload
客户端将mfschunk共享挂载到/home/lixy命令
modprobe fuse
/usr/local/mfs/bin/mfsmount /home/lixy -H 192.168.20.233
四metadata.mfs.bak的定时备份
将主MFSMASTER的metadata.mfs.bak定时备份到 备用的MFSmaster服务器上
这里scp方法每5分钟从主服务器取metadata.mfs.bak备份一次 用于启动
在crontab -e里添加以下内容
[*] */5 * * * *scp192.168.20.237:/usr/local/mfs/var/mfs/metadata.mfs.back/usr/local/mfs/var/mfs
复制代码
五keepalived 安装 配置 管理
[*] wget http://www.keepalived.org/software/keepalived-1.2.2.tar.gz
[*] tar zxvf keepalived-1.2.2.tar.gz
[*] cd keepalived-1.2.2
[*] ./configure --prefix=/
[*] make
[*] make install
复制代码
#配置说明
1 KEEPALIVED MASTER 每2秒检测一次 mfsmaster状态
2 KEEPAVLIED BACKUP 不检测MFS状态 只等待KEEPALIVED事情触发恢复
192.168.20.237 keepavled MASTER主配置文件
[*] ! Configuration File for keepalived
[*] global_defs {
[*] router_id 9ai9.net
[*] }
[*] vrrp_script check_run {
[*] script "/root/keepalived_check_mfsmaster.sh"
[*] interval 2
[*] }
[*] vrrp_sync_group VG1 {
[*] group {
[*] VI_1
[*] }
[*] }
[*] vrrp_instance VI_1 {
[*] state MASTER
[*] interface eth0
[*] virtual_router_id 88
[*] priority 100
[*] advert_int 1
[*] nopreempt
[*] authentication {
[*] auth_type PASS
[*] auth_pass 9ai9.net
[*] }
[*] track_script {
[*] check_run
[*] }
[*] virtual_ipaddress {
[*] 192.168.20.233
[*] }
[*] }
复制代码
vim /root/keepalived_check_mfsmaster.sh
注：这里 MFSMASTER_HOST的IP地址要是192.168.20.237 非VIP
[*] #!/bin/bash
[*] #write by qhappy QQ 87179505
[*] #20110310
[*]
[*] MFSMASTER_HOST=192.168.20.237
[*] MFSMASTER_PORT=9420
[*] if [ `uname -i` = x86_64 ];then
[*] CHECK_MASTER=/root/qhappy_check_tcp_64
[*] else
[*] CHECK_MASTER=/root/qhappy_check_tcp_32
[*] fi
[*] CHECK_TIME=2
[*] #mfsmasteris working MFS_OK is 1 , mfsmaster down MFS_OK is 0
[*] MFS_OK=1
[*] function check_mfsmaster (){
[*] $CHECK_MASTER -H $MFSMASTER_HOST -p $MFSMASTER_PORT >/dev/null 2>&1
[*] if [ $? = 0 ] ;then
[*] MFS_OK=1
[*] else
[*] MFS_OK=0
[*] fi
[*] return $MFS_OK
[*] }
[*] while [ $CHECK_TIME -ne 0 ]
[*] do
[*] let "CHECK_TIME -= 1"
[*] check_mfsmaster
[*] if [ $MFS_OK = 1 ] ; then
[*] CHECK_TIME=0
[*] exit 0
[*] fi
[*] if [ $MFS_OK -eq 0 ] &&[ $CHECK_TIME -eq 0 ]
[*] then
[*] /etc/init.d/keepalived stop
[*] exit 1
[*] fi
[*] done
复制代码
chmod 755/root/keepalived_check_mfsmaster.sh
192.168.20.38keepalived BACKUP的 配置
vim/etc/keepalived/keepalived.conf
[*] ! Configuration File for keepalived
[*] global_defs {
[*] router_id 9ai9.net
[*] }
[*]
[*] vrrp_sync_group VG1 {
[*] group {
[*] VI_1
[*] }
[*] notify_master "/root/keepalived_notify.sh master"
[*] notify_backup "/root/keepalived_notify.sh backup"
[*] }
[*] vrrp_instance VI_1 {
[*] state BACKUP
[*] interface eth0
[*] virtual_router_id 88
[*] priority 80
[*] advert_int 1
[*] authentication {
[*] auth_type PASS
[*] auth_pass 9ai9.net
[*] }
[*]
[*] virtual_ipaddress {
[*] 192.168.20.233
[*] }
[*] }
复制代码
vim /root/keepalived_notify.sh
[*] #!/bin/bash
[*] #write by qhappy QQ 87179505
[*] #20110310
[*]
[*] MFS_HOME=/usr/local/mfs
[*] MFSMARSTER=${MFS_HOME}/sbin/mfsmaster
[*] MFSMETARESTORE=${MFS_HOME}/sbin/mfsmetarestore
[*] MFS_DATA_PATH=${MFS_HOME}/var/mfs
[*]
[*] function backup2master(){
[*] $MFSMETARESTORE -m ${MFS_DATA_PATH}/metadata.mfs.bak -o ${MFS_DATA_PATH}/metadata.mfs $MFS_DATA_PATH/changelog_ml*.mfs
[*] #$MFSMETARESTORE -a
[*] $MFSMARSTER start
[*] }
[*]
[*] function master2backup(){
[*] $MFSMARSTER stop
[*] }
[*]
[*] function ERROR(){
[*] echo "USAGE: keepalived_notify.sh master|backup "
[*] }
[*]
[*] case $1 in
[*] master)
[*] backup2master
[*] ;;
[*] backup)
[*] master2backup
[*] ;;
[*] *)
[*] ERROR
[*] ;;
[*] esac
复制代码
chomd 755/root/keepalived_notify.sh
各服务角色 启动顺序
1 mfsmaster
2 keepalived
3 mfsmetalogger
4 mfschunckserver
5 客户端挂载
关闭顺序和上面的相反
五 故障切换测试
192.168.20.237 执行
/usr/local/mfs/sbin/mfsmaster stop
192.168.20.237日志截图
192.168.20.38 日志截图
续上图
六 参考资料
moosefs官网 http://www.moosefs.org/
ChinaUnix http://bbs.chinaunix.net/thread-1644309-1-1.html
笔者 QQ 87179505
七 资料下载
下载的内容为keepalived 检测mfsmaster 状态的文件 脚本中定义的使用路径是 /root 也请放在这个路径下
页:
[1]