ceph安装配置文档（centos6.6）
目录一：简介... 1
二：部署环境介绍... 1
三：集群配置准备工作... 2
四：安装ceph软件包... 6
五：安装ceph对象网关... 9
六、搭建ceph集群... 10
6.1、配置mon节点... 10
6.2：添加osd节点... 13
6.2.1、添加第一块osd节点... 13
6.3：添加元数据服务器... 17
6.3.1、添加第一个元数据服务器... 17
七：安装client端RBD、cephFS挂载演示... 18
7.1：客户端内核要求、安装装准备... 18
7.2、RBD方式挂载... 19
7.3、cephFS挂载... 20
八：检查ceph集群状态常用命令... 23
8.1、磁盘压力测试软件推荐... 23
九:故障排查... 23
一：简介Ceph 生态系统架构可以划分为四部分：
1. Clients：客户端（数据用户）2. mds：Metadata servercluster，元数据服务器（缓存和同步分布式元数据）3. osd：Object storagecluster，对象存储集群（将数据和元数据作为对象存储，执行其它关键职能）4. mon：Cluster monitors，集群监视器（执行监视功能）二：部署环境介绍
主机名公网IP（eth0）
私网IP（eth1）
双网卡绑定 bond0运行服务cpu内存操作系统
内核磁盘备注
client172.16.4.17 Clientcpu: e5520*2
5506/4核/8线程*24g*4 CentOS6.6
3.18.2
node01
172.16.2.27
mon、mds、osd
cpu: e5520*2
5506/4核/8线程*24g*4 CentOS6.6
1T sas 7500转*4
第一块装系统
node02
172.16.2.28
mon、mds、osd
cpu: e5520*2
5506/4核/8线程*24g*4 CentOS6.6
1T sas 7500转*4
第一块装系统
node03
172.16.2.29
mon、mds、osd
cpu: e5520*2
5506/4核/8线程*24g*4 CentOS6.6
1T sas 7500转*4
第一块装系统
三：集群配置准备工作 node01节点执行3.1：创建工作文件夹，建立ip地址list，为文件传输做准备mkdir /workspace/#cd /workspace/# vi cephlist.txt 主机列表clientnode01node02node03
主机ssh登陆证书ssh-keygencd .ssh/ssh-copy-id 127.1将配置完成的证书传输到其它服务器sync –avp *172.17.2.27:/root/.ssh/sync –avp *172.17.2.28:/root/.ssh/sync –avp *172.17.2.29:/root/.ssh/
3.2：系统基础设置，建议整理为脚本，批量传输到所有集群主机执行3.2.1.修改SELINUX模式sed -i's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinuxsed -i's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config3.2.2.set max userprocessessed -i's/1024/102400/' /etc/security/limits.d/90-nproc.confecho "* -sigpending 256612" >> /etc/security/limits.d/90-nproc.conf
3.2.3.set ulimitecho "ulimit-SHn 102400" >> /etc/rc.localcat >>/etc/security/limits.conf << EOF* soft nofile 102400* hard nofile 102400* soft nproc 102400* - sigpending256612* hard nproc 102400EOF#执行profilesource/etc/profile3.3: 修改host文件# vi /etc/hosts追加内容如下：172.16.4.17 client172.16.2.27 node01172.16.2.28 node02172.16.2.29 node03172.16.4.240 cephNaNmind.com将host文件传输到其它主机for ip in $(cat/workspace/cephlist.txt);do echo -----$ip-----;rsync -avp /etc/hosts $ip:/etc/;done3.4：安装centos的yum源软件包 =>全部节点安装3.4.1:升级基础包rpm -Uvhhttp://mirrors.sohu.com/centos/6 ... -2.1.el6.noarch.rpmrpm -Uvh http://mirrors.sohu.com/centos/6 ... 2-16.el6.x86_64.rpmrpm -Uvhhttp://mirrors.sohu.com/centos/6 ... 6.centos.noarch.rpmrpm -Uvhhttp://mirrors.sohu.com/centos/6 ... 0-30.el6.noarch.rpm
3.4.2：配置一个国内yum源vi/etc/yum.repos.d/my.repo 详见批注3.4.3：添加时间同步定时任务vi /etc/crontab3.4.4：时间同步,此处是我们的内部时间源20 */1 * * * root/usr/sbin/ntpdate 172.16.5.100 && /sbin/hwclock –wfor ip in $(cat /workspace/cephlist.txt);doecho -----$ip-----;rsync -avp --delete /etc/crontab $ip:/etc/;done4:5：传输yum源文件到其它节点服务器--delete 删除那些DST中SRC没有的文件for ip in $(cat/workspace/cephlist.txt);do echo -----$ip-----;rsync -avp --delete/etc/yum.repos.d $ip:/etc/;done
3.5、更新yum源（所有节点执行）Screenyum clean all&&yum update –y3.6、添加ceph yum源或者自制yum源；在所有节点安装ceph的所有的yum源 （所有节点执行）3.6.1、安装软件包key(1)、release.asckeyrpm --import'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc'（2）、autobuild.asckeyrpm --import'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc'
3.6.2、安装ceph附加包源ceph-extras.repo，设置priority=2，确保新的包（如qemu）优先级比标准包的高。vi/etc/yum.repos.d/ceph-extras.repo 详见附件#制作私有yum源下载软件包地址wget -S -c -r -np-L http://ceph.com/packages/ceph-extras/rpm/centos6/3.6.3、安装ceph源vi/etc/yum.repos.d/ceph.repo ##制作私有yum源下载软件包地址 wget -S -c -r -np -L http://ceph.com/rpm/rhel6/
3.6.4、安装ceph的apache yum源vi/etc/yum.repos.d/ceph-apache.repo ##制作私有yum源下载软件包地址wget -S -c -r -np-L http://gitbuilder.ceph.com/apach ... 4-basic/ref/master/
3.6.5、添加ceph-fastcgi.repofile to /etc/yum.repos.dvi/etc/yum.repos.d/mod_fastcgi.repo 详见批注##制作私有yum源下载软件包地址wget -S -c -r -np-L http://gitbuilder.ceph.com/mod_f ... 4-basic/ref/master/
3.7：如果制作内网yum源，完成上面给出的下载链接后，进行如下配置：3.7.1：删除无用文件cd/usr/share/nginx/htmlfind ./ -nameindex.html* >1.txtfor i in $(cat1.txt);do rm -rf $i;donefind ./ -nameindex.html*
3.7.2：调整目录结构下载至本地的软件包，目录结构根据实际情况作调整，3.7.3：添加、修改nginx配置： location / { root/usr/share/nginx/html; autoindex on;3.7.4、修改yum源，repo文件，将yum请求的地址改为私有地址；例如：vi/etc/yum.repos.d/ceph-extras.repo name=Ceph ExtrasSourcesbaseurl=http://172.16.2.240/ceph-extras/packages/ceph-extras/rpm/centos6/SRPMSenabled=1priority=2gpgcheck=0type=rpm-mdgpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
3.8、安装国内epel yum源rpm -Uvhhttp://mirrors.sohu.com/fedora-e ... ease-6-8.noarch.rpm备：rpm -Uvhhttp://dl.fedoraproject.org/pub/ ... ease-6-8.noarch.rpm
官方推荐rpm安装ceph，安装下面插件：3.9：安装yum-plugin-priorities. yum install yum-plugin-priorities -yyum update –y3.9.1：确认配置文件内容cat/etc/yum/pluginconf.d/priorities.confenabled = 1 name=CentOS-6 -Base - 163.combaseurl=http://mirrors.163.com/centos/6/os/$basearch/#mirrorlist=http://mirrorlist.centos.org/?release=6&arch=$basearch&repo=osgpgcheck=1gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
#released updatesname=CentOS-6 -Updates - 163.combaseurl=http://mirrors.163.com/centos/6/updates/$basearch/#mirrorlist=http://mirrorlist.centos.org/?release=6&arch=$basearch&repo=updatesgpgcheck=1gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
#additionalpackages that may be usefulname=CentOS-6 -Extras - 163.combaseurl=http://mirrors.163.com/centos/6/extras/$basearch/#mirrorlist=http://mirrorlist.centos.org/?release=6&arch=$basearch&repo=extrasgpgcheck=1gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
#additionalpackages that extend functionality of existing packagesname=CentOS-6 -Plus - 163.combaseurl=http://mirrors.163.com/centos/6/centosplus/$basearch/#mirrorlist=http://mirrorlist.centos.org/?release=6&arch=$basearch&repo=centosplusgpgcheck=1enabled=0gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
#contrib -packages by Centos Usersname=CentOS-6 -Contrib - 163.combaseurl=http://mirrors.163.com/centos/6/contrib/$basearch/#mirrorlist=http://mirrorlist.centos.org/?release=6&arch=$basearch&repo=contribgpgcheck=1enabled=0gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
name=Ceph Extras Sourcesbaseurl=http://ceph.com/packages/ceph-extras/rpm/centos6/SRPMSenabled=1priority=2gpgcheck=1type=rpm-mdgpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
name=Ceph packagesfor $basearchbaseurl=http://ceph.com/rpm/rhel6/$basearchenabled=1gpgcheck=1type=rpm-mdgpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
name=Ceph noarchpackagesbaseurl=http://ceph.com/rpm/rhel6/noarchenabled=1gpgcheck=1type=rpm-mdgpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
name=Ceph sourcepackagesbaseurl=http://ceph.com/rpm/rhel6/SRPMSenabled=0gpgcheck=1type=rpm-mdgpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
name=Apache noarchpackages for Cephbaseurl=http://gitbuilder.ceph.com/apache2-rpm-centos6-x86_64-basic/ref/master/enabled=1priority=2gpgcheck=1type=rpm-mdgpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc
name=Apache sourcepackages for Cephbaseurl=http://gitbuilder.ceph.com/apache2-rpm-centos6-x86_64-basic/ref/master/enabled=0priority=2gpgcheck=1type=rpm-mdgpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc
name=FastCGIbasearch packages for Cephbaseurl=http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos6-x86_64-basic/ref/masterenabled=1priority=2yumgpgcheck=1type=rpm-mdgpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc
name=FastCGInoarch packages for Cephbaseurl=http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos6-x86_64-basic/ref/masterenabled=1priority=2gpgcheck=1type=rpm-mdgpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc
name=FastCGI sourcepackages for Cephbaseurl=http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos6-x86_64-basic/ref/masterenabled=0priority=2gpgcheck=1type=rpm-mdgpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc
name=Ceph ExtrasSourcesbaseurl=http://ceph.com/packages/ceph-extras/rpm/centos6/SRPMSenabled=1priority=2gpgcheck=1type=rpm-mdgpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
四：安装ceph软件包4.1、安装ceph部署机 ，使用ceph-deploy方式安装部署yum install -yceph-deploy4.2、安装ceph存储集群（1）、安装ceph必备软件 ==allyum install -y snappy leveldb gdisk python-argparse gperftools-libs（2）、安装ceph软件==allyum install -y ceph4.3、安装ceph对象网关==allyum install -y yum-plugin-prioritiesyum update -y4.4、安装apachefastcgi ==all（1）、安装apachefastcgi 需要yum install httpd mod_fastcgi，安装之前先执行下面的操作yuminstall -y httpd mod_fastcgi
4.4.1、确认 FastCGImodule启用.cat/etc/httpd/conf.d/fastcgi.conf|grep mod_fastcgi.so 输出结果如下：LoadModulefastcgi_module modules/mod_fastcgi.so
4.4.2、确认 Rewritemodule 启用cat/etc/httpd/conf/httpd.conf|grep mod_rewrite.so输出结果如下：LoadModulerewrite_module modules/mod_rewrite.so
4.4.3、编辑配置文件httpd.conf,修改httpd.confServerName为本机主机名vim/etc/httpd/conf/httpd.confecho"ServerName$HOSTNAME">> /etc/httpd/conf/httpd.conf
4.4.4、启动httpd进程/etc/init.d/httpd restart
4.5、安装SSL (安装此步骤有报错）yum install -ymod_ssl openssl4.5.1、创建证书生成私钥.openssl genrsa-out ca.key 20484.5.2、生成CSR文件openssl req -new-key ca.key -out ca.csr
ountry Name (2 letter code)
使用国际标准组织(ISO)国码格式，填写2个字母的国家代号。中国请填写CN。
State or Province Name (full name)
省份，比如填写Shanghai
Locality Name (eg, city)
城市，比如填写Shanghai
Organization Name (eg, company)
组织单位，比如填写公司名称的拼音
Organizational Unit Name (eg, section)
比如填写IT Dept
Common Name (eg, your websites domainname):
行使 SSL 加密的网站地址。请注意这里并不是单指您的域名，而是直接使用 SSL 的网站名称 例如:pay.abc.com。 一个网站这里定义是：
abc.com 是一个网站；
www.abc.com 是另外一个网站；
pay.abc.com 又是另外一个网站。
Email Address
邮件地址，可以不填
A challenge password
可以不填
An optional company name
可以不填
4.5.3、生成证书opensslx509 -req -days 365 -in ca.csr -signkey ca.key -out ca.crt
4.5.4、copy生成证书至apache监听位置.cp ca.crt/etc/pki/tls/certscp ca.key/etc/pki/tls/private/ca.keycp ca.csr/etc/pki/tls/private/ca.csr
4.5.5、修改apache ssl配置文件 /etc/httpd/conf.d/ssl.conf.vi +105/etc/httpd/conf.d/ssl.conf将105SSLCertificateFile /etc/pki/tls/certs/localhost.crt改为： 105 SSLCertificateFile/etc/pki/tls/certs/ca.crt将 112 SSLCertificateKeyFile/etc/pki/tls/private/localhost.key 改为 112 SSLCertificateKeyFile/etc/pki/tls/private/ca.key4.5.6：将证书文件传到其它主机--delete 删除那些DST中SRC没有的文件for ip in $(cat/workspace/cephlist.txt);do echo -----$ip-----;rsync -avp --delete /etc/pki/tls/certs$ip:/etc/pki/tls/;donefor ip in $(cat/workspace/cephlist.txt);do echo -----$ip-----;rsync -avp --delete /etc/pki/tls/private$ip:/etc/pki/tls/;donefor ip in $(cat/workspace/cephlist.txt);do echo -----$ip-----;rsync -avp --delete/etc/httpd/conf.d/ssl.conf $ip:/etc/httpd/conf.d/ssl.conf;done
4.5.7、批量重启服务for ip in $(cat /workspace/cephlist.txt);do echo-----$ip-----;ssh root@$ip /etc/init.d/httpd restart;done
官网步骤，写了host文件略过啦（10）、AddWildcard to DNS （The address of the DNS must also be specified in the Cephconfiguration file with the rgw dns name = {hostname} setting.address=/.ceph-node/192.168.0.1五：安装ceph对象网关5.1、基础软件包yum install -y ceph-radosgwceph5.2、安装ceph对象网关代理yum install -yradosgw-agent5.3、为块存储安装虚拟化软件5.3.1、如果之前有qemu模块先删除，确保之后安装的是最完善的yumremove -y qemu-kvm qemu-kvm-tools qemu-img5.3.2、卸载后重新安装qemuyum install -yqemu-kvm qemu-kvm-tools qemu-img5.4、安装qemu客户代理yum install -yqemu-guest-agent
5.5、在所有节点安装ceph依赖的软件及模块yum install *argparse* -yyum install redhat-lsb-yyum install xfs* -y
############官方文档无此内容############(4)、安装libvirt软件包yuminstall -y libvirt以上步骤参考配置连接http://docs.ceph.com/docs/master/install/install-ceph-gateway/配置ceph对象网关http://docs.ceph.com/docs/master/radosgw/config/
六、搭建ceph集群 http://docs.ceph.com/docs/master/radosgw/config/6.1、配置mon节点6.1.1、建立第一个mon节点1、登录监控节点node01节点ls /etc/ceph #查看ceph配置文件目录是否存在2、创建ceph配置文件并配置ceph配置文件内的内容3、创建一个ceph配置文件touch/etc/ceph/ceph.conf 4、执行此命令可以得到一个唯一的标识，作为ceph集群IDuuidgenbc5ec6c4-3cc9-48ad-94d9-cee3cd75952a
fsid = bc5ec6c4-3cc9-48ad-94d9-cee3cd75952a #此标识就是上面得到的，把此条命令加入ceph的配置文件mon initialmembers = node01,node02,node03 #node01、node02、node03作为ceph集群的监控节点，把此条命令加入到ceph配置文件mon host = 172.16.2.27,172.16.2.28, 172.16.2.29#监控节点的地址，把此条命令加入ceph的配置文件中
5、ceph配置文件预览vi /etc/ceph/ceph.conffsid = bc5ec6c4-3cc9-48ad-94d9-cee3cd75952amon initial members = node01,node02,node03mon host = 172.16.2.27, 172.16.2.28, 172.16.2.29public network = 172.16.2.0/24auth cluster required = cephxauth service required = cephxauth client required = cephxosd journal size = 1024filestore xattr use omap = trueosd pool default size = 3osd pool default min size = 1osd crush chooseleaf type = 1osd_mkfs_type = xfsmax mds = 5mds max file size = 100000000000000mds cache size = 1000000mon osd down out interval = 900 #设置osd节点down后900s，把此osd节点逐出ceph集群，把之前映射到此节点的数据映射到其他节点。cluster_network = 172.16.2.0/24mon clock drift allowed = .50 #把时钟偏移设置成0.5s，默认是0.05s,由于ceph集群中存在异构PC，导致时钟偏移总是大于0.05s，为了方便同步直接把时钟偏移设置成0.5s
6、在node01创建各种密钥#为监控节点创建管理密钥ceph-authtool--create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
#为ceph amin用户创建管理集群的密钥并赋予访问权限ceph-authtool--create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin--set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
#添加client.adminkey到 ceph.mon.keyringceph-authtool/tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
验证cat/tmp/ceph.mon.keyring
7、在node01监控节点创建一个mon数据目录mkdir -p/var/lib/ceph/mon/ceph-node01
8、在node01创建一个boot引导启动osd的keymkdir -p/var/lib/ceph/bootstrap-osd/ceph-authtool -C /var/lib/ceph/bootstrap-osd/ceph.keyring
9、在node01节点上初始化mon节点， ceph-mon--mkfs -i node01 --keyring /tmp/ceph.mon.keyring
10、为了防止重新被安装创建一个空的done文件touch/var/lib/ceph/mon/ceph-node01/done
11、创建一个空的初始化文件touch/var/lib/ceph/mon/ceph-node01/sysvinit
12、启动ceph进程/sbin/serviceceph -c /etc/ceph/ceph.conf start mon.node01
10、查看asok mon状态# ceph--cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node01.asok mon_status
6.1.2、添加其他mon（2）节点：1、复制node01节点的/etc/ceph目录到node022、复制node01节点上的/var/lib/ceph/bootstrap-osd/ceph.keyring文件到node023、复制node01节点上的/tmp/ceph.mon.keyring4、在node02节点上新建一个/var/lib/ceph/bootstrap-osd/目录5、在node02节点上建立一个/var/lib/ceph/mon/ceph-node02目录6、在node02节点上初始化mon节点7、为了防止重新被安装创建一个空的done文件8、创建一个空的初始化文件9、启动ceph进程
scp /etc/ceph/*node02:/etc/ceph/ scp /var/lib/ceph/bootstrap-osd/ceph.keyringnode02:/var/lib/ceph/bootstrap-osd/scp /tmp/ceph.mon.keyring node02:/tmp/
mkdir /var/lib/ceph/bootstrap-osd/mkdir -p /var/lib/ceph/mon/ceph-node02ceph-mon--mkfs -i node02 --keyring /tmp/ceph.mon.keyringtouch/var/lib/ceph/mon/ceph-node02/donetouch/var/lib/ceph/mon/ceph-node02/sysvinit/sbin/service ceph -c/etc/ceph/ceph.conf start mon.node02验证状态：ceph --cluster=ceph--admin-daemon /var/run/ceph/ceph-mon.node02.asok mon_status
6.1.3、添加其他mon（3）节点：添加方法同 6.1.2 步骤，注意修改node名称6.2：添加osd节点6.2.1、添加第一块osd节点1、创建一个OSD，生成一个osdnumber# ceph osd create0
2、为osd节点创建一个osd目录#mkdir -p/var/lib/ceph/osd/ceph-0
3、格式化已准备好的osd硬盘（格式化为xfs格式）# mkfs.xfs-f /dev/sdb
4、创建osd挂载目录#mount /dev/sdb /var/lib/ceph/osd/ceph-0#mount -o remount,user_xattr /var/lib/ceph/osd/ceph-0查看挂载的情况#mount/dev/sda2 on /type ext4 (rw)proc on /proc typeproc (rw)/dev/sdbon /var/lib/ceph/osd/ceph-1 type xfs (rw,user_xattr)
5、把上面的挂载信息写入分区表# vi /etc/fstab/dev/sdb /var/lib/ceph/osd/ceph-0xfsdefaults 0 0/dev/sdb /var/lib/ceph/osd/ceph-0xfsremount,user_xattr0 0
6、初始化osd数据目录# ceph-osd-i 0 --mkfs --mkkey2015-04-0720:43:08.092553 7fab822b2800 -1 journal FileJournal::_open: disabling aio fornon-block journal.Use journal_force_aioto force use of aio anyway2015-04-0720:43:08.181608 7fab822b2800 -1 journal FileJournal::_open: disabling aio fornon-block journal.Use journal_force_aioto force use of aio anyway2015-04-0720:43:08.202946 7fab822b2800 -1 filestore(/var/lib/ceph/osd/ceph-0) could notfind 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory2015-04-0720:43:08.236009 7fab822b2800 -1 created object store /var/lib/ceph/osd/ceph-0journal /var/lib/ceph/osd/ceph-0/journal for osd.0 fsidfe3655d0-80d9-4dcf-a2c8-dcd1caf819ff2015-04-0720:43:08.236076 7fab822b2800 -1 auth: error reading file:/var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring:(2) No such file or directory2015-04-0720:43:08.236182 7fab822b2800 -1 created new key in keyring/var/lib/ceph/osd/ceph-0/keyring
6、注册osd的认证密钥ceph auth addosd. osd 'allow *' mon 'allow profile osd' -i/var/lib/ceph/osd/ceph-/keyring
#ceph auth add osd.0osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-0/keyring
查看认证节点ceph auth listosd.1 key:AQDwSsZUQMkeGRAAmYYw9CoUUdK73IjDSH/gnw== caps: allow profile osd7、为此osd节点创建一个crushmap#ceph osd crush add-bucket node01 host
8、防放置 Node01 到根路径#ceph osd crush move node01 root=defaultmoved item id -2name 'node01' to location {root=default} in crush map
9、放置osd.0到 bucketnode01#ceph osd crush add osd.0 1.0 host=node01add item id 0 name'osd.0' weight 1 at location {host=node01} to crush map
10、创建一个初始化目录#touch /var/lib/ceph/osd/ceph-0/sysvinit
11、启动osd进程/etc/init.d/cephstart osd.0
12、查看osd目录树#ceph osd tree# id weight type name up/down reweight-1 1 root default-2 1 host node010 1 osd.0 up 1
6.2.2、添加第二个osd节点 #1、创建一个OSD，生成一个osd number2、为osd节点创建一个osd目录3、格式化已准备好的osd硬盘，并挂在到上一步创建的osd目录（格式化为xfs格式）4、挂在目录5、初始化osd数据目录6、注册osd的认证密钥7、为此osd节点创建一个crush map8、Place the Ceph Node under the root default9、放置osd.0到 bucket node0110、创建一个初始化目录11、启动osd进程 、对应参数如下
ceph osd createmkdir -p/var/lib/ceph/osd/ceph-1mkfs.xfs-f /dev/sdbmount/dev/sdb /var/lib/ceph/osd/ceph-1mount -oremount,user_xattr /var/lib/ceph/osd/ceph-1
ceph-osd-i 1 --mkfs --mkkey cephauth add osd.1 osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-1/keyringcephosd crush add-bucket node02 hostcephosd crush move node02 root=defaultcephosd crush add osd.1 1.0 host=node02touch/var/lib/ceph/osd/ceph-1/sysvinit
/etc/init.d/cephstart osd.1
12、把上面的挂载信息写入分区表vi /etc/fstab/dev/sdb /var/lib/ceph/osd/ceph-1xfsdefaults 0 0/dev/sdb /var/lib/ceph/osd/ceph-1xfsremount,user_xattr0 0
验证：mount/dev/sdb on/var/lib/ceph/osd/ceph-1 type xfs (rw,user_xattr)
13、查看osd目录树#ceph osd tree# id weight type name up/down reweight-1 2 root default-2 1 host node010 1 osd.0 up 1-3 1 host node021 1 osd.1 up 1
6.2.3、添加其它osd节点，参考6.2.1、6.2.2 步骤添加node01 服务器上的其它磁盘为osd 添加node02 服务器上的其它磁盘为osd 添加node03 服务器上的其它磁盘为osd
验证：mount
12、查看osd目录树#ceph osd tree
添加其它osd方法步骤同上添加删除osd参考文档http://docs.openfans.org/ceph/ceph4e2d658765876863/ceph-1/ceph-storage-cluster3010ceph5b5850a896c67fa43011/operations301064cd4f5c3011/adding-removing-osds3010589e52a0-52209664osd3011
6.3：添加元数据服务器6.3.1、添加第一个元数据服务器1、为mds元数据服务器创建一个目录#mkdir -p /var/lib/ceph/mds/ceph-node012、为bootstrap-mds客户端创建一个密钥注：(如果下面的密钥在目录里已生成可以省略此步骤）#ceph-authtool --create-keyring /var/lib/ceph/bootstrap-mds/ceph.keyring --gen-key-n client.bootstrap-mds
3、在ceph auth库中创建bootstrap-mds客户端，赋予权限添加之前创建的密钥 注（查看ceph auth list 用户权限认证列表 如果已有client.bootstrap-mds此用户，此步骤可以省略）#ceph auth add client.bootstrap-mds mon 'allow profile bootstrap-mds' -i/var/lib/ceph/bootstrap-mds/ceph.keyringadded key forclient.bootstrap-mds
4、在root家目录里创建ceph.bootstrap-mds.keyring文件touch/root/ceph.bootstrap-mds.keyring
5、把keyring/var/lib/ceph/bootstrap-mds/ceph.keyring里的密钥导入家目录下的ceph.bootstrap-mds.keyring文件里ceph-authtool--import-keyring /var/lib/ceph/bootstrap-mds/ceph.keyring ceph.bootstrap-mds.keyring
6、在ceph auth库中创建mds.node01用户，并赋予权限和创建密钥，密钥保存在/var/lib/ceph/mds/ceph-node01/keyring文件里ceph--cluster ceph --name client.bootstrap-mds --keyring/var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node01 osd'allow rwx' mds 'allow' mon 'allow profile mds' -o /var/lib/ceph/mds/ceph-node01/keyring
7、为mds创建一个初始化文件用于启动使用（此文件为空文件）# touch/var/lib/ceph/mds/ceph-node01/sysvinit8、为了防止重新被安装创建一个空的done文件#touch /var/lib/ceph/mds/ceph-node01/done9、情况mds服务进程#service ceph start mds.node01=== mds.node01 ===Starting Ceph mds.node01on node1...starting mds.node01at :/06.3.2：添加其它元数据服务器，参考6.3.1 步骤
添加第二个元数据服务器 添加第三个元数据服务器
1、查看osd目录树 ceph osd tree
查看集群状态 ceph -w七：安装client端RBD、cephFS挂载演示7.1：客户端内核要求、安装装准备7.1.1、升级client端内核A.操作系统使用CentOS6.6，因为系统已经包含xfs的支持可以直接使用不需要再次编译。B.由于CentOS6.5系统的内核为2.6.32，因此要关闭硬盘的写入缓存，若高于此版本不需要关闭。 i. #hdparm -W 0/dev/sdb 0
7.1.2：升级系统内核kernel 2.6.34以前的版本是没有Modulerbd的，把系统内核版本升级到最新rpm --import http://elrepo.org/RPM-GPG-KEY-elrepo.orgrpm -Uvhhttp://elrepo.org/elrepo-release-6-5.el6.elrepo.noarch.rpmyum--enablerepo=elrepo-kernel install kernel-ml -y安装完内核后修改/etc/grub.conf配置文件使修改配置文件中的 Default=1 toDefault=07.1.3、验证内核支持 [首先加载ceph.ko]#modprobe -l|grep cephkernel/fs/ceph/ceph.kokernel/net/ceph/libceph.ko#modprobeceph
机器重启后生效 init 6
7.2、RBD方式挂载7.2.1、安装ceph软件包
Yum install ceph –y
7.2.2、将node01配置文件copyclient
rsync -avp node01:/etc/ceph/* /etc/ceph/
7.2.3、新建一个cephpool
ceph osd pool create{pool-name} {pg-num} [{pgp-num}]
# cephosd pool create kman 256
7.2.3、在pool中新建一个镜像
# rbd create rbdtest-1--size 80960 -p kman
7.2.4、把镜像映射到pool块设备中
# rbd map rbdtest-1-p kman
取消映射块设备:
使用RBD命令要取消映射块设备镜像，指定rm选项，设备名称（即，一般以作为块设备映像名称相同）
sudo rbd unmap/dev/rbd/{poolname}/{imagename}
例如：
sudo rbd unmap/dev/rbd/rbd/foo
7.2.5、查看镜像映射map
# rbdshowmapped
id pool image snap device
0kman rbdtest-1 - /dev/rbd0
7.2.6、格式化映射的设备块
# mkfs.ext4 -m0/dev/rbd0
参数：-m reserved-blocks-percentage
保留块百分比,-m:设置预留空间
默认是5%。也就是2G的空间，文件系统只是有2G*95%
7.2.7、挂载新建的分区
#mkdir/rbdtest
# mount/dev/rbd0 /rbdtest
# df -h
/dev/sda5 1.7T 4.8G1.7T 1% /data
/dev/rbd0 78G 56M 78G1% /rbdtest
7.2.8、进入新建的分区并dd测试性能
# cd /rbdtest
7.2.9、其它参考命令
查看创建的池
ceph osd lspools
删除池
ceph osd pool delete kman kman--yes-i-really-really-mean-it
创建池参考资料
http://docs.openfans.org/ceph/ce ... 1/pools30106c603011
7.3、cephFS挂载参考文档
http://docs.openfans.org/ceph/ce ... eph5e2e52a998753011
创建cephfs文件系统
对于一个刚创建的MDS服务，虽然服务是运行的，但是它的状态直到创建 pools 以及文件系统的时候才会变为Active.
# ceph mds dump
dumped mdsmap epoch 1
epoch1
flags0
created 0.000000
modified 2015-01-26 21:49:35.368742
tableserver 0
root 0
session_timeout 0
session_autoclose 0
max_file_size 0
last_failure 0
last_failure_osd_epoch0
compat compat={},rocompat={},incompat={}
max_mds 0
in
up {}
failed
stopped
data_pools
metadata_pool 0
inline_data disabled
7.3.1、创建cephfs 存储池
ceph osd pool create {pool-name} {pg-num}[{pgp-num}]
{
Option implication:
{pool-name}--> 存储池名称，必须唯一。
{pg-num}--> 存储池拥有的归置组总数。
{pgp-num}--> 用于归置的归置组总数。
ceph osd pool create cephfs_data10
ceph osd pool createcephfs_metadata 10
ceph fs newleadorfscephfs_metadata cephfs_data
验证数据生成
# ceph mds dump
dumped mdsmap epoch 5
epoch5
flags0
created 2015-01-27 16:10:16.789446
modified 2015-01-27 16:10:21.504363
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 100000000000000
last_failure 0
last_failure_osd_epoch0
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeableranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds usesversioned encoding,6=dirfrag is stored in omap,8=no anchor table}
max_mds 5
in 0
up {0=4341}
failed
stopped
data_pools 2
metadata_pool 3
inline_data disabled
4341:172.16.2.29:6815/31531 'node03' mds.0.1 up:active seq 1493
7.3.2、客户端挂载
1、安装客户端挂载软件
yum install ceph-fuse -y
2、创建挂载目录
mkdir /mycephfs
3、模拟挂载
mount-t ceph node01:6789:/ /mycephfs -v -oname=admin,secret=AQCnrMZUaFsiCxAAXzM3aF9WjUBnwbN6PtvZEw==
返回结果：
parsing options:rw,name=admin,secret=AQCnrMZUaFsiCxAAXzM3aF9WjUBnwbN6PtvZEw==
4、证书来源位置
more /etc/ceph/ceph.client.admin.keyring #key证书来源
key = AQCnrMZUaFsiCxAAXzM3aF9WjUBnwbN6PtvZEw==
auid = 0
caps mds = "allow"
caps mon = "allow *"
caps osd = "allow *"
5、另外一种命令挂载方式
# mount-t ceph node01:6789:/ /mycephfs -v -oname=admin,secretfile=/etc/ceph/ceph.client.admin.keyring
6、若果有多个mon监控节点，可以挂载多可节点，保证了cephFS的安全行，当有一个节点down的时候不影响写入数据
# mount.cephnode01,node02,node03:/ /mycephfs -v -o name=admin,secret= AQCnrMZUaFsiCxAAXzM3aF9WjUBnwbN6PtvZEw==
7、验证挂载信息：
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 1.7T 4.8G1.7T 1% /data
/dev/rbd0 78G 56M78G 1% /rbdtest
172.16.2.27:6789:/8.2T9.8G 8.2T 1% /mycephfs
8、把挂载的信息写到fstab里
# vi /etc/fstab
172.16.2.27,172.16.2.28,172.16.2.29://mycephfscephname=admin,secret= AQCnrMZUaFsiCxAAXzM3aF9WjUBnwbN6PtvZEw==,noatime 0
9、指定key文件的调用方式
sudoceph-fuse -k ./ceph.client.admin.keyring -m 192.168.40.107:6789 ~/mycephfs
10、挂载后查看结果，注意观察类型
df-Th
11、取消挂载
取消挂载的操作如下:
sudoumount /mnt/mycephfs
八：检查ceph集群状态常用命令#ceph health //ceph健康状态#ceph status //ceph当前全部状态#ceph -w //实时监控ceph状态及变化#ceph osddump //所有osd详细状态#ceph osd tree //osd所在位置，及状态#cephquorum_status //mon优先级状态#ceph mon dump //mon节点状态#ceph mds dump //mds详细状态
8.1、磁盘压力测试软件推荐安装fio压测工具、Linux系统自带，yum即可安装
yum install fio –y
http://www.360doc.com/content/12/0809/11/2459_229177650.shtmlhttp://www.gongzi.org/linux-io-test.html
九:故障排查9.1、创建osd节点挂载磁盘报错，报错信息如下
#mount -o user_xattr /dev/sdb /var/lib/ceph/osd/ceph-0mount:wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program, orother error In some cases useful info is found in syslog -try dmesg | tailor so执行上面的命令会报错解决的办法是：用下面的两条命令替代上面的一条命令。# mount /dev/sdb /var/lib/ceph/osd/ceph-0# mount -o remount,user_xattr /var/lib/ceph/osd/ceph-0
9.2：执行ceph 命令报错Traceback (mostrecent call last):File "/usr/bin/ceph", line 63, in<module> import radosImportError: Nomodule named rados解决：注释自己安装python的环境变量# more /etc/profile#exportPYTHON=/opt/python/#exportPATH=$PYTHON/bin:$PATH
#其它类似报错Traceback (mostrecent call last):File"/usr/local/bin/ceph", line 53, in <module>import argparseImportError: Nomodule named argparse解决办法：yuminstall python-pippipinstall argparsepipinstall argparse --upgrade
/usr/local/bin/cephosd createTraceback (mostrecent call last):File"/usr/local/bin/ceph", line 56, in <module>import radosImportError: Nomodule named rados cp -vf /usr/local/ceph/lib/python2.6/site-packages/*/usr/lib64/python2.6echo/usr/local/ceph/lib >/etc/ld.so.conf.d/ceph.conf
ldconfig
9.3：创建osd报错#ceph-osd -i 3 --mkfs --mkkey2015-01-2712:29:06.775251 7f4c5d342800 -1 OSD::mkfs: ObjectStore::mkfs failed with error-162015-01-2712:29:06.775288 7f4c5d342800 -1**ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-3: (16)Device or resource busy
排查解决：#mount查看挂载分区/dev/sda2 on /type ext4 (rw)proc on /proc typeproc (rw)sysfs on /sys typesysfs (rw)devpts on /dev/ptstype devpts (rw,gid=5,mode=620)tmpfs on /dev/shmtype tmpfs (rw)/dev/sda1 on /boottype ext4 (rw)none on/proc/sys/fs/binfmt_misc type binfmt_misc (rw)/dev/sdb on/var/lib/ceph/osd/ceph-0 type xfs (rw,user_xattr)此处的磁盘挂载出错啦，应该是一个新磁盘 /dev/sdc ,/dev/sdbon /var/lib/ceph/osd/ceph-3 type xfs (rw,user_xattr)
9.4：报错在ceph auth库中创建mds.node01用户，并赋予权限和创建密钥，密钥保存在/var/lib/ceph/mds/ceph-node01/keyring文件里root@node01 ~]# ^C#ceph --cluster ceph --name client.bootstrap-mds --keyring/var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node01 osd'allow rwx' mds 'allow' mon 'allow profile mds' -o/var/lib/ceph/mds/ceph-node01/keyring2015-01-27 13:54:44.1895357f79f9ec57000 librados:client.bootstrap-mds authentication error (1) Operation not permittedError connectingto cluster: PermissionError
解决办法：确认key文件信息：more/var/lib/ceph/bootstrap-mds/ceph.keyring key = AQDvRcZUALD4MxAAv4s4cMr6XraBt+Pw2Iowyg==通过ceph auth list 命令查看审核信息，找到client.bootstrap-mds key:AQDvRcZUALD4MxAAv4s4cMr6XraBt+Pw2Iowyg== caps: allow profile bootstrap-mds确保/var/lib/ceph/bootstrap-mds/ceph.keyring 文件内容与ceph auth list 查看信息一致
9.5：报错ceph -w clusterbc5ec6c4-3cc9-48ad-94d9-cee3cd75952a health HEALTH_WARN too few pgs per osd (7 < min 20) monmap e1: 3 mons at{node01=172.16.2.27:6789/0,node02=172.16.2.28:6789/0,node03=172.16.2.29:6789/0},election epoch 8, quorum 0,1,2 node01,node02,node03 osdmap e73: 9 osds: 9 up, 9 in pgmap v176: 64 pgs, 1 pools, 0 bytesdata, 0 objects 9524 MB used, 8365 GB / 8374 GB avail 64 active+clean解决办法：解决办法, 需要修改pg_num , pgp_num .先要获取pool name, 如下, 返回pool name : rbd. ceph osd poolstatspool rbd id 0nothing is going on修改pool对应的pg_num和pgp_num.ceph osd pool set rbdpg_num 128specified pg_num128 <= current 128 #要大于当前128
cephosd pool set rbd pg_num 300验证：
ceph -s clusterbc5ec6c4-3cc9-48ad-94d9-cee3cd75952a health HEALTH_WARN too few pgs per osd (14< min 20); pool rbd pg_num 128 > pgp_num 64 monmap e1: 3 mons at{node01=172.16.2.27:6789/0,node02=172.16.2.28:6789/0,node03=172.16.2.29:6789/0},election epoch 8, quorum 0,1,2 node01,node02,node03 osdmap e75: 9 osds: 9 up, 9 in pgmap v184: 128 pgs, 1 pools, 0 bytesdata, 0 objects 9527 MB used, 8365 GB / 8374 GB avail 128 active+clean
同时还需要调整pgp_num# ceph osd pool set rbd pgp_num 300
ceph -s clusterbc5ec6c4-3cc9-48ad-94d9-cee3cd75952a health HEALTH_WARN toofew pgs per osd (14 < min 20) monmap e1: 3 mons at{node01=172.16.2.27:6789/0,node02=172.16.2.28:6789/0,node03=172.16.2.29:6789/0},election epoch 8, quorum 0,1,2 node01,node02,node03 osdmap e78: 9 osds: 9 up, 9 in pgmap v225: 128 pgs, 1 pools, 0 bytesdata, 0 objects 9529 MB used, 8365 GB / 8374 GBavail 128 active+clean最终验证：#ceph -s clusterbc5ec6c4-3cc9-48ad-94d9-cee3cd75952a health HEALTH_OK monmap e1: 3 mons at{node01=172.16.2.27:6789/0,node02=172.16.2.28:6789/0,node03=172.16.2.29:6789/0},election epoch 8, quorum 0,1,2 node01,node02,node03 osdmap e83: 9 osds: 9 up, 9 in pgmap v265: 300 pgs, 1 pools, 0 bytesdata, 0 objects 9538 MB used, 8365 GB / 8374 GBavail 300 active+clean
9.6: 报错#ceph -w clusterbc5ec6c4-3cc9-48ad-94d9-cee3cd75952a health HEALTH_WARN pool cephfs_data has too fewpgs
页:
[1]