CentOS 7.5安装部署Jewel版本Ceph集群
参考文档https://www.linuxidc.com/Linux/2017-09/146760.htm
https://www.cnblogs.com/luohaixian/p/8087591.html
http://docs.ceph.com/docs/master/start/quick-start-preflight/#rhel-centos
简介
Ceph的核心组件包括Ceph OSD、Ceph Monitor、Ceph MDS和Ceph RWG。
Ceph OSD：OSD的英文全称是Object Storage Device，它的主要功能是存储数据、复制数据、平衡数据、恢复数据等，与其它OSD间进行心跳检查等，并将一些变化情况上报给Ceph Monitor。一般情况下一块硬盘对应一个OSD，由OSD来对硬盘存储进行管理，当然一个分区也可以成为一个OSD。
Ceph Monitor：由该英文名字我们可以知道它是一个监视器，负责监视Ceph集群，维护Ceph集群的健康状态，同时维护着Ceph集群中的各种Map图，比如OSD Map、Monitor Map、PG Map和CRUSH Map，这些Map统称为Cluster Map，Cluster Map是RADOS的关键数据结构，管理集群中的所有成员、关系、属性等信息以及数据的分发，比如当用户需要存储数据到Ceph集群时，OSD需要先通过Monitor获取最新的Map图，然后根据Map图和object id等计算出数据最终存储的位置。
Ceph MDS：全称是Ceph MetaData Server，主要保存的文件系统服务的元数据，但对象存储和块存储设备是不需要使用该服务的。
Ceph RWG：RGW为Rados Gateway的缩写，ceph通过RGW为互联网云服务提供商提供对象存储服务。RGW在librados之上向应用提供访问ceph集群的RestAPI， 支持Amazon S3和openstack swift两种接口。对RGW最直接的理解就是一个协议转换层，把从上层应用符合S3或Swift协议的请求转换成rados的请求， 将数据保存在rados集群中。
架构图
安装部署
一、基础环境
0、服务分布
mon ceph0、ceph2、cphe3 注意mon为奇数节点
osd ceph0、ceph1、ceph2、ceph3
rgw ceph1
deployceph0
1、host解析
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
172.20.1.138 idcv-ceph0
172.20.1.139 idcv-ceph1
172.20.1.140 idcv-ceph2
172.20.1.141 idcv-ceph3
2、ntp时间同步
# ntpdate 172.20.0.63
3、ssh免密码登陆
# ssh-keygen
# ssh-copy-id root@idcv-ceph1
# ssh-copy-id root@idcv-ceph2
# ssh-copy-id root@idcv-ceph3
4、update系统
# yum update
5、关闭selinux
# sed -i 's/enforcing/disabled/g' /etc/selinux/config
6、关闭iptables
# systemctl disable firewalld
7、reboot
# reboot
二、安装部署deploy节点
1、设置国内yum源
# cat /etc/yum.repos.d/ceph.repo
name=Ceph packages for $basearch
baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
priority=1
name=Ceph noarch packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
priority=1
name=Ceph source packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
priority=1
2、安装ceph-deploy
# yum install ceph-deploy
# ceph-deploy --version
1.5.39
# ceph -v
ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
3、创建部署目录并部署集群
# mkdir cluster
# cd cluster
# ceph-deploy new idcv-ceph0 idcv-ceph1idcv-ceph2 idcv-ceph3
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.39): /usr/bin/ceph-deploy new idcv-ceph0 idcv-ceph1 idcv-ceph2 idcv-ceph3
ceph-deploy options:
username : None
func :
verbose : False
overwrite_conf : False
quiet : False
cd_conf :
cluster : ceph
ssh_copykey : True
mon : ['idcv-ceph0', 'idcv-ceph1', 'idcv-ceph2', 'idcv-ceph3']
public_network : None
ceph_conf : None
cluster_network : None
default_release : False
fsid : None
Creating new cluster named ceph
making sure passwordless SSH succeeds
connected to host: idcv-ceph0
detect platform information from remote host
detect machine type
find the location of an executable
Running command: /usr/sbin/ip link show
Running command: /usr/sbin/ip addr show
IP addresses found:
Resolving host idcv-ceph0
Monitor idcv-ceph0 at 172.20.1.138
making sure passwordless SSH succeeds
connected to host: idcv-ceph0
Running command: ssh -CT -o BatchMode=yes idcv-ceph1
connection detected need for sudo
connected to host: idcv-ceph1
detect platform information from remote host
detect machine type
find the location of an executable
Running command: sudo /usr/sbin/ip link show
Running command: sudo /usr/sbin/ip addr show
IP addresses found:
Resolving host idcv-ceph1
Monitor idcv-ceph1 at 172.20.1.139
making sure passwordless SSH succeeds
connected to host: idcv-ceph0
Running command: ssh -CT -o BatchMode=yes idcv-ceph2
connection detected need for sudo
connected to host: idcv-ceph2
detect platform information from remote host
detect machine type
find the location of an executable
Running command: sudo /usr/sbin/ip link show
Running command: sudo /usr/sbin/ip addr show
IP addresses found:
Resolving host idcv-ceph2
Monitor idcv-ceph2 at 172.20.1.140
making sure passwordless SSH succeeds
connected to host: idcv-ceph0
Running command: ssh -CT -o BatchMode=yes idcv-ceph3
connection detected need for sudo
connected to host: idcv-ceph3
detect platform information from remote host
detect machine type
find the location of an executable
Running command: sudo /usr/sbin/ip link show
Running command: sudo /usr/sbin/ip addr show
IP addresses found:
Resolving host idcv-ceph3
Monitor idcv-ceph3 at 172.20.1.141
Monitor initial members are ['idcv-ceph0', 'idcv-ceph1', 'idcv-ceph2', 'idcv-ceph3']
Monitor addrs are ['172.20.1.138', '172.20.1.139', '172.20.1.140', '172.20.1.141']
Creating a random mon key...
Writing monitor keyring to ceph.mon.keyring...
Writing initial config to ceph.conf...
三、安装mon服务
1、修改cpeh.conf文件
注意mon为奇数，如果为偶数，有一个不会安装，另外设置好public_network，并稍微增大mon之间时差允许范围(默认为0.05s，现改为2s)
# cat ceph.conf
fsid = 812d3acb-eaa8-4355-9a74-64f2cd5209b3
mon_initial_members = idcv-ceph0, idcv-ceph1, idcv-ceph2, idcv-ceph3
mon_host = 172.20.1.138,172.20.1.139,172.20.1.140,172.20.1.141
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 172.20.0.0/20
mon_clock_drift_allowed = 2
2、开始部署mon服务
#ceph-deploy mon create-initial
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.39): /usr/bin/ceph-deploy mon create-initial
ceph-deploy options:
username : None
verbose : False
overwrite_conf : False
subcommand : create-initial
quiet : False
cd_conf :
cluster : ceph
func :
ceph_conf : None
default_release : False
keyrings : None
Deploying mon, cluster ceph hosts idcv-ceph0 idcv-ceph1 idcv-ceph2 idcv-ceph3
detecting platform for host idcv-ceph0 ...
connected to host: idcv-ceph0
detect platform information from remote host
detect machine type
find the location of an executable
distro info: CentOS Linux 7.5.1804 Core
determining if provided host has same hostname in remote
get remote short hostname
deploying mon to idcv-ceph0
get remote short hostname
remote hostname: idcv-ceph0
write cluster configuration to /etc/ceph/{cluster}.conf
create the mon path if it does not exist
checking for done path: /var/lib/ceph/mon/ceph-idcv-ceph0/done
done path does not exist: /var/lib/ceph/mon/ceph-idcv-ceph0/done
creating keyring file: /var/lib/ceph/tmp/ceph-idcv-ceph0.mon.keyring
create the monitor keyring file
Running command: ceph-mon --cluster ceph --mkfs -i idcv-ceph0 --keyring /var/lib/ceph/tmp/ceph-idcv-ceph0.mon.keyring --setuser 167 --setgroup 167
ceph-mon: renaming mon.noname-a 172.20.1.138:6789/0 to mon.idcv-ceph0
ceph-mon: set fsid to 812d3acb-eaa8-4355-9a74-64f2cd5209b3
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-idcv-ceph0 for mon.idcv-ceph0
unlinking keyring file /var/lib/ceph/tmp/ceph-idcv-ceph0.mon.keyring
create a done file to avoid re-doing the mon deployment
create the init path if it does not exist
Running command: systemctl enable ceph.target
Running command: systemctl enable ceph-mon@idcv-ceph0
Created symlink from /etc/systemd/system/ceph-mon.target.wants/ceph-mon@idcv-ceph0.service to /usr/lib/systemd/system/ceph-mon@.service.
Running command: systemctl start ceph-mon@idcv-ceph0
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph0.asok mon_status
****
status for monitor: mon.idcv-ceph0
{
"election_epoch": 0,
"extra_probe_peers": [
"172.20.1.139:6789/0",
"172.20.1.140:6789/0",
"172.20.1.141:6789/0"
],
"monmap": {
"created": "2018-07-03 11:06:12.249491",
"epoch": 0,
"fsid": "812d3acb-eaa8-4355-9a74-64f2cd5209b3",
"modified": "2018-07-03 11:06:12.249491",
"mons": [
{
"addr": "172.20.1.138:6789/0",
"name": "idcv-ceph0",
"rank": 0
},
{
"addr": "0.0.0.0:0/1",
"name": "idcv-ceph1",
"rank": 1
},
{
"addr": "0.0.0.0:0/2",
"name": "idcv-ceph2",
"rank": 2
},
{
"addr": "0.0.0.0:0/3",
"name": "idcv-ceph3",
"rank": 3
}
]
},
"name": "idcv-ceph0",
"outside_quorum": [
"idcv-ceph0"
],
"quorum": [],
"rank": 0,
"state": "probing",
"sync_provider": []
}
****
monitor: mon.idcv-ceph0 is running
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph0.asok mon_status
detecting platform for host idcv-ceph1 ...
connection detected need for sudo
connected to host: idcv-ceph1
detect platform information from remote host
detect machine type
find the location of an executable
distro info: CentOS Linux 7.5.1804 Core
determining if provided host has same hostname in remote
get remote short hostname
deploying mon to idcv-ceph1
get remote short hostname
remote hostname: idcv-ceph1
write cluster configuration to /etc/ceph/{cluster}.conf
RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
detecting platform for host idcv-ceph2 ...
connection detected need for sudo
connected to host: idcv-ceph2
detect platform information from remote host
detect machine type
find the location of an executable
distro info: CentOS Linux 7.5.1804 Core
determining if provided host has same hostname in remote
get remote short hostname
deploying mon to idcv-ceph2
get remote short hostname
remote hostname: idcv-ceph2
write cluster configuration to /etc/ceph/{cluster}.conf
create the mon path if it does not exist
checking for done path: /var/lib/ceph/mon/ceph-idcv-ceph2/done
done path does not exist: /var/lib/ceph/mon/ceph-idcv-ceph2/done
creating keyring file: /var/lib/ceph/tmp/ceph-idcv-ceph2.mon.keyring
create the monitor keyring file
Running command: sudo ceph-mon --cluster ceph --mkfs -i idcv-ceph2 --keyring /var/lib/ceph/tmp/ceph-idcv-ceph2.mon.keyring --setuser 167 --setgroup 167
ceph-mon: renaming mon.noname-c 172.20.1.140:6789/0 to mon.idcv-ceph2
ceph-mon: set fsid to 812d3acb-eaa8-4355-9a74-64f2cd5209b3
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-idcv-ceph2 for mon.idcv-ceph2
unlinking keyring file /var/lib/ceph/tmp/ceph-idcv-ceph2.mon.keyring
create a done file to avoid re-doing the mon deployment
create the init path if it does not exist
Running command: sudo systemctl enable ceph.target
Running command: sudo systemctl enable ceph-mon@idcv-ceph2
Created symlink from /etc/systemd/system/ceph-mon.target.wants/ceph-mon@idcv-ceph2.service to /usr/lib/systemd/system/ceph-mon@.service.
Running command: sudo systemctl start ceph-mon@idcv-ceph2
Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph2.asok mon_status
****
status for monitor: mon.idcv-ceph2
{
"election_epoch": 0,
"extra_probe_peers": [
"172.20.1.138:6789/0",
"172.20.1.139:6789/0",
"172.20.1.141:6789/0"
],
"monmap": {
"created": "2018-07-03 11:06:15.703352",
"epoch": 0,
"fsid": "812d3acb-eaa8-4355-9a74-64f2cd5209b3",
"modified": "2018-07-03 11:06:15.703352",
"mons": [
{
"addr": "172.20.1.138:6789/0",
"name": "idcv-ceph0",
"rank": 0
},
{
"addr": "172.20.1.140:6789/0",
"name": "idcv-ceph2",
"rank": 1
},
{
"addr": "0.0.0.0:0/2",
"name": "idcv-ceph1",
"rank": 2
},
{
"addr": "0.0.0.0:0/3",
"name": "idcv-ceph3",
"rank": 3
}
]
},
"name": "idcv-ceph2",
"outside_quorum": [
"idcv-ceph0",
"idcv-ceph2"
],
"quorum": [],
"rank": 1,
"state": "probing",
"sync_provider": []
}
****
monitor: mon.idcv-ceph2 is running
Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph2.asok mon_status
detecting platform for host idcv-ceph3 ...
connection detected need for sudo
connected to host: idcv-ceph3
detect platform information from remote host
detect machine type
find the location of an executable
distro info: CentOS Linux 7.5.1804 Core
determining if provided host has same hostname in remote
get remote short hostname
deploying mon to idcv-ceph3
get remote short hostname
remote hostname: idcv-ceph3
write cluster configuration to /etc/ceph/{cluster}.conf
create the mon path if it does not exist
checking for done path: /var/lib/ceph/mon/ceph-idcv-ceph3/done
done path does not exist: /var/lib/ceph/mon/ceph-idcv-ceph3/done
creating keyring file: /var/lib/ceph/tmp/ceph-idcv-ceph3.mon.keyring
create the monitor keyring file
Running command: sudo ceph-mon --cluster ceph --mkfs -i idcv-ceph3 --keyring /var/lib/ceph/tmp/ceph-idcv-ceph3.mon.keyring --setuser 167 --setgroup 167
ceph-mon: renaming mon.noname-d 172.20.1.141:6789/0 to mon.idcv-ceph3
ceph-mon: set fsid to 812d3acb-eaa8-4355-9a74-64f2cd5209b3
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-idcv-ceph3 for mon.idcv-ceph3
unlinking keyring file /var/lib/ceph/tmp/ceph-idcv-ceph3.mon.keyring
create a done file to avoid re-doing the mon deployment
create the init path if it does not exist
Running command: sudo systemctl enable ceph.target
Running command: sudo systemctl enable ceph-mon@idcv-ceph3
Created symlink from /etc/systemd/system/ceph-mon.target.wants/ceph-mon@idcv-ceph3.service to /usr/lib/systemd/system/ceph-mon@.service.
Running command: sudo systemctl start ceph-mon@idcv-ceph3
Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph3.asok mon_status
****
status for monitor: mon.idcv-ceph3
{
"election_epoch": 1,
"extra_probe_peers": [
"172.20.1.138:6789/0",
"172.20.1.139:6789/0",
"172.20.1.140:6789/0"
],
"monmap": {
"created": "2018-07-03 11:06:18.695039",
"epoch": 0,
"fsid": "812d3acb-eaa8-4355-9a74-64f2cd5209b3",
"modified": "2018-07-03 11:06:18.695039",
"mons": [
{
"addr": "172.20.1.138:6789/0",
"name": "idcv-ceph0",
"rank": 0
},
{
"addr": "172.20.1.140:6789/0",
"name": "idcv-ceph2",
"rank": 1
},
{
"addr": "172.20.1.141:6789/0",
"name": "idcv-ceph3",
"rank": 2
},
{
"addr": "0.0.0.0:0/2",
"name": "idcv-ceph1",
"rank": 3
}
]
},
"name": "idcv-ceph3",
"outside_quorum": [],
"quorum": [],
"rank": 2,
"state": "electing",
"sync_provider": []
}
****
monitor: mon.idcv-ceph3 is running
Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph3.asok mon_status
GenericError: Failed to create 1 monitors
3、注意mon节点只能是奇数，根据上面报错有一个节点没有安装成功mon服务，需要把idcv-ceph1删掉
# cat ceph.conf
fsid = 812d3acb-eaa8-4355-9a74-64f2cd5209b3
mon_initial_members = idcv-ceph0, idcv-ceph1, idcv-ceph2, idcv-ceph3
mon_host = 172.20.1.138,172.20.1.139,172.20.1.140,172.20.1.141
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 172.20.0.0/20
mon_clock_drift_allowed = 2
# ceph mon remove idcv-ceph1
removing mon.idcv-ceph1 at 0.0.0.0:0/1, there will be 3 monitors
# ceph -s
cluster 812d3acb-eaa8-4355-9a74-64f2cd5209b3
health HEALTH_ERR
64 pgs are stuck inactive for more than 300 seconds
64 pgs stuck inactive
64 pgs stuck unclean
no osds
monmap e2: 3 mons at {idcv-ceph0=172.20.1.138:6789/0,idcv-ceph2=172.20.1.140:6789/0,idcv-ceph3=172.20.1.141:6789/0}
election epoch 8, quorum 0,1,2 idcv-ceph0,idcv-ceph2,idcv-ceph3
osdmap e1: 0 osds: 0 up, 0 in
flags sortbitwise,require_jewel_osds
pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
64 creating
4、也可以修改ceph.conf文件，再覆盖部署一次
# cat ceph.conf
fsid = 812d3acb-eaa8-4355-9a74-64f2cd5209b3
mon_initial_members = idcv-ceph0, idcv-ceph2, idcv-ceph3
mon_host = 172.20.1.138,172.20.1.140,172.20.1.141
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 172.20.0.0/20
mon_clock_drift_allowed = 2
# ceph-deploy --overwrite-conf mon create-initial
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.39): /usr/bin/ceph-deploy --overwrite-conf mon create-initial
ceph-deploy options:
username : None
verbose : False
overwrite_conf : True
subcommand : create-initial
quiet : False
cd_conf :
cluster : ceph
func :
ceph_conf : None
default_release : False
keyrings : None
Deploying mon, cluster ceph hosts idcv-ceph0 idcv-ceph2 idcv-ceph3
detecting platform for host idcv-ceph0 ...
connected to host: idcv-ceph0
detect platform information from remote host
detect machine type
find the location of an executable
distro info: CentOS Linux 7.5.1804 Core
determining if provided host has same hostname in remote
get remote short hostname
deploying mon to idcv-ceph0
get remote short hostname
remote hostname: idcv-ceph0
write cluster configuration to /etc/ceph/{cluster}.conf
create the mon path if it does not exist
checking for done path: /var/lib/ceph/mon/ceph-idcv-ceph0/done
create a done file to avoid re-doing the mon deployment
create the init path if it does not exist
Running command: systemctl enable ceph.target
Running command: systemctl enable ceph-mon@idcv-ceph0
Running command: systemctl start ceph-mon@idcv-ceph0
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph0.asok mon_status
****
status for monitor: mon.idcv-ceph0
{
"election_epoch": 8,
"extra_probe_peers": [
"172.20.1.139:6789/0",
"172.20.1.140:6789/0",
"172.20.1.141:6789/0"
],
"monmap": {
"created": "2018-07-03 11:06:12.249491",
"epoch": 2,
"fsid": "812d3acb-eaa8-4355-9a74-64f2cd5209b3",
"modified": "2018-07-03 11:21:27.254076",
"mons": [
{
"addr": "172.20.1.138:6789/0",
"name": "idcv-ceph0",
"rank": 0
},
{
"addr": "172.20.1.140:6789/0",
"name": "idcv-ceph2",
"rank": 1
},
{
"addr": "172.20.1.141:6789/0",
"name": "idcv-ceph3",
"rank": 2
}
]
},
"name": "idcv-ceph0",
"outside_quorum": [],
"quorum": [
0,
1,
2
],
"rank": 0,
"state": "leader",
"sync_provider": []
}
****
monitor: mon.idcv-ceph0 is running
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph0.asok mon_status
detecting platform for host idcv-ceph2 ...
connection detected need for sudo
connected to host: idcv-ceph2
detect platform information from remote host
detect machine type
find the location of an executable
distro info: CentOS Linux 7.5.1804 Core
determining if provided host has same hostname in remote
get remote short hostname
deploying mon to idcv-ceph2
get remote short hostname
remote hostname: idcv-ceph2
write cluster configuration to /etc/ceph/{cluster}.conf
create the mon path if it does not exist
checking for done path: /var/lib/ceph/mon/ceph-idcv-ceph2/done
create a done file to avoid re-doing the mon deployment
create the init path if it does not exist
Running command: sudo systemctl enable ceph.target
Running command: sudo systemctl enable ceph-mon@idcv-ceph2
Running command: sudo systemctl start ceph-mon@idcv-ceph2
Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph2.asok mon_status
****
status for monitor: mon.idcv-ceph2
{
"election_epoch": 8,
"extra_probe_peers": [
"172.20.1.138:6789/0",
"172.20.1.139:6789/0",
"172.20.1.141:6789/0"
],
"monmap": {
"created": "2018-07-03 11:06:12.249491",
"epoch": 2,
"fsid": "812d3acb-eaa8-4355-9a74-64f2cd5209b3",
"modified": "2018-07-03 11:21:27.254076",
"mons": [
{
"addr": "172.20.1.138:6789/0",
"name": "idcv-ceph0",
"rank": 0
},
{
"addr": "172.20.1.140:6789/0",
"name": "idcv-ceph2",
"rank": 1
},
{
"addr": "172.20.1.141:6789/0",
"name": "idcv-ceph3",
"rank": 2
}
]
},
"name": "idcv-ceph2",
"outside_quorum": [],
"quorum": [
0,
1,
2
],
"rank": 1,
"state": "peon",
"sync_provider": []
}
****
monitor: mon.idcv-ceph2 is running
Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph2.asok mon_status
detecting platform for host idcv-ceph3 ...
connection detected need for sudo
connected to host: idcv-ceph3
detect platform information from remote host
detect machine type
find the location of an executable
distro info: CentOS Linux 7.5.1804 Core
determining if provided host has same hostname in remote
get remote short hostname
deploying mon to idcv-ceph3
get remote short hostname
remote hostname: idcv-ceph3
write cluster configuration to /etc/ceph/{cluster}.conf
create the mon path if it does not exist
checking for done path: /var/lib/ceph/mon/ceph-idcv-ceph3/done
create a done file to avoid re-doing the mon deployment
create the init path if it does not exist
Running command: sudo systemctl enable ceph.target
Running command: sudo systemctl enable ceph-mon@idcv-ceph3
Running command: sudo systemctl start ceph-mon@idcv-ceph3
Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph3.asok mon_status
****
status for monitor: mon.idcv-ceph3
{
"election_epoch": 8,
"extra_probe_peers": [
"172.20.1.138:6789/0",
"172.20.1.139:6789/0",
"172.20.1.140:6789/0"
],
"monmap": {
"created": "2018-07-03 11:06:12.249491",
"epoch": 2,
"fsid": "812d3acb-eaa8-4355-9a74-64f2cd5209b3",
"modified": "2018-07-03 11:21:27.254076",
"mons": [
{
"addr": "172.20.1.138:6789/0",
"name": "idcv-ceph0",
"rank": 0
},
{
"addr": "172.20.1.140:6789/0",
"name": "idcv-ceph2",
"rank": 1
},
{
"addr": "172.20.1.141:6789/0",
"name": "idcv-ceph3",
"rank": 2
}
]
},
"name": "idcv-ceph3",
"outside_quorum": [],
"quorum": [
0,
1,
2
],
"rank": 2,
"state": "peon",
"sync_provider": []
}
****
monitor: mon.idcv-ceph3 is running
Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph3.asok mon_status
processing monitor mon.idcv-ceph0
connected to host: idcv-ceph0
detect platform information from remote host
detect machine type
find the location of an executable
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph0.asok mon_status
mon.idcv-ceph0 monitor has reached quorum!
processing monitor mon.idcv-ceph2
connection detected need for sudo
connected to host: idcv-ceph2
detect platform information from remote host
detect machine type
find the location of an executable
Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph2.asok mon_status
mon.idcv-ceph2 monitor has reached quorum!
processing monitor mon.idcv-ceph3
connection detected need for sudo
connected to host: idcv-ceph3
detect platform information from remote host
detect machine type
find the location of an executable
Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.idcv-ceph3.asok mon_status
mon.idcv-ceph3 monitor has reached quorum!
all initial monitors are running and have formed quorum
Running gatherkeys...
Storing keys in temp directory /tmp/tmpBqY1be
connected to host: idcv-ceph0
detect platform information from remote host
detect machine type
get remote short hostname
fetch remote file
Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.idcv-ceph0.asok mon_status
Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-idcv-ceph0/keyring auth get client.admin
Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-idcv-ceph0/keyring auth get client.bootstrap-mds
Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-idcv-ceph0/keyring auth get client.bootstrap-mgr
Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-idcv-ceph0/keyring auth get-or-create client.bootstrap-mgr mon allow profile bootstrap-mgr
Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-idcv-ceph0/keyring auth get client.bootstrap-osd
Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-idcv-ceph0/keyring auth get client.bootstrap-rgw
Storing ceph.client.admin.keyring
Storing ceph.bootstrap-mds.keyring
Storing ceph.bootstrap-mgr.keyring
keyring 'ceph.mon.keyring' already exists
Storing ceph.bootstrap-osd.keyring
Storing ceph.bootstrap-rgw.keyring
Destroy temp directory /tmp/tmpBqY1be
# ls
ceph.bootstrap-mds.keyringceph.bootstrap-osd.keyringceph.client.admin.keyringceph-deploy-ceph.lo
五、部署OSD角色
先准备后激活
ceph-deploy --overwrite-conf osd prepare idcv-ceph0:/dev/sdbidcv-ceph1:/dev/sdbidcv-ceph2:/dev/sdb idcv-ceph3:/dev/sdb --zap-disk
ceph-deploy --overwrite-conf osd activate idcv-ceph0:/dev/sdb1idcv-ceph1:/dev/sdb1idcv-ceph2:/dev/sdb1idcv-ceph3:/dev/sdb1
# ceph-deploy --overwrite-conf osd prepare idcv-ceph0:/dev/sdbidcv-ceph1:/dev/sdbidcv-ceph2:/dev/sdb idcv-ceph3:/dev/sdb --zap-disk
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.39): /usr/bin/ceph-deploy --overwrite-conf osd prepare idcv-ceph0:/dev/sdb idcv-ceph1:/dev/sdb idcv-ceph2:/dev/sdb idcv-ceph3:/dev/sdb --zap-disk
ceph-deploy options:
username : None
block_db : None
disk : [('idcv-ceph0', '/dev/sdb', None), ('idcv-ceph1', '/dev/sdb', None), ('idcv-ceph2', '/dev/sdb', None), ('idcv-ceph3', '/dev/sdb', None)]
dmcrypt : False
verbose : False
bluestore : None
block_wal : None
overwrite_conf : True
subcommand : prepare
dmcrypt_key_dir : /etc/ceph/dmcrypt-keys
quiet : False
cd_conf :
cluster : ceph
fs_type : xfs
filestore : None
func :
ceph_conf : None
default_release : False
zap_disk : True
Preparing cluster ceph disks idcv-ceph0:/dev/sdb: idcv-ceph1:/dev/sdb: idcv-ceph2:/dev/sdb: idcv-ceph3:/dev/sdb:
connected to host: idcv-ceph0
detect platform information from remote host
detect machine type
find the location of an executable
Distro info: CentOS Linux 7.5.1804 Core
Deploying osd to idcv-ceph0
write cluster configuration to /etc/ceph/{cluster}.conf
Preparing host idcv-ceph0 disk /dev/sdb journal None activate False
find the location of an executable
Running command: /usr/sbin/ceph-disk -v prepare --zap-disk --cluster ceph --fs-type xfs -- /dev/sdb
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
command: Running command: /usr/bin/ceph-osd --check-allows-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
command: Running command: /usr/bin/ceph-osd --check-wants-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
command: Running command: /usr/bin/ceph-osd --check-needs-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
set_type: Will colocate journal with data on /dev/sdb
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
zap: Zapping partition table on /dev/sdb
command_check_call: Running command: /usr/sbin/sgdisk --zap-all -- /dev/sdb
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.
****
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
command_check_call: Running command: /usr/sbin/sgdisk --clear --mbrtogpt -- /dev/sdb
Creating new GPT entries.
The operation has completed successfully.
update_partition: Calling partprobe on zapped device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /usr/sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
ptype_tobe_for_name: name = journal
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
create_partition: Creating journal partition num 2 size 5120 on /dev/sdb
command_check_call: Running command: /usr/sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:ca6594bd-a4b2-4be7-9aa5-69ba91ce7441 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdb
The operation has completed successfully.
update_partition: Calling partprobe on created device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /usr/sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb2 uuid path is /sys/dev/block/8:18/dm/uuid
prepare_device: Journal is GPT partition /dev/disk/by-partuuid/ca6594bd-a4b2-4be7-9aa5-69ba91ce7441
prepare_device: Journal is GPT partition /dev/disk/by-partuuid/ca6594bd-a4b2-4be7-9aa5-69ba91ce7441
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
set_data_partition: Creating osd partition on /dev/sdb
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
ptype_tobe_for_name: name = data
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
create_partition: Creating data partition num 1 size 0 on /dev/sdb
command_check_call: Running command: /usr/sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:3b210c8e-b2ac-4266-9e59-623c031ebb89 --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be --mbrtogpt -- /dev/sdb
The operation has completed successfully.
update_partition: Calling partprobe on created device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /usr/sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is /sys/dev/block/8:17/dm/uuid
populate_data_path_device: Creating xfs fs on /dev/sdb1
command_check_call: Running command: /usr/sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdb1
meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=6225855 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=24903419, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=12159, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
mount: Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.kvs_nq with options noatime,inode64
command_check_call: Running command: /usr/bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.kvs_nq
command: Running command: /usr/sbin/restorecon /var/lib/ceph/tmp/mnt.kvs_nq
populate_data_path: Preparing osd data dir /var/lib/ceph/tmp/mnt.kvs_nq
command: Running command: /usr/sbin/restorecon -R /var/lib/ceph/tmp/mnt.kvs_nq/ceph_fsid.2933.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.kvs_nq/ceph_fsid.2933.tmp
command: Running command: /usr/sbin/restorecon -R /var/lib/ceph/tmp/mnt.kvs_nq/fsid.2933.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.kvs_nq/fsid.2933.tmp
command: Running command: /usr/sbin/restorecon -R /var/lib/ceph/tmp/mnt.kvs_nq/magic.2933.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.kvs_nq/magic.2933.tmp
command: Running command: /usr/sbin/restorecon -R /var/lib/ceph/tmp/mnt.kvs_nq/journal_uuid.2933.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.kvs_nq/journal_uuid.2933.tmp
adjust_symlink: Creating symlink /var/lib/ceph/tmp/mnt.kvs_nq/journal -> /dev/disk/by-partuuid/ca6594bd-a4b2-4be7-9aa5-69ba91ce7441
command: Running command: /usr/sbin/restorecon -R /var/lib/ceph/tmp/mnt.kvs_nq
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.kvs_nq
unmount: Unmounting /var/lib/ceph/tmp/mnt.kvs_nq
command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.kvs_nq
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
command_check_call: Running command: /usr/sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdb
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
update_partition: Calling partprobe on prepared device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /usr/sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command_check_call: Running command: /usr/bin/udevadm trigger --action=add --sysname-match sdb1
checking OSD status...
find the location of an executable
Running command: /bin/ceph --cluster=ceph osd stat --format=json
Host idcv-ceph0 is now ready for osd use.
connection detected need for sudo
connected to host: idcv-ceph1
detect platform information from remote host
detect machine type
find the location of an executable
Distro info: CentOS Linux 7.5.1804 Core
Deploying osd to idcv-ceph1
write cluster configuration to /etc/ceph/{cluster}.conf
Preparing host idcv-ceph1 disk /dev/sdb journal None activate False
find the location of an executable
Running command: sudo /usr/sbin/ceph-disk -v prepare --zap-disk --cluster ceph --fs-type xfs -- /dev/sdb
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
command: Running command: /usr/bin/ceph-osd --check-allows-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
command: Running command: /usr/bin/ceph-osd --check-wants-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
command: Running command: /usr/bin/ceph-osd --check-needs-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
set_type: Will colocate journal with data on /dev/sdb
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
zap: Zapping partition table on /dev/sdb
command_check_call: Running command: /sbin/sgdisk --zap-all -- /dev/sdb
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
command_check_call: Running command: /sbin/sgdisk --clear --mbrtogpt -- /dev/sdb
Creating new GPT entries.
The operation has completed successfully.
update_partition: Calling partprobe on zapped device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
ptype_tobe_for_name: name = journal
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
create_partition: Creating journal partition num 2 size 5120 on /dev/sdb
command_check_call: Running command: /sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:09dad07a-985e-4733-a228-f7b1105b7385 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdb
The operation has completed successfully.
update_partition: Calling partprobe on created device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb2 uuid path is /sys/dev/block/8:18/dm/uuid
prepare_device: Journal is GPT partition /dev/disk/by-partuuid/09dad07a-985e-4733-a228-f7b1105b7385
prepare_device: Journal is GPT partition /dev/disk/by-partuuid/09dad07a-985e-4733-a228-f7b1105b7385
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
set_data_partition: Creating osd partition on /dev/sdb
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
ptype_tobe_for_name: name = data
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
create_partition: Creating data partition num 1 size 0 on /dev/sdb
command_check_call: Running command: /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:2809f370-e6ad-4d29-bf6b-57fe1f2004c6 --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be --mbrtogpt -- /dev/sdb
The operation has completed successfully.
update_partition: Calling partprobe on created device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is /sys/dev/block/8:17/dm/uuid
populate_data_path_device: Creating xfs fs on /dev/sdb1
command_check_call: Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdb1
meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=6225855 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=24903419, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=12159, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
mount: Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.HAg1vC with options noatime,inode64
command_check_call: Running command: /usr/bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.HAg1vC
command: Running command: /sbin/restorecon /var/lib/ceph/tmp/mnt.HAg1vC
populate_data_path: Preparing osd data dir /var/lib/ceph/tmp/mnt.HAg1vC
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.HAg1vC/ceph_fsid.2415.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.HAg1vC/ceph_fsid.2415.tmp
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.HAg1vC/fsid.2415.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.HAg1vC/fsid.2415.tmp
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.HAg1vC/magic.2415.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.HAg1vC/magic.2415.tmp
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.HAg1vC/journal_uuid.2415.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.HAg1vC/journal_uuid.2415.tmp
adjust_symlink: Creating symlink /var/lib/ceph/tmp/mnt.HAg1vC/journal -> /dev/disk/by-partuuid/09dad07a-985e-4733-a228-f7b1105b7385
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.HAg1vC
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.HAg1vC
unmount: Unmounting /var/lib/ceph/tmp/mnt.HAg1vC
command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.HAg1vC
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
command_check_call: Running command: /sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdb
The operation has completed successfully.
update_partition: Calling partprobe on prepared device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command_check_call: Running command: /usr/bin/udevadm trigger --action=add --sysname-match sdb1
checking OSD status...
find the location of an executable
Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
Host idcv-ceph1 is now ready for osd use.
connection detected need for sudo
connected to host: idcv-ceph2
detect platform information from remote host
detect machine type
find the location of an executable
Distro info: CentOS Linux 7.5.1804 Core
Deploying osd to idcv-ceph2
write cluster configuration to /etc/ceph/{cluster}.conf
Preparing host idcv-ceph2 disk /dev/sdb journal None activate False
find the location of an executable
Running command: sudo /usr/sbin/ceph-disk -v prepare --zap-disk --cluster ceph --fs-type xfs -- /dev/sdb
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
command: Running command: /usr/bin/ceph-osd --check-allows-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
command: Running command: /usr/bin/ceph-osd --check-wants-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
command: Running command: /usr/bin/ceph-osd --check-needs-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
set_type: Will colocate journal with data on /dev/sdb
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
zap: Zapping partition table on /dev/sdb
command_check_call: Running command: /sbin/sgdisk --zap-all -- /dev/sdb
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
command_check_call: Running command: /sbin/sgdisk --clear --mbrtogpt -- /dev/sdb
Creating new GPT entries.
The operation has completed successfully.
update_partition: Calling partprobe on zapped device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
ptype_tobe_for_name: name = journal
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
create_partition: Creating journal partition num 2 size 5120 on /dev/sdb
command_check_call: Running command: /sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:857f0966-30d5-4ad1-9e0c-abff0fbbbc4e --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdb
The operation has completed successfully.
update_partition: Calling partprobe on created device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb2 uuid path is /sys/dev/block/8:18/dm/uuid
prepare_device: Journal is GPT partition /dev/disk/by-partuuid/857f0966-30d5-4ad1-9e0c-abff0fbbbc4e
prepare_device: Journal is GPT partition /dev/disk/by-partuuid/857f0966-30d5-4ad1-9e0c-abff0fbbbc4e
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
set_data_partition: Creating osd partition on /dev/sdb
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
ptype_tobe_for_name: name = data
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
create_partition: Creating data partition num 1 size 0 on /dev/sdb
command_check_call: Running command: /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:dac63cc2-6876-4004-ba3b-7786be39d392 --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be --mbrtogpt -- /dev/sdb
The operation has completed successfully.
update_partition: Calling partprobe on created device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is /sys/dev/block/8:17/dm/uuid
populate_data_path_device: Creating xfs fs on /dev/sdb1
command_check_call: Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdb1
meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=6225855 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=24903419, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=12159, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
mount: Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.jhzVmR with options noatime,inode64
realtime =none extsz=4096 blocks=0, rtextents=0
command_check_call: Running command: /usr/bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.jhzVmR
command: Running command: /sbin/restorecon /var/lib/ceph/tmp/mnt.jhzVmR
populate_data_path: Preparing osd data dir /var/lib/ceph/tmp/mnt.jhzVmR
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.jhzVmR/ceph_fsid.2354.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.jhzVmR/ceph_fsid.2354.tmp
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.jhzVmR/fsid.2354.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.jhzVmR/fsid.2354.tmp
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.jhzVmR/magic.2354.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.jhzVmR/magic.2354.tmp
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.jhzVmR/journal_uuid.2354.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.jhzVmR/journal_uuid.2354.tmp
adjust_symlink: Creating symlink /var/lib/ceph/tmp/mnt.jhzVmR/journal -> /dev/disk/by-partuuid/857f0966-30d5-4ad1-9e0c-abff0fbbbc4e
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.jhzVmR
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.jhzVmR
unmount: Unmounting /var/lib/ceph/tmp/mnt.jhzVmR
command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.jhzVmR
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
command_check_call: Running command: /sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdb
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
update_partition: Calling partprobe on prepared device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command_check_call: Running command: /usr/bin/udevadm trigger --action=add --sysname-match sdb1
checking OSD status...
find the location of an executable
Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
Host idcv-ceph2 is now ready for osd use.
connection detected need for sudo
connected to host: idcv-ceph3
detect platform information from remote host
detect machine type
find the location of an executable
Distro info: CentOS Linux 7.5.1804 Core
Deploying osd to idcv-ceph3
write cluster configuration to /etc/ceph/{cluster}.conf
Preparing host idcv-ceph3 disk /dev/sdb journal None activate False
find the location of an executable
Running command: sudo /usr/sbin/ceph-disk -v prepare --zap-disk --cluster ceph --fs-type xfs -- /dev/sdb
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
command: Running command: /usr/bin/ceph-osd --check-allows-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
command: Running command: /usr/bin/ceph-osd --check-wants-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
command: Running command: /usr/bin/ceph-osd --check-needs-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster ceph --setuser ceph --setgroup ceph
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
set_type: Will colocate journal with data on /dev/sdb
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
zap: Zapping partition table on /dev/sdb
command_check_call: Running command: /sbin/sgdisk --zap-all -- /dev/sdb
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
command_check_call: Running command: /sbin/sgdisk --clear --mbrtogpt -- /dev/sdb
Creating new GPT entries.
The operation has completed successfully.
update_partition: Calling partprobe on zapped device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
ptype_tobe_for_name: name = journal
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
create_partition: Creating journal partition num 2 size 5120 on /dev/sdb
command_check_call: Running command: /sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:52677a68-3cf4-4d9a-b2d4-8c823e1cb901 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdb
The operation has completed successfully.
update_partition: Calling partprobe on created device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb2 uuid path is /sys/dev/block/8:18/dm/uuid
prepare_device: Journal is GPT partition /dev/disk/by-partuuid/52677a68-3cf4-4d9a-b2d4-8c823e1cb901
prepare_device: Journal is GPT partition /dev/disk/by-partuuid/52677a68-3cf4-4d9a-b2d4-8c823e1cb901
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
set_data_partition: Creating osd partition on /dev/sdb
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
ptype_tobe_for_name: name = data
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
create_partition: Creating data partition num 1 size 0 on /dev/sdb
command_check_call: Running command: /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:a85b0288-85ce-4887-8249-497ba880fe10 --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be --mbrtogpt -- /dev/sdb
The operation has completed successfully.
update_partition: Calling partprobe on created device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is /sys/dev/block/8:17/dm/uuid
populate_data_path_device: Creating xfs fs on /dev/sdb1
command_check_call: Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdb1
meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=6225855 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=24903419, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=12159, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
mount: Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.gjITlj with options noatime,inode64
command_check_call: Running command: /usr/bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.gjITlj
command: Running command: /sbin/restorecon /var/lib/ceph/tmp/mnt.gjITlj
populate_data_path: Preparing osd data dir /var/lib/ceph/tmp/mnt.gjITlj
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.gjITlj/ceph_fsid.2372.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.gjITlj/ceph_fsid.2372.tmp
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.gjITlj/fsid.2372.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.gjITlj/fsid.2372.tmp
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.gjITlj/magic.2372.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.gjITlj/magic.2372.tmp
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.gjITlj/journal_uuid.2372.tmp
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.gjITlj/journal_uuid.2372.tmp
adjust_symlink: Creating symlink /var/lib/ceph/tmp/mnt.gjITlj/journal -> /dev/disk/by-partuuid/52677a68-3cf4-4d9a-b2d4-8c823e1cb901
command: Running command: /sbin/restorecon -R /var/lib/ceph/tmp/mnt.gjITlj
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.gjITlj
unmount: Unmounting /var/lib/ceph/tmp/mnt.gjITlj
command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.gjITlj
get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
command_check_call: Running command: /sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdb
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
update_partition: Calling partprobe on prepared device /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdb /sbin/partprobe /dev/sdb
command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
command_check_call: Running command: /usr/bin/udevadm trigger --action=add --sysname-match sdb1
checking OSD status...
find the location of an executable
Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
Host idcv-ceph3 is now ready for osd use.
# ceph-deploy --overwrite-conf osd activate idcv-ceph0:/dev/sdb1idcv-ceph1:/dev/sdb1idcv-ceph2:/dev/sdb1idcv-ceph3:/dev/sdb1
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.39): /usr/bin/ceph-deploy --overwrite-conf osd activate idcv-ceph0:/dev/sdb1 idcv-ceph1:/dev/sdb1 idcv-ceph2:/dev/sdb1 idcv-ceph3:/dev/sdb1
ceph-deploy options:
username : None
verbose : False
overwrite_conf : True
subcommand : activate
quiet : False
cd_conf :
cluster : ceph
func :
ceph_conf : None
default_release : False
disk : [('idcv-ceph0', '/dev/sdb1', None), ('idcv-ceph1', '/dev/sdb1', None), ('idcv-ceph2', '/dev/sdb1', None), ('idcv-ceph3', '/dev/sdb1', None)]
Activating cluster ceph disks idcv-ceph0:/dev/sdb1: idcv-ceph1:/dev/sdb1: idcv-ceph2:/dev/sdb1: idcv-ceph3:/dev/sdb1:
connected to host: idcv-ceph0
detect platform information from remote host
detect machine type
find the location of an executable
Distro info: CentOS Linux 7.5.1804 Core
activating host idcv-ceph0 disk /dev/sdb1
will use init type: systemd
find the location of an executable
Running command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/sdb1
main_activate: path = /dev/sdb1
get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is /sys/dev/block/8:17/dm/uuid
command: Running command: /usr/sbin/blkid -o udev -p /dev/sdb1
command: Running command: /sbin/blkid -p -s TYPE -o value -- /dev/sdb1
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
mount: Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.X6wbv9 with options noatime,inode64
command_check_call: Running command: /usr/bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.X6wbv9
command: Running command: /usr/sbin/restorecon /var/lib/ceph/tmp/mnt.X6wbv9
activate: Cluster uuid is 812d3acb-eaa8-4355-9a74-64f2cd5209b3
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
activate: Cluster name is ceph
activate: OSD uuid is 3b210c8e-b2ac-4266-9e59-623c031ebb89
activate: OSD id is 0
activate: Marking with init system systemd
command: Running command: /usr/sbin/restorecon -R /var/lib/ceph/tmp/mnt.X6wbv9/systemd
command: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.X6wbv9/systemd
activate: ceph osd.0 data dir is ready at /var/lib/ceph/tmp/mnt.X6wbv9
mount_activate: ceph osd.0 already mounted in position; unmounting ours.
unmount: Unmounting /var/lib/ceph/tmp/mnt.X6wbv9
command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.X6wbv9
start_daemon: Starting ceph osd.0...
command_check_call: Running command: /usr/bin/systemctl disable ceph-osd@0
Removed symlink /etc/systemd/system/ceph-osd.target.wants/ceph-osd@0.service.
command_check_call: Running command: /usr/bin/systemctl disable ceph-osd@0 --runtime
command_check_call: Running command: /usr/bin/systemctl enable ceph-osd@0
Created symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd@0.service to /usr/lib/systemd/system/ceph-osd@.service.
command_check_call: Running command: /usr/bin/systemctl start ceph-osd@0
checking OSD status...
find the location of an executable
Running command: /bin/ceph --cluster=ceph osd stat --format=json
Running command: systemctl enable ceph.target
connection detected need for sudo
connected to host: idcv-ceph1
detect platform information from remote host
detect machine type
find the location of an executable
Distro info: CentOS Linux 7.5.1804 Core
activating host idcv-ceph1 disk /dev/sdb1
will use init type: systemd
find the location of an executable
Running command: sudo /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/sdb1
main_activate: path = /dev/sdb1
get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is /sys/dev/block/8:17/dm/uuid
command: Running command: /sbin/blkid -o udev -p /dev/sdb1
command: Running command: /sbin/blkid -p -s TYPE -o value -- /dev/sdb1
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
mount: Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.zUV3_1 with options noatime,inode64
command_check_call: Running command: /usr/bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.zUV3_1
command: Running command: /sbin/restorecon /var/lib/ceph/tmp/mnt.zUV3_1
activate: Cluster uuid is 812d3acb-eaa8-4355-9a74-64f2cd5209b3
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
activate: Cluster name is ceph
activate: OSD uuid is 2809f370-e6ad-4d29-bf6b-57fe1f2004c6
allocate_osd_id: Allocating OSD id...
command: Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise 2809f370-e6ad-4d29-bf6b-57fe1f2004c6
mount_activate: Failed to activate
unmount: Unmounting /var/lib/ceph/tmp/mnt.zUV3_1
command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.zUV3_1
Traceback (most recent call last):
File "/usr/sbin/ceph-disk", line 9, in
load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5371, in run
main(sys.argv)
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5322, in main
args.func(args)
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3445, in main_activate
reactivate=args.reactivate,
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3202, in mount_activate
(osd_id, cluster) = activate(path, activate_key_template, init)
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3365, in activate
keyring=keyring,
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1013, in allocate_osd_id
raise Error('ceph osd create failed', e, e.output)
ceph_disk.main.Error: Error: ceph osd create failed: Command '/usr/bin/ceph' returned non-zero exit status 1: 2018-07-03 11:47:35.463545 7f83104507000 librados: client.bootstrap-osd authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError
RuntimeError: command returned non-zero exit status: 1
RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/sdb1
2、查看了下idcv-ceph1没有加上去
# lsblk
NAME MAJ:MIN RMSIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K0 disk
sda 8:0 0100G0 disk
├─sda1 8:1 0500M0 part /boot
└─sda2 8:2 0 99.5G0 part
└─centos-root 253:0 0 99.5G0 lvm/
sdb 8:16 0100G0 disk
├─sdb1 8:17 0 95G0 part /var/lib/ceph/osd/ceph-0
└─sdb2 8:18 0 5G0 part
sr0 11:0 1 1024M0 rom
# ceph -s
cluster 812d3acb-eaa8-4355-9a74-64f2cd5209b3
health HEALTH_OK
monmap e2: 3 mons at {idcv-ceph0=172.20.1.138:6789/0,idcv-ceph2=172.20.1.140:6789/0,idcv-ceph3=172.20.1.141:6789/0}
election epoch 8, quorum 0,1,2 idcv-ceph0,idcv-ceph2,idcv-ceph3
osdmap e14: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v27: 64 pgs, 1 pools, 0 bytes data, 0 objects
100 MB used, 284 GB / 284 GB avail
64 active+clean
#
3、使用这个方法赋予角色OSD
# ceph-deploy install --no-adjust-repos --osd idcv-ceph1
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.39): /usr/bin/ceph-deploy install --no-adjust-repos --osd idcv-ceph1
ceph-deploy options:
verbose : False
testing : None
cd_conf :
cluster : ceph
dev_commit : None
install_mds : False
stable : None
default_release : False
username : None
adjust_repos : False
func :
install_mgr : False
install_all : False
repo : False
host : ['idcv-ceph1']
install_rgw : False
install_tests : False
repo_url : None
ceph_conf : None
install_osd : True
version_kind : stable
install_common : False
overwrite_conf : False
quiet : False
dev : master
nogpgcheck : False
local_mirror : None
release : None
install_mon : False
gpg_url : None
Installing stable version jewel on cluster ceph hosts idcv-ceph1
Detecting platform for host idcv-ceph1 ...
connection detected need for sudo
connected to host: idcv-ceph1
detect platform information from remote host
detect machine type
Distro info: CentOS Linux 7.5.1804 Core
installing Ceph on idcv-ceph1
Running command: sudo yum clean all
Loaded plugins: fastestmirror, priorities
Cleaning repos: Ceph Ceph-noarch base ceph-source epel extras updates
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Cleaning up list of fastest mirrors
Running command: sudo yum -y install ceph
Loaded plugins: fastestmirror, priorities
Determining fastest mirrors
base: mirrors.tuna.tsinghua.edu.cn
epel: mirrors.huaweicloud.com
extras: mirror.bit.edu.cn
updates: mirrors.huaweicloud.com
12 packages excluded due to repository priority protections
Package 1:ceph-10.2.10-0.el7.x86_64 already installed and latest version
Nothing to do
Running command: sudo ceph --version
ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
4、节点cpeh1 还是安装不上osd角色，这边准备初始化ceph1重新添加
ceph-deploy purge 节点
ceph-deploy purgedata 节点
清楚安装包和残余数据
ceph-dpeloy install --no-adjust-repos --osd ceph1
直接装包 赋予OSD存储角色之后在添加OSD
具体步骤如下：
ceph-deploy purgeidcv-ceph1
ceph-deploy purgedata idcv-ceph1
ceph-deploy --overwrite-conf osd prepare idcv-ceph1:/dev/sdb
ceph-deploy --overwrite-conf osd activate idcv-ceph1:/dev/sdb1
5、部署成功osd查看集群状态
# ceph -s
cluster 812d3acb-eaa8-4355-9a74-64f2cd5209b3
health HEALTH_OK
monmap e2: 3 mons at {idcv-ceph0=172.20.1.138:6789/0,idcv-ceph2=172.20.1.140:6789/0,idcv-ceph3=172.20.1.141:6789/0}
election epoch 8, quorum 0,1,2 idcv-ceph0,idcv-ceph2,idcv-ceph3
osdmap e27: 4 osds: 4 up, 4 in
flags sortbitwise,require_jewel_osds
pgmap v64: 104 pgs, 6 pools, 1588 bytes data, 171 objects
138 MB used, 379 GB / 379 GB avail
104 active+clean
六、部署RGW服务
1、部署cdph1为对象网关
# ceph-deploy install --no-adjust-repos --rgw idcv-ceph1
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.39): /usr/bin/ceph-deploy install --no-adjust-repos --rgw idcv-ceph1
ceph-deploy options:
verbose : False
testing : None
cd_conf :
cluster : ceph
dev_commit : None
install_mds : False
stable : None
default_release : False
username : None
adjust_repos : False
func :
install_mgr : False
install_all : False
repo : False
host : ['idcv-ceph1']
install_rgw : True
install_tests : False
repo_url : None
ceph_conf : None
install_osd : False
version_kind : stable
install_common : False
overwrite_conf : False
quiet : False
dev : master
nogpgcheck : False
local_mirror : None
release : None
install_mon : False
gpg_url : None
Installing stable version jewel on cluster ceph hosts idcv-ceph1
Detecting platform for host idcv-ceph1 ...
connection detected need for sudo
connected to host: idcv-ceph1
detect platform information from remote host
detect machine type
Distro info: CentOS Linux 7.5.1804 Core
installing Ceph on idcv-ceph1
Running command: sudo yum clean all
Loaded plugins: fastestmirror, priorities
Cleaning repos: Ceph Ceph-noarch base ceph-source epel extras updates
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Cleaning up list of fastest mirrors
Running command: sudo yum -y install ceph-radosgw
Loaded plugins: fastestmirror, priorities
Determining fastest mirrors
base: mirrors.aliyun.com
epel: mirrors.aliyun.com
extras: mirrors.aliyun.com
updates: mirror.bit.edu.cn
12 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package ceph-radosgw.x86_64 1:10.2.10-0.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
ceph-radosgw x86_64 1:10.2.10-0.el7 Ceph 266 k
Transaction Summary
================================================================================
Install1 Package
Total download size: 266 k
Installed size: 795 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 1:ceph-radosgw-10.2.10-0.el7.x86_64 1/1
Verifying: 1:ceph-radosgw-10.2.10-0.el7.x86_64 1/1
Installed:
ceph-radosgw.x86_64 1:10.2.10-0.el7
Complete!
Running command: sudo ceph --version
ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
2、设置idcv-ceph1为管理网关
# ceph-deploy admin idcv-ceph1
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.39): /usr/bin/ceph-deploy admin idcv-ceph1
ceph-deploy options:
username : None
verbose : False
overwrite_conf : False
quiet : False
cd_conf :
cluster : ceph
client : ['idcv-ceph1']
func :
ceph_conf : None
default_release : False
Pushing admin keys and conf to idcv-ceph1
connection detected need for sudo
connected to host: idcv-ceph1
detect platform information from remote host
detect machine type
write cluster configuration to /etc/ceph/{cluster}.conf
3、创建生成网关实例idcv-ceph1
# ceph-deploy rgw create idcv-ceph1
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.39): /usr/bin/ceph-deploy rgw create idcv-ceph1
ceph-deploy options:
username : None
verbose : False
rgw : [('idcv-ceph1', 'rgw.idcv-ceph1')]
overwrite_conf : False
subcommand : create
quiet : False
cd_conf :
cluster : ceph
func :
ceph_conf : None
default_release : False
Deploying rgw, cluster ceph hosts idcv-ceph1:rgw.idcv-ceph1
connection detected need for sudo
connected to host: idcv-ceph1
detect platform information from remote host
detect machine type
Distro info: CentOS Linux 7.5.1804 Core
remote host will use systemd
deploying rgw bootstrap to idcv-ceph1
write cluster configuration to /etc/ceph/{cluster}.conf
rgw keyring does not exist yet, creating one
create a keyring file
create path recursively if it doesn't exist
Running command: sudo ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.idcv-ceph1 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.idcv-ceph1/keyring
Running command: sudo systemctl enable ceph-radosgw@rgw.idcv-ceph1
Created symlink from /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.idcv-ceph1.service to /usr/lib/systemd/system/ceph-radosgw@.service.
Running command: sudo systemctl start ceph-radosgw@rgw.idcv-ceph1
Running command: sudo systemctl enable ceph.target
The Ceph Object Gateway (RGW) is now running on host idcv-ceph1 and default port 7480
4、测试网关服务
# curl 172.20.1.139:7480
页:
[1]