OpenStack Kilo版加CEPH部署手册
OpenStack Kilo版加CEPH部署手册作者: yz
联系方式: QQ: 949587200
日期: 2015-7-13
版本: Kilo
转载地址：
http://mp.weixin.qq.com/s__biz=MzAxOTAzMDEwMA==&mid=209225237&idx=1&sn=357afdabafc03e8fb75eb4e1fb9d4bf9&scene=5&ptlang=2052&ADUIN=724042315&ADSESSION=1437612842&ADTAG=CLIENT.QQ.5389_.0&ADPUBNO=26466#rd
 
目录
实验环境
前言
架构图
架构部署
服务器系统安装
全局工作
导入软件源
Iptables服务安装
NTP服务安装
Ceph安装
第一个监视器节点安装
其余监视器节点安装
部署OSD
与Openstack结合
OpenStack控制节点安装
Ceph配置
Mysql服务安装
Rabbitmq服务安装
Keystone服务安装
Glance服务安装
Neutron服务安装
Nova服务安装
Cinder服务安装
计算节点安装
Ceph配置
Neutron服务安装
Nova服务安装
Horizon安装
实验环境
硬件:
曙光 I610r-GV(1台)
CPU：Intel(R) Xeon(R) CPU E5606@ 2.13GHz * 1
内存：32GB
硬盘：SAS 10K 300G * 1
网卡：Intel Corporation 82574L Gigabit Network Connection * 2
网卡：Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection(双端口) * 1
曙光 I610r-GV(3台)
CPU：Intel(R) Xeon(R) CPU E5606@ 2.13GHz * 1
内存：32GB
硬盘：SAS 10K 300G * 1，SSD 160G * 3
网卡：Intel Corporation 82574L Gigabit Network Connection * 2
网卡：Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection(双端口) * 1
系统：
CentOS 7.1 x64
OpenStack版本：
Kilo（2015.1.0）
Ceph版本：
Hammer（0.94.2）
前言
1. 文档很初级，只是为了带新手入门，很多参数使用默认值
2. 文档中Ceph不使用ceph-deploy部署
3. 后续会有telemetry，lbaas，sahara，swift，trove部署文档
4. 如果网卡不够，可以将管理，虚机，存储合并为一个网络
5. 文档中ceph只建立一个池子
6. rdo中某些服务的conf文件红帽有一些修改，如果大家享用官方原版可以在launchpad下载源码包自行生成（这是说给强迫症用户的）
7. dvr模式下每个l3节点所绑定网卡都需要有外网ip
架构图
架构部署
服务器系统安装
1. CentOS 7.1 x64使用最小化安装方式
2. 设置主机名，关闭selinux基础工作不在文档出现
全局工作
导入软件源
1. 导入软件源
rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -ivh https://repos.fedorapeople.org/repos/openstack/openstack-kilo/rdo-release-kilo-1.noarch.rpm
rpm --import 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc'
2. 新建/etc/yum.repos.d/ceph.repo文件添加如下内容
name=Ceph packages for $basearch
baseurl=http://ceph.com/rpm-hammer/el7/$basearch
enabled=1
priority=2
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
name=Ceph noarch packages
baseurl=http://ceph.com/rpm-hammer/el7/noarch
enabled=1
priority=2
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
Iptables服务安装
1. 安装Iptables
yum install -y iptables-services
2. 启动并设置开机自启动
systemctl stop firewalld
systemctl disable firewalld
systemctl start iptables
systemctl enable iptables
NTP服务安装
1. 安装NTP服务
yum install -y ntp
2. 启动并设置开机自启动
systemctl start ntpd
systemctl enable ntpd
Ceph安装
第一个监视器节点安装
1. 安装Ceph
yum install -y ceph
2. 生成集群uuid
uuidgen
3. 新建/etc/ceph/ceph.conf文件添加如下内容
fsid = 第二步生成的值
mon initial members = Kilo-com-1,
mon host = 20.0.0.3,
4. 建立拥有操作监视器权限的凭证
ceph-authtool --create-keyring /etc/ceph/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
5. 建立一个名称为client.admin的管理员凭证
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
6. 将管理员凭证添加到监视器权限凭证内
ceph-authtool /etc/ceph/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
7. 建立监视器节点对应关系地图
monmaptool --create --add Kilo-com-1 20.0.0.3--fsid 第二步生成的值也是 /tmp/monmap
8. 建立监视器服务数据存放目录
mkdir /var/lib/ceph/mon/ceph-Kilo-com-1
9. 初始化监视器数据目录
ceph-mon --mkfs -i Kilo-com-1--monmap /tmp/monmap --keyring /etc/ceph/ceph.mon.keyring
10. 编辑/etc/ceph/ceph.conf文件添加如下内容
public network = 20.0.0.0/24
cluster network = 30.0.0.0/24
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
filestore xattr use omap = true
osd pool default pg num = 256
osd pool default pgp num = 256
11. 添加部署完毕标志文件及服务启动标志文件
touch /var/lib/ceph/mon/ceph-Kilo-com-1/done
touch /var/lib/ceph/mon/ceph-Kilo-com-1/sysvinit
12. 启动监视器服务
/etc/init.d/ceph start mon
13. 查看状态
属于正常状态
14. 添加防火墙规则允许其他节点连接
iptables -I INPUT 2 -s 20.0.0.0/24 -p tcp -m tcp --dport 6789 -j ACCEPT
iptables-save > /etc/sysconfig/iptables
其余监视器节点安装
1. 安装Ceph
yum install -y ceph
2. 从第一监控器节点靠谱配置文件和凭证
scp /etc/ceph/ceph.client.admin.keyring /etc/ceph/ceph.mon.keyring /etc/ceph/ceph.conf 10.0.0.4:/etc/ceph/
3. 获取监视器节点对应关系地图
ceph mon getmap -o /tmp/monmap
4. 建立监视器服务数据存放目录
mkdir /var/lib/ceph/mon/ceph-Kilo-com-2
5. 初始化监视器数据目录
ceph-mon --mkfs -i Kilo-com-2 --monmap /tmp/monmap --keyring /etc/ceph/ceph.mon.keyring
6. 编辑/etc/ceph/ceph.conf文件添加如下内容
mon initial members = 追加当前节点主机名,并以逗号结尾
mon host = 追加当前节点虚机网络，并以逗号结尾
7. 添加部署完毕标志文件及服务启动标志文件
touch /var/lib/ceph/mon/ceph-Kilo-com-2/done
touch /var/lib/ceph/mon/ceph-Kilo-com-2/sysvinit
8. 启动Ceph mon节点服务
/etc/init.d/ceph start mon
9. 添加防火墙规则允许其他节点连接
iptables -I INPUT 2 -s 20.0.0.0/24 -p tcp -m tcp --dport 6789 -j ACCEPT
iptables-save > /etc/sysconfig/iptables
10. 添加当前监视器到监视器集群
ceph mon add Kilo-com-2 20.0.0.4:6789
11. 将/etc/ceph/ceph.conf覆盖到其他节点的ceph目录并重启服务
部署OSD
1. 申请OSD号，计划当前节点有几个OSD就执行几次，输出结果后续使用，我这里每个节点三个OSD,我需要执行三次
ceph osd tree
2. 通过OSD号创建OSD数据目录
mkdir /var/lib/ceph/osd/ceph-0
mkdir /var/lib/ceph/osd/ceph-1
mkdir /var/lib/ceph/osd/ceph-2
3. 格式化OSD使用磁盘为xfs文件系统
mkfs.xfs /dev/sdb
mkfs.xfs /dev/sdc
mkfs.xfs /dev/sdd
4. 编辑/etc/fstab添加如下内容
/dev/sdb /var/lib/ceph/osd/ceph-0 xfs defaults 0 0
/dev/sdb /var/lib/ceph/osd/ceph-0 xfs remount,user_xattr 0 0
/dev/sdc /var/lib/ceph/osd/ceph-1 xfs defaults 0 0
/dev/sdc /var/lib/ceph/osd/ceph-1 xfs remount,user_xattr 0 0
/dev/sdd /var/lib/ceph/osd/ceph-2 xfs defaults 0 0
/dev/sdd /var/lib/ceph/osd/ceph-2 xfs remount,user_xattr 0 0
5. 挂载磁盘
mount /dev/sdb
mount /dev/sdc
mount /dev/sdd
6. 初始化OSD数据目录
ceph-osd -i 0 --mkfs --mkjournal --mkkey
ceph-osd -i 1 --mkfs --mkjournal --mkkey
ceph-osd -i 2 --mkfs --mkjournal --mkkey
7. 注册OSD凭证
ceph auth add osd.0 osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-0/keyring
ceph auth add osd.1 osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-1/keyring
ceph auth add osd.2 osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-2/keyring
8. 添加OSD到CRUSH关系图
ceph osd crush add-bucket Kilo-com-1 host
9. 将节点移动到default树下
ceph osd crush move Kilo-com-1 root=default
10. 将OSD添加到Kilo-com-1下
ceph osd crush add osd.0 1.0 host=Kilo-com-1
ceph osd crush add osd.1 1.0 host=Kilo-com-1
ceph osd crush add osd.2 1.0 host=Kilo-com-1
11. 添加服务启动标志文件
touch /var/lib/ceph/osd/ceph-0/sysvinit
touch /var/lib/ceph/osd/ceph-1/sysvinit
touch /var/lib/ceph/osd/ceph-2/sysvinit
12. 添加防火墙规则
iptables -I INPUT 2 -s 20.0.0.0/24 -p tcp -m multiport --dports 6800:6900 -j ACCEPT
iptables -I INPUT 2 -s 30.0.0.0/24 -p tcp -m multiport --dports 6800:6900 -j ACCEPT
iptables-save > /etc/sysconfig/iptables
13. 启动当前节点OSD
/etc/init.d/ceph start osd
14. 查看当前OSD树状态
15. 查看ceph状态
会看到健康状态是HEALTH_WARN，原因是PGs过少,我们来添加
16. 增加PG和PGP数量
ceph osd pool set rbd pg_num 256
ceph osd pool set rbd pgp_num 256
注：如果在执行第二条命令时报Error EBUSY: currently creating pgs, wait，说明第一条命令还没处理完
17. 再次查看ceph状态
与Openstack结合
1. 建立存储池
ceph osd pool create storages 256
2. 建立一个名称为client.storages的凭证，对storages池有操作权限
ceph auth get-or-create client.storages mon 'allow rx' osd 'allow class-read object_prefix rbd_children, allow rwx pool=storages'
OpenStack控制节点安装
Ceph配置
1. 安装ceph-common
yum install -y ceph-common
2. 拷贝配置文件和管理员凭证到本机
scp /etc/ceph/ceph.client.admin.keyring /etc/ceph/ceph.conf 10.0.0.2:/etc/ceph/
3. 导出client.storages凭证
ceph auth get-or-create client.storages >> /etc/ceph/ceph.client.storages.keyring
4. 安装libvirt
yum install -y libvirt
5. 编辑/etc/libvirt/libvirtd.conf文件添加或修改如下行
listen_tls = 0
listen_tcp = 1
listen_addr = "0.0.0.0"
auth_tcp = "none"
6. 编辑/etc/sysconfig/libvirtd文件添加或修改如下行
LIBVIRTD_ARGS="--listen"
7. 启动并设置开机自启动
systemctl start libvirtd
systemctl enable libvirtd
8. 添加防火墙规则
iptables -I INPUT 2 -s 10.0.0.0/24 -p tcp -m tcp --dport 16509 -j ACCEPT
iptables-save > /etc/sysconfig/iptables
9. 生成libvirt secret使用uuid
uuidgen
10. 新建ceph-storages-secrets.xml文件添加如下内容
<secret ephemeral='no' private='no'>
<uuid>第九步生成值</uuid>
<usage type='ceph'>
<name>client.storages secret</name>
</usage>
</secret>
11. 设置libvirt secret值
virsh secret-set-value --secret第九步生成值 --base64 $(ceph auth get-key client.storages)
Mysql服务安装
1. 安装MYSQL服务
yum install -y mariadb-server
2. 编辑/etc/my.cnf文件的下添加如下内容
collation-server = utf8_general_ci
init-connect = 'SET NAMES utf8'
character-set-server = utf8
skip-name-resolve
skip-host-cache
3. 启动并设置开机自启动
systemctl start mariadb
systemctl enable mariadb
4. 初始化mysql设置root密码为openstack
mysql_secure_installation#此命令为交互命令，过程自行解决
5. 添加防火墙规则
iptables -I INPUT 2 -s 10.0.0.0/24 -p tcp -m tcp --dport 3306 -j ACCEPT
iptables-save > /etc/sysconfig/iptables
Rabbitmq服务安装
1. 安装RABBITMQ服务
yum -y install rabbitmq-server
2. 启动并设置开机自启动
systemctl start rabbitmq-server
systemctl enable rabbitmq-server
3. 更改RABBITMQ消息队列服务guest用户默认密码为openstack
rabbitmqctl change_password guest openstack
4. 添加防火墙规则
iptables -I INPUT 2 -s 10.0.0.0/24 -p tcp -m tcp --dport 5672 -j ACCEPT
iptables-save > /etc/sysconfig/iptables
Keystone服务安装
1. 安装Keystone
yum install -y openstack-keystone
2. 生成管理员token
openssl rand -hex 10
3. 建立数据库及数据库用户
mysql -uroot -popenstack -e 'create database keystone'
mysql -uroot -popenstack -e 'grant all on keystone.* to"keystone"@"%" identified by "keystone"'
4. 编辑/etc/keystone/keystone.conf文件添加或修改如下行
admin_token = 第二步生成的值
log_dir = /var/log/keystone
use_stderr = false
connection = mysql://keystone:keystone@10.0.0.2/keystone
use_db_reconnect = true
rabbit_host = 10.0.0.2
rabbit_password = openstack
5. 初始化数据库
su -s /bin/sh -c 'keystone-manage db_sync' keystone
6. 启动并设置开机自启动
systemctl start openstack-keystone
systemctl enable openstack-keystone
7. 添加防火墙规则
iptables -I INPUT 2 -s 10.0.0.0/24 -ptcp -m tcp --dport 5000 -j ACCEPT
iptables -I INPUT 2 -s 10.0.0.0/24 -ptcp -m tcp --dport 35357 -j ACCEPT
iptables-save > /etc/sysconfig/iptables
8. 建立tenant，user,role,service,endpoint
export OS_SERVICE_TOKEN=第二步生成的值
export OS_SERVICE_ENDPOINT=http://10.0.0.2:35357/v2.0
keystone tenant-create --name admin --description "AdminTenant"
keystone user-create --name admin --pass admin --email admin@example.com
keystone role-create --name admin
keystone user-role-add --tenant admin --user admin --role admin
keystone role-create --name _member_
keystone user-role-add --tenant admin --user admin --role _member_
keystone tenant-create --name service --description "ServiceTenant"
keystone service-create --name keystone --type identity --description"OpenStack Identity Service"
keystone endpoint-create --service-id $(keystone service-list | awk '/identity / {print $2}') --publicurl http://10.0.0.2:5000/v2.0 --internalurlhttp://10.0.0.2:5000/v2.0--adminurlhttp://10.0.0.2:35357/v2.0 --region regionOne
9. 新建/root/admin-openrc文件添加如下行，作为以后新建公用镜像，网络使用的认证凭证
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://10.0.0.2:35357/v2.0
Glance服务安装
1. 安装Glance
yum install -y openstack-glance
2. 建立user,role,service,endpoint
export OS_SERVICE_TOKEN=安装Keystone时第二步生成的值
exportOS_SERVICE_ENDPOINT=http://10.0.0.2:35357/v2.0
keystone user-create --name glance--pass glance
keystone user-role-add --user glance--tenant service --role admin
keystone service-create --name glance--type image --description "OpenStack Image Service"
keystone endpoint-create --service-id$(keystone service-list | awk '/ image / {print $2}') --publicurlhttp://10.0.0.2:9292 --internalurl http://10.0.0.2:9292 --adminurlhttp://10.0.0.2:9292 --region regionOne
3. 建立数据库及数据库用户
mysql -uroot -popenstack -e 'createdatabase glance'
mysql -uroot -popenstack -e 'grant allon glance.* to "glance"@"%" identified by"glance"'
4. 编辑/etc/glance/glance-api.conf文件添加或修改如下行
use_stderr = false
show_image_direct_url = true
rabbit_host = 10.0.0.2
rabbit_password = openstack
connection = mysql://glance:glance@10.0.0.2/glance
use_db_reconnect = true
auth_uri = http://10.0.0.2:5000
identity_uri = http://10.0.0.2:35357
admin_tenant_name = service
admin_user = glance
admin_password = glance
flavor = keystone
stores = glance.store.rbd.Store,
default_store = rbd
rbd_store_user = storages
rbd_store_pool = storages
5. 编辑/etc/glance/glance-registry.conf文件添加或修改如下行
use_stderr = false
rabbit_host = 10.0.0.2
rabbit_password = openstack
connection = mysql://glance:glance@10.0.0.2/glance
use_db_reconnect = true
auth_uri = http://10.0.0.2:5000
identity_uri = http://10.0.0.2:35357
admin_tenant_name = service
admin_user = glance
admin_password = glance
flavor = keystone
6. 初始化数据库
su -s /bin/sh -c 'glance-manage db_sync' glance
7. 启动并设置开机自启动
systemctl start openstack-glance-api
systemctl startopenstack-glance-registry
systemctl enable openstack-glance-api
systemctl enableopenstack-glance-registry
8. 添加防火墙规则
iptables -I INPUT 2 -s 10.0.0.0/24 -ptcp -m tcp --dport 9191 -j ACCEPT
iptables -I INPUT 2 -s 10.0.0.0/24 -ptcp -m tcp --dport 9292 -j ACCEPT
iptables-save > /etc/sysconfig/iptables
Neutron服务安装
1. 安装Neutron
yum install -y openstack-neutron openstack-neutron-ml2openstack-neutron-openvswitch
2. 建立user,role,service,endpoint
keystone user-create --name neutron--pass neutron
keystone user-role-add --user neutron--tenant service --role admin
keystone service-create --name neutron--type network --description "OpenStack Network Service"
keystone endpoint-create --service-id$(keystone service-list | awk '/ network / {print $2}') --publicurlhttp://10.0.0.2:9696 --adminurl http://10.0.0.2:9696 --internalurl http://10.0.0.2:9696--region regionOne
3. 建立数据库及数据库用户
mysql -uroot -popenstack -e 'createdatabase neutron'
mysql -uroot -popenstack -e 'grant allon neutron.* to "neutron"@"%" identified by"neutron"'
4. 编辑/etc/neutron/neutron.conf文件添加或修改如下行
router_distributed = true
use_stderr = false
log_dir = /var/log/neutron
core_plugin = ml2
service_plugins = router
auth_strategy = keystone
host = Kilo-con
allow_overlapping_ips = true
notify_nova_on_port_status_changes =true
notify_nova_on_port_data_changes = true
nova_url = http://10.0.0.2:8774/v2
nova_region_name = regionOne
nova_admin_username = nova
nova_admin_tenant_name = service
nova_admin_password = nova
nova_admin_auth_url = http://10.0.0.2:35357/v2.0
rabbit_host = 10.0.0.2
rabbit_password = openstack
root_helper = sudo neutron-rootwrap/etc/neutron/rootwrap.conf
auth_uri = http://10.0.0.2:5000
identity_uri = http://10.0.0.2:35357
admin_tenant_name = service
admin_user = neutron
admin_password = neutron
connection =mysql://neutron:neutron@10.0.0.2/neutron
use_db_reconnect = true
rabbit_host = 10.0.0.2
rabbit_password = openstack
5. 编辑/etc/neutron/dhcp_agent.ini文件添加或修改如下行
interface_driver =neutron.agent.linux.interface.OVSInterfaceDriver
dnsmasq_config_file =/etc/neutron/neutron-dnsmasq.conf
6. 新建/etc/neutron/neutron-dnsmasq.conf添加如下行
dhcp-option-force=26,1450
log-facility =/var/log/neutron/dnsmasq.log
7. 编辑/etc/neutron/l3_agent.ini文件添加或修改如下行
interface_driver =neutron.agent.linux.interface.OVSInterfaceDriver
handle_internal_only_routers = false
enable_metadata_proxy = false
agent_mode = dvr_snat
8. 编辑/etc/neutron/plugins/ml2/ml2_conf.ini文件添加或修改如下行
type_drivers = flat,vxlan
tenant_network_types = vxlan
mechanism_drivers = openvswitch,l2population
vni_ranges = 1000:5000
9. 编辑/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini文件添加或修改如下行
local_ip = 20.0.0.2
bridge_mappings = external:br-ex
tunnel_types = vxlan
vxlan_udp_port = 4789
l2_population = true
arp_responder = true
enable_distributed_routing = true
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
10.建立ml2插件配置文件软链
ln -sv /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini
11.编辑/etc/sysctl.conf文件添加或修改如下行
net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
12.重加载内核参数
sysctl -p
13.初始化数据库
su -s /bin/sh -c "neutron-db-manage upgrade kilo" neutron
14.启动并设置开机自启动openvswitch
systemctl start openvswitch
systemctl enable openvswitch
15.建立openvswitch端口
ovs-vsctl add-br br-int
ovs-vsctl add-br br-ex
16.像br-ex添加网卡
ovs-vsctl add-port br-ex eth3
17.启动并设置开机自启动neutron服务
systemctl start neutron-server
systemctl start neutron-dhcp-agent
systemctl startneutron-openvswitch-agent
systemctl start neutron-l3-agent
systemctl start neutron-ovs-cleanup
systemctl start neutron-netns-cleanup
systemctl enable neutron-server
systemctl enable neutron-dhcp-agent
systemctl enableneutron-openvswitch-agent
systemctl enable neutron-l3-agent
systemctl enable neutron-ovs-cleanup
systemctl enable neutron-netns-cleanup
18.添加防火墙规则
iptables -I INPUT 2 -s 10.0.0.0/24 -ptcp -m tcp --dport 9696 -j ACCEPT
iptables-save > /etc/sysconfig/iptables
19.建立内网，外网，路由及绑定子网和接口到路由器
source /root/admin-openrc
neutron net-create --shared--provider:network_type vxlan internal-network
neutron subnet-create internal-network100.100.100.0/24 --name internal-network-subnet --gateway 100.100.100.1--allocation-pool start=100.100.100.10,end=100.100.100.200 --enable-dhcp--ip-version 4 --dns-nameserver 202.106.0.20
neutronnet-create ext-net --shared --router:external --provider:network_type flat--provider:physical_network external
neutronsubnet-create ext-net 200.200.200.0/24 --name ext-network-subnet --gateway200.200.200.1 --allocation-pool start=200.200.200.10,end=200.200.200.200--disable-dhcp --ip-version 4 --dns-nameserver 202.106.0.20
neutronrouter-create router
neutronrouter-interface-add router internal-network-subnet
neutronrouter-gateway-set router ext-net
页:
[1]