Centos6 安装cdh5.7
Centos6 安装cdh5.7[*] 安装前的准备工作
[*] JDK环境
版本：jdk-7u79-linux-x64.tar.gz
下载地址：oracle官网（http://www.oracle.com/technetwork/cn/java/javase/downloads/jdk7-downloads-1880260-zhs.html）
安装上 会生成 /usr/local/jdk1.7.0_79
需要建立软连接 ln –s /usr/local/jdk1.7.0_79 /usr/local/java
在来一个 ln –s /usr/local/java /usr/share/java
还来一个 ln -s /usr/local/java /usr/java/jdk1.7
加环境变量
export JAVA_HOME=/usr/local/java
export PATH=${JAVA_HOME}/bin:${PATH}
[*] mysql连接
http://dev.mysql.com/downloads/connector/j/
mysql-connector-java-5.1.47-bin.jar 我下载的这个
把下载的包放到/usr/local/
并重新命名 mysql-connector-java.jar
http://i2.运维网.com/images/blog/201811/13/266d5c74d5e25aec848f88363c07bcf4.jpg
[*] CDH
先下载 cloudera manager安装包 下载地址：http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6- cm5.7.0_x86_64.tar.gz
http://i2.运维网.com/images/blog/201811/13/8a0e21fcdccb599fd1ac0e43c73d8c01.jpg
下载 CDH 安装包（parcel 包）下载地址：http://archive.cloudera.com/cdh5/parcels/5.7.0/
http://i2.运维网.com/images/blog/201811/13/315501b46449da6e64a28863556c628a.jpg
http://i2.运维网.com/images/blog/201811/13/fbc51afb04ff1093e7ab0481daef3849.jpg
CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel
CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel.sha1
manifest.json
需要上面这3个
[*] 系统需要做的准备
Ntp 时间 所有机器的时间要一样 （百度吧不会的简单）
Ssh 互相（root用户）
防火墙 selinux都不开（没在考虑范围）
Hosts 里面要有所有服务器的解析 Ip hostname
echo 0 > /proc/sys/vm/swappiness
echo never > /sys/kernel/mm/transparent_hugepage/defrag
自己设置开机启动吧
[*]安装cdh
-----------------------------------------------------------manager机器---------------------------------------------------先操作 cloudera-manager 机器
mkdir -p /var/cloudera-scm-server
mkdir -p /opt/cloudera/parcel-repo
tar xf /opt/cloudera-manager-el6-cm5.7.0_x86_64.tar.gz -C /opt/cloudera-manager
http://i2.运维网.com/images/blog/201811/13/cf83c30ffbedea6eb63b2cd49e5f4348.jpg
把另外一个包放到这/opt/cloudera/parcel-repo（注意把sha1 换成这样sha）
http://i2.运维网.com/images/blog/201811/13/628706b396546fc28eab833218ce9e8e.jpg
useradd --system --home=/opt/cloudera-manager/cm-5.7.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
chown cloudera-scm:cloudera-scm /var/cloudera-scm-server
chown cloudera-scm:cloudera-scm /opt/cloudera-manager
chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
-------------------------------------------------- manager机器-----------------------------------------------------------
-------------------------------------------------- 所有机器----------------------------------------------------------------
所有机器加用户 cloudera-scm
useradd --system --home=/opt/cloudera-manager/cm-5.7.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
vim /opt/cloudera-manager/cm-5.7.0/etc/cloudera-scm-agent/config.ini
# Hostname of the CM server.
server_host=node2 这个要指向manager的机器的地址 ip 也行
http://i2.运维网.com/images/blog/201811/13/b8d11149ea2d16e23756ae13b6d4250b.jpg
-------------------------------------------------- 所有机器----------------------------------------------------------------
-------------------------------------------------- manager机器-----------------------------------------------------------
在manager的机器指向cm server的数据库初始化
方法一 ：
/opt/cloudera-manager/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql –h1.1.1.1 -uroot –pxxxx --scm-host 2.2.2.2 scm scm scm
上面这个命令需要root权限 没有scm库才执行
http://i2.运维网.com/images/blog/201811/13/121e601cb36089240d0e95e86fe01e44.jpg
方法二：（建议这样，数据库我们有dba管理人家不能给你root给你一个库的权限）
create database scm DEFAULT CHARSET utf8 COLLATE utf8_general_ci; 建库
grant all privileges on scm.* to scm@'node2' identified by 'scm'; 授权
http://i2.运维网.com/images/blog/201811/13/ef21c105aa7d70c8c1fea5402ab88cc0.png
/opt/cloudera-manager/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql -hnode1 --scm-host node2 scm scm scm #node1是数据库的地址 Node2是manager地址 scm库 scm 用户 scm密码
http://i2.运维网.com/images/blog/201811/13/36be3c31c35f97a58118933a2ac58622.jpg
vim /opt/cloudera-manager/cm-5.7.0/etc/cloudera-scm-server/db.properties 看看生成什么
http://i2.运维网.com/images/blog/201811/13/7f48478f445464eba603f6a457e175e1.png
/opt/cloudera-manager/cm-5.7.0/etc/init.d/cloudera-scm-server start 启动服务
如果启动失败需要修改 /etc/init.d/cloudera-scm-server
CMF_DEFAULTS=/opt/cloudera-manager/cm-5.7.0/etc/default
http://i2.运维网.com/images/blog/201811/13/6b8e623f12f6e3f49392b7f720edbe67.jpg
添加系统服务：
cp /opt/cloudera-manager/cm-5.7.0/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
chkconfig --add cloudera-scm-server
chkconfig cloudera-scm-server on
http://i2.运维网.com/images/blog/201811/13/7a693463d1a8c8a956a1d863ff4e501e.jpg
-------------------------------------------------- manager机器-----------------------------------------------------------
-------------------------------------------------- 所有机器-----------------------------------------------------------
同理把/opt/cloudera-manager/cm-5.7.0/etc/init.d/下的cloudera-scm-agent 也设置成系统服务并启动
cp cloudera-scm-agent /etc/init.d/
chkconfig --add cloudera-scm-agent
vim /etc/init.d/cloudera-scm-agent 不这样会报启动命令里面的命令找不到路径
CMF_DEFAULTS=/opt/cloudera-manager/cm-5.7.0/etc/default
http://i2.运维网.com/images/blog/201811/13/d4544d5b469896eee38ecfd3ab633ed5.jpg
/etc/init.d/cloudera-scm-agent start 启动
http://i2.运维网.com/images/blog/201811/13/a584c2d45fe5c4b3be703ea8f3e26a83.jpg
把 /etc/init.d/cloudera-scm-agent 复制到全部机器
http://i2.运维网.com/images/blog/201811/13/4f443ceb647923ace66d23a94ec8d24c.png
vi /opt/cloudera-manager/cm-5.7.0/etc/cloudera-scm-agent/config.ini 注意这个文件 到要指定到manager那台机器
http://i2.运维网.com/images/blog/201811/13/74ea50a7c139bff27a85fd1dbaefeced.jpg
每台都加系统服务和启动
http://i2.运维网.com/images/blog/201811/13/a3ce464146aceb4281afdd084bd4f9f9.jpg
Web页面登录cdh安装其他软件
7180端口访问 manager的机器 admin admin 默认
http://i2.运维网.com/images/blog/201811/13/f0cc114c1c612067bfcc4b87e5cc59da.png
http://i2.运维网.com/images/blog/201811/13/bb58c5530203a8b81b2e90ca91d8b251.jpg
http://i2.运维网.com/images/blog/201811/13/f4b29f2333b37f0fd8c364abe17a9360.jpg
http://i2.运维网.com/images/blog/201811/13/af32b710dbfdc421be5cc99d173c5325.jpg
http://i2.运维网.com/images/blog/201811/13/a6bf998f8ec0d72ab8d468bb93d8f5c2.png
这里就注意了选择新机器 如果没有安装刚才的agent这里也能安装比咱们上面的简单
下面是版本的选择
http://i2.运维网.com/images/blog/201811/13/1672d1fc9f9ee19b2eaf5885f95e3dfa.jpg
http://i2.运维网.com/images/blog/201811/13/3b0f67d692a4c178123f71361fb59052.jpg
http://i2.运维网.com/images/blog/201811/13/a2750adff71bda91081e6638f850511d.jpg
http://i2.运维网.com/images/blog/201811/13/42e24a0a4d029f56705f69de09afc91d.png
http://i2.运维网.com/images/blog/201811/13/d789be345a4a5397721b232571a30af5.png
到这就安装了一个没有安装agent的机器
选择当前的管理主机
http://i2.运维网.com/images/blog/201811/13/dfaf63a0e5daa697a4f87336b885fedd.png
http://i2.运维网.com/images/blog/201811/13/c18de9c7f36e0236f56661dd8659aa32.jpg
http://i2.运维网.com/images/blog/201811/13/277349ddf011fe0ff2a28d16d171c06b.jpg
http://i2.运维网.com/images/blog/201811/13/3d6e7099d43bea43f887d836474840d1.jpg
http://i2.运维网.com/images/blog/201811/13/f4244c5988b6a59a38adc94ff5b1ab9d.jpg
http://i2.运维网.com/images/blog/201811/13/c3f84412270fa165a89e9f760f28b8da.png
http://i2.运维网.com/images/blog/201811/13/b6fee4409589954d7d79b9ec47c617a6.jpg
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci; 记得授权数据库
grant all privileges on *.* to hive@'%' identified by 'xxx';
点击测试通过就行
http://i2.运维网.com/images/blog/201811/13/d9da99fafedb022ad0b48e7c76588b64.jpg
http://i2.运维网.com/images/blog/201811/13/6d999391011a2b83d6fce6699cf37eec.jpg
http://i2.运维网.com/images/blog/201811/13/8f4f01da865d3e7a233f48c59ff3a0b0.jpg
http://i2.运维网.com/images/blog/201811/13/20f2a24519bf0ff81a0951941c4acb8c.jpg
/etc/sudoers里面的Defaults requiretty需要注释掉。
http://i2.运维网.com/images/blog/201811/13/cf5d671a4ff99a6f14b3acf8d2e556cb.jpg
在安装服务界面，执行失败时，点击查看详细信息，会给出这样类似的错误信息。查看/opt/cloudera-manager/cm-5.4.1/run/cloudera-scm-agent/process目录下相关日志。 发现上面这个错误信息是由于JAVA_HOME问题，启动脚本中，JAVA_HOME并不是根据环境变量来获取，而是在它列举的几个目录下寻找，所以只要把你的java路径ln -s过去就行，比如我的 ln -s /usr/local/java /usr/java/jdk1.7
如果还是不行后退一步在下一步
http://i2.运维网.com/images/blog/201811/13/8b65521a5440ed7ca2e58fc5869e5987.jpg
http://i2.运维网.com/images/blog/201811/13/da2392c3b2b795e6f7f44293beaef76d.jpg
http://i2.运维网.com/images/blog/201811/13/b8d96b6ff12349af47bf472f179d0e13.jpg
到这manager就完成了安装
添加服务oozie
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci; 建库授权
grant all privileges on *.* to oozie@'%' identified by 'xxxx';
http://i2.运维网.com/images/blog/201811/13/1b2003b4c297708dad7f3804a383ca16.jpg
http://i2.运维网.com/images/blog/201811/13/9f6152fdf74a7bb256887d6707100040.jpg
http://i2.运维网.com/images/blog/201811/13/b03b8c135bb627dcb78a4e40bf457bb4.jpg
http://i2.运维网.com/images/blog/201811/13/14b7e85eb61db899ba16f553e75249d4.jpg
http://i2.运维网.com/images/blog/201811/13/c950ccb3613e7c7366dd5c35b7899c41.jpg
http://i2.运维网.com/images/blog/201811/13/16da7205a0d41ba7338e2c9cfa692b4b.jpg
http://i2.运维网.com/images/blog/201811/13/9b6bf80ee608f0b65d9b5c00215e6a53.jpg
http://i2.运维网.com/images/blog/201811/13/c1b5c60d30076b8243f5025a182cfbde.jpg
http://i2.运维网.com/images/blog/201811/13/ec1594207f598978bf68ebedc642801b.jpg
添加kafka服务
有个红色警告
http://i2.运维网.com/images/blog/201811/13/9f84b8ca09b2304ec52986d6992f0689.jpg
此角色需要在可以启动前激活以下其他 Parcel：。
http://i2.运维网.com/images/blog/201811/13/fb47fb3be72584062ec435c6645e6d31.jpg
http://i2.运维网.com/images/blog/201811/13/48fea441e2d7715467c833f4e0d90aba.jpg
http://i2.运维网.com/images/blog/201811/13/4b92020ed8f0a6bb596f42cca8db2e6c.jpg
http://i2.运维网.com/images/blog/201811/13/07246e9f2db387c400a77ae31950182b.jpg
http://i2.运维网.com/images/blog/201811/13/a40a23d8e052b804ba790337c912afe8.jpg
http://i2.运维网.com/images/blog/201811/13/35cd39a58f2af2bb1614e7d96d0bee1c.jpg
http://i2.运维网.com/images/blog/201811/13/14d3a274bc5cef164855a655fd9a4bca.jpg
http://i2.运维网.com/images/blog/201811/13/0f435f389983321079374993aacf7f54.jpg
出现上面的返回
http://i2.运维网.com/images/blog/201811/13/c5342089f2e3d84e8fa6f427689cb1ce.jpg
http://i2.运维网.com/images/blog/201811/13/e167ba41a5c86a0f99a3dc87f49eff69.jpg
报错了不要紧，点到集群主页修改配置 不用点完成上面那个也行
http://i2.运维网.com/images/blog/201811/13/3f6693ef4ba62ac7f427f0f208306061.jpg
修改完成配置再启动完美解决 所有的记录安装就到这了有问题可以联系我一起来解决
http://i2.运维网.com/images/blog/201811/13/72aca4faa76465f7ceb4cbbb48a7a032.jpg
[*]更换kafka版本2
https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#pcm_kafka 这可以查看版本对应关系
http://i2.运维网.com/images/blog/201811/13/f9403b347adcf658ff8665073ad00ee2.jpg
访问cdh控制面板：
点击图标
http://i2.运维网.com/images/blog/201811/13/96e3d424264f28a2efbb5a16985c4521.jpg
再点击配置
http://i2.运维网.com/images/blog/201811/13/2602b3d9cc3db99daff4009d30a57a2f.jpg
远程parcel地址点击➕号选择
http://i2.运维网.com/images/blog/201811/13/a4a03d0de9e6dca3287d86d9ccf60367.jpg
https://archive.cloudera.com/kafka/parcels/2.0.0/ 加上这个版本的parcel地址 点击保存
http://i2.运维网.com/images/blog/201811/13/193371ea02e4af379b1d1b7fb785ef4c.jpg
在到集群里面点击kafka
http://i2.运维网.com/images/blog/201811/13/7e6b2f59d1709f32b96fcf7448de3c07.jpg
把服务停止。
在到礼物里面把相应的版本停止
http://i2.运维网.com/images/blog/201811/13/30be4c603f4a22dae66a8d386a5a8a2a.jpg
找到新的版本后点击下载 在点激活 ，
/opt/cloudera/parcels/ 多了一个KAFKA的软连接
http://i2.运维网.com/images/blog/201811/13/9601802f84e08dcb0cdd3de63374952f.jpg
http://i2.运维网.com/images/blog/201811/13/3feb39a77523e79e1b787097ead08aea.jpg
执行一个命令看看情况:
kafka-topics --zookeeper 10.10.32.21:2181 –describe ip为ZooKeeper
的地址 看Leader：值有没有-1的没有就正常有就有问题了。
http://i2.运维网.com/images/blog/201811/13/ea74aaaeb1b6d8a7ef2611112e67c4d4.jpg
页:
[1]