Setup Multi Hbase master on Hadoop Cluster

hcwzwx

　　http://2hei.net/setup-multi-hbase-master-on-hadoop-cluster.html
　　Setup Multi Hbase master on Hadoop Cluster to avoid single point failure. When active master failed/down for some reason exceed timeout we expected, backup master will be active and take over the role of master, see the value of zookeeper.session.timeout.

　　$cat /usr/lib/hbase/conf/hbase-site.xml
<name>zookeeper.session.timeout</name>
<value>180000</value>
<description>ZooKeeper session timeout. HBase passes this to the zk quorum as suggested maximum time for a session.In milliseconds.</description>
</property>

• Test scenarios:
  

　　hadoop01.cluster.2hei.net (Hbase master)
hadoop02.cluster.2hei.net (Hbase backup master)

• Configuration Steps:
  

　　1. add backup master hostname to /usr/lib/hbase/conf/backup-masters

　　$cat /usr/lib/hbase/conf/backup-masters
hadoop02.cluster.2hei.net

　　2. start hbase (hbase not manage zookeeper):

　　#start zookeeper
/usr/lib/zookeeper/bin/zkServer.sh start
　　#start master
sh /usr/lib/hbase/bin/hbase-daemon.sh start master
　　#start backup master
sh /usr/lib/hbase/bin/hbase-daemon.sh start master –backup

• Test steps:
  

　　1. stop hbase master

　　sh /usr/lib/hbase/bin/hbase-daemon.sh stop master

　　2. check hbase master(hadoop01) logs:

　　2012-07-23 08:44:53,058 INFO org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for -ROOT-,,0.70236052 so generated a random one; hri=-ROOT-,,0.70236052, src=, dest=hadoop07.cluster.2hei.net,60020,1338437695217; 5 (online=5, exclude=null) available servers
2012-07-23 08:44:53,058 INFO org.apache.hadoop.hbase.master.AssignmentManager: Assigning region -ROOT-,,0.70236052 to hadoop07.cluster.2hei.net,60020,1338437695217
2012-07-23 08:44:53,058 DEBUG org.apache.hadoop.hbase.master.ServerManager: New connection to hadoop07.cluster.2hei.net,60020,1338437695217
— stop master —
Mon July 23 08:45:47 UTC 2012 Killing master

　　3. check hbase backup master(hadoop02) logs:

　　2012-07-23 04:14:56,348 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2012-07-23 04:14:56,356 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster started in backup mode. Stalling until master znode is written.
2012-07-23 04:14:56,389 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Another master is the active master, 192.168.0.100:60000; waiting to become the next active master
2012-07-23 05:34:15,361 WARN org.apache.hadoop.ipc.HBaseServer: Incorrect header or version mismatch from 192.168.0.201:44928 got version 47 expected version 3
— stop master —
2012-07-23 08:47:58,003 DEBUG org.apache.hadoop.hbase.master.LogCleaner: Add log cleaner in chain: org.apache.hadoop.hbase.master.TimeToLiveLogCleaner
2012-07-23 08:47:58,257 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-07-23 08:47:58,334 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-07-23 08:47:58,344 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 60010
2012-07-23 08:47:58,344 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 60010 webServer.getConnectors()[0].getLocalPort() returned 60010
2012-07-23 08:47:58,344 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 60010
2012-07-23 08:47:58,344 INFO org.mortbay.log: jetty-6.1.26
2012-07-23 08:47:58,879 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:60010

　　We can see hadoop02 is listening 60010 and to be the active hbase master now.
reference:  http://wiki.apache.org/hadoop/Hbase/MultipleMasters