Installing and Configuring Openfiler with DRBD and Heartbeat(HA)--2

shuaiwei588

Configure LVM Partition
　　Create the /dev/drbd1 as a PV (Physical Volume) for the data volume group, which will be used to create Logical Volumes for data.
　　First, edit /etc/lvm/lvm.conf and modify the filter line:
　　From:

filter = [ "a/.*/" ]
　　To:

filter = [ "r|/dev/sda5|" ]
　　Note: Change /dev/sda5 to reflect the partition of your LVM. Also remember to apply these changes on both filer01 and filer02.
　　Create the LVM Physical Volume (only do this on our Primary node, as it will replicate to the Second node via drbd):
root@filer1 /# pvcreate /dev/drbd1
Physical volume "/dev/drbd1" successfully created
　　

Configure Heartbeat
　　As mentioned before, Heartbeat controls failover between hosts. The two nodes run the Heartbeat service, that sends out a heartbeat pulse on the secondary interface (eth1). If one node dies, then Heartbeat detects this and roles the surviving node to Primary
(if it wasn't already) using startup scripts available in /etc/ha.d/resources.d.
　　Make modifications to /etc/ha.d/ha.cf and
/etc/ha.d/authkeys. Make these changes on both nodes.
　　In /etc/ha.d/authkeys, add:

auth 2
2 crc
　　The /etc/ha.d/authkeys file may not exist and will need to be created, as it does not appear to exist in Openfiler 2.3.
　　Next, restrict permissions to authkeys to just "root":
root@filer01 ~# chmod 600 /etc/ha.d/authkeys

root@filer02 ~# chmod 600 /etc/ha.d/authkeys
　　Create a /etc/ha.d/ha.cf on both nodes (needs to be identical on both, just like
/etc/drbd.conf):

debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
bcast eth1
keepalive 5
warntime 10
deadtime 120
initdead 120
udpport 694
auto_failback off
node filer01
node filer02
　　Enable Heartbeat to startup at boot:
root@filer01 ~# chkconfig --level 2345 heartbeat on

root@filer02 ~# chkconfig --level 2345 heartbeat on
　　

Openfiler Data Configuration
　　As mentioned above, a 512 MB partition was created to keep the configuration and HA services available during a failover. To get this working, copy the services and Openfiler configuration data over to the new partition - symbolically linking it back to
it's original location.
　　filer01:
root@filer01 ~# mkdir /cluster_metadata

root@filer01 ~# mount /dev/drbd0 /cluster_metadata

root@filer01 ~# mv /opt/openfiler/ /opt/openfiler.local

root@filer01 ~# mkdir /cluster_metadata/opt

root@filer01 ~# cp -a /opt/openfiler.local /cluster_metadata/opt/openfiler

root@filer01 ~# ln -s /cluster_metadata/opt/openfiler /opt/openfiler

root@filer01 ~# rm /cluster_metadata/opt/openfiler/sbin/openfiler

root@filer01 ~# ln -s /usr/sbin/httpd /cluster_metadata/opt/openfiler/sbin/openfiler

root@filer01 ~# rm /cluster_metadata/opt/openfiler/etc/rsync.xml

root@filer01 ~# ln -s /opt/openfiler.local/etc/rsync.xml /cluster_metadata/opt/openfiler/etc/
　　Then edit our /opt/openfiler.local/etc/rsync.xml file:

<?xml version=&quot;1.0&quot; ?>
<rsync>
<remote hostname=&quot;10.188.188.2&quot;/> ## IP address of peer node.
<item path=&quot;/etc/ha.d/haresources&quot;/>
<item path=&quot;/etc/ha.d/ha.cf&quot;/>
<item path=&quot;/etc/ldap.conf&quot;/>
<item path=&quot;/etc/openldap/ldap.conf&quot;/>
<item path=&quot;/etc/ldap.secret&quot;/>
<item path=&quot;/etc/nsswitch.conf&quot;/>
<item path=&quot;/etc/krb5.conf&quot;/>
</rsync>
root@filer01 ~# mkdir -p /cluster_metadata/etc/httpd/conf.d
　　filer02:
root@filer2 ~# mkdir /cluster_metadata

root@filer2 ~# mv /opt/openfiler/ /opt/openfiler.local

root@filer2 ~# ln -s /cluster_metadata/opt/openfiler /opt/openfiler
　　Change the /opt/openfiler.local/etc/rsync.xml to reflect below:

<?xml version=&quot;1.0&quot; ?>
<rsync>
<remote hostname=&quot;10.188.1881&quot;/> ## IP address of peer node.
<item path=&quot;/etc/ha.d/haresources&quot;/>
<item path=&quot;/etc/ha.d/ha.cf&quot;/>
<item path=&quot;/etc/ldap.conf&quot;/>
<item path=&quot;/etc/openldap/ldap.conf&quot;/>
<item path=&quot;/etc/ldap.secret&quot;/>
<item path=&quot;/etc/nsswitch.conf&quot;/>
<item path=&quot;/etc/krb5.conf&quot;/>
</rsync>
　　

Heartbeat Cluster Configuration
　　Then modify the /cluster_metadata/opt/openfiler/etc/cluster.xml config file. This config file generates the
/etc/ha.d/haresources file, which tells Heartbeat what it should do in a failover.
　　filer01 Only:

<?xml version=&quot;1.0&quot; ?>
<cluster>
<clustering state=&quot;on&quot; />
<nodename value=&quot;filer01&quot; />
<resource value=&quot;MailTo::it@company.com::ClusterFailover&quot;/>
<resource value=&quot;IPaddr::192.168.1.17/24&quot; />
<resource value=&quot;drbddisk::&quot;>
<resource value=&quot;LVM::vg0drbd&quot;>
<resource value=&quot;Filesystem::/dev/drbd0::/cluster_metadata::ext3::defaults,noatime&quot;>
<resource value=&quot;MakeMounts&quot;/>
</cluster>
　　Note how the HA IP address is declared here (192.168.1.17). As mentioned before, Heartbeat controls the setup of the network interface, the mounting of the LVM volume group, and the mounting of drbd0 (/cluster_metadata).
　　

Samba and NFS Support
　　Modify Samba and NFS so it's available on our /cluster_metadata drbd resource.
　　filer01:
root@filer01 ~# mkdir /cluster_metadata/etc

root@filer01 ~# mv /etc/samba/ /cluster_metadata/etc/

root@filer01 ~# ln -s /cluster_metadata/etc/samba/ /etc/samba

root@filer01 ~# mkdir -p /cluster_metadata/var/spool

root@filer01 ~# mv /var/spool/samba/ /cluster_metadata/var/spool/

root@filer01 ~# ln -s /cluster_metadata/var/spool/samba/ /var/spool/samba

root@filer01 ~# mkdir -p /cluster_metadata/var/lib

root@filer01 ~# mv /var/lib/nfs/ /cluster_metadata/var/lib/

root@filer01 ~# ln -s /cluster_metadata/var/lib/nfs/ /var/lib/nfs

root@filer01 ~# mv /etc/exports /cluster_metadata/etc/

root@filer01 ~# ln -s /cluster_metadata/etc/exports /etc/exports
　　Note: This moves /var/spool/samba into
/cluster_metadata, which is only a 512 MB partition. So, if large print jobs are put through Samba, the free space on this volume will get eaten up pretty quickly. So, if this is the case, a separate DRBD resource should be created for the
/var directory. Or, reconsider hosting print services on a different server.
　　filer02:
root@filer02 ~# rm -rf /etc/samba/

root@filer02 ~# ln -s /cluster_metadata/etc/samba/ /etc/samba

root@filer02 ~# rm -rf /var/spool/samba/

root@filer02 ~# ln -s /cluster_metadata/var/spool/samba/ /var/spool/samba

root@filer02 ~# rm -rf /var/lib/nfs/

root@filer02 ~# ln -s /cluster_metadata/var/lib/nfs/ /var/lib/nfs

root@filer02 ~# rm -rf /etc/exports

root@filer02 ~# ln -s /cluster_metadata/etc/exports /etc/exports
　　

iSCSI Support
　　filer01:
root@filer01 ~# mv /etc/ietd.conf /cluster_metadata/etc/

root@filer01 ~# ln -s /cluster_metadata/etc/ietd.conf /etc/ietd.conf

root@filer01 ~# mv /etc/initiators.allow /cluster_metadata/etc/

root@filer01 ~# ln -s /cluster_metadata/etc/initiators.allow /etc/initiators.allow

root@filer01 ~# mv /etc/initiators.deny /cluster_metadata/etc/

root@filer01 ~# ln -s /cluster_metadata/etc/initiators.deny /etc/initiators.deny
　　filer02:
root@filer02 ~# rm /etc/ietd.conf

root@filer02 ~# ln -s /cluster_metadata/etc/ietd.conf /etc/ietd.conf

root@filer02 ~# rm /etc/initiators.allow

root@filer02 ~# ln -s /cluster_metadata/etc/initiators.allow /etc/initiators.allow

root@filer02 ~# rm /etc/initiators.deny

root@filer02 ~# ln -s /cluster_metadata/etc/initiators.deny /etc/initiators.deny
　　

FTP Support
　　filer01:
root@filer01 ~# mv /etc/proftpd /cluster_metadata/etc/

root@filer01 ~# ln -s /cluster_metadata/etc/proftpd/ /etc/proftpd
　　filer02:
root@filer02 ~# rm -rf /etc/proftpd

root@filer02 ~# ln -s /cluster_metadata/etc/proftpd/ /etc/proftpd
　　

Configure Volume Group
　　filer01:
　　Create a Volume group from /dev/drbd1:
root@filer1 etc# vgcreate vg0drbd /dev/drbd1
Volume group &quot;vg0drbd&quot; successfully created
　　Note: If planning on using Windows to connect to these iSCSI targets, do not use the &quot;_&quot; character or any other special characters when creating the volume group.
　　Once the Heartbeat service has been configured and started (see below), the Openfiler web administration GUI should be available on
https://192.168.1.17:446. Once there, LVM volumes can be created and to exported via iSCSI etc.
　　

Starting Heartbeat and First-Time Configuration
　　In order to get Openfiler to write the /etc/ha.d/haresources file based on the cluster.xml config file, restart the Openfiler service and then log onto the web interface (using the Primary node's direct IP) and click on Services and enable iSCSI.
　　Make sure to do this on the Primary node (filer01).
root@filer01 ~# rm /opt/openfiler/etc/httpd/modules

root@filer01 ~# ln -s /usr/lib64/httpd/modules /opt/openfiler/etc/httpd/modules
　　Note: If you use a 32-bit system, just take out the “64”.
root@filer01 ~# service openfiler restart
　　With any luck, Openfiler has written this file out to /etc/ha.d/haresources. If haresources was created, copy it over to filer02.
　　Note: Before starting Heartbeat, a volume must be created:
root@filer01 ~# lvcreate -L 400M -n filer vg0drbd
　　It appears if you log onto the web interface and activate a service such as NFS or iSCSI. this will force Openfiler to rewrite the
/etc/ha.d/haresources file. Copy (via scp) this file over to the second node as follows:
root@filer01 ~# scp /etc/ha.d/haresources root@filer02:/etc/ha.d/haresources
　　Since heartbeat was added to the start up scripts earlier, reboot filer01, then reboot filer02.
　　If all goes well, access the primary node via a web browser on the High Available ip address:
https://192.168.1.17:446.
　　If the web server is not accessible, a good place to look for errors is in /var/log/ha-log or /var/log/ha-debug files.
　　Note: The rsync configuration is meant to synchronize changes between the two nodes, but this will not happen unless using the most recent build. Therefore, see the fix at the following page:
　　https://forums.openfiler.com/viewtopic.php?id=2380
　　Once Openfiler is up and running, delete the the filer volume created earlier and then create new volumes (be sure to create the new volumes before stopping the Heartbeat service, or it will not start).