The Linux-HA heartbeat code is used for building highly available failover clusters. It can do 2 node IP takeover for an unlimited number of IP interfaces. It works by sending a "heartbeat" between 2 machines either over a serial cable, ethernet, or both. If the heartbeat fails, the secondary machine will assume the primary machine has failed, and take over services that were running on a primary machine. For more information on the heartbeat software, you can visit linux-ha.org.
drbd comes with 2 scripts that make it very easy to integrate with the heartbeat package. The first script is drbd, and it installs to the /etc/rc.d/init.d/drbd. It is intended to be used to start up drbd services upon boot. The second script is datadisk, and it installs Into /etc/ha.d/resource.d/datadisk. The datadisk script handles switching a drbd device from secondary to primary state, and it is called from the /etc/ha.d/haresources file.
This example will build upon the previous example from the drbd.conf section. In that example, we have 2 machines named thost1 (10.1.1.31) and thost2 (10.1.1.32). In this example, we will st up a web server which has it's html pages stored on a shared drbd device. Eventually, you'll want all of these steps to happen automatically, but for this HOWTO I'll just go over the steps to do things manually.
The first step is to correctly configure the heartbeat package. I'm not going to go into setting up the heartbeat package, I assume if you have gotten this far, you have heartbeat already running. Assuming heartbeat is running, and your drbd.conf file is correctly configured, the first thing you need to do is start up drbd on both nodes. On thost1, issue the following commands:
$ insmod drbd $ /etc/rc.d/init.d/drbd start
Do the same thing on thost2:
$ insmod drbd $ /etc/rc.d/init.d/drbd start
At this point drbd should be running. Make sure drbd is the primary interface on thost1, and then go ahead and mount the disk. If you haven't done so already, create a file system on the device. Lastly, configure the web server on both machines to point to the location you will mount the drbd device to become document root for your web server.
[root@10-0-1-31 ha.d]# cat /proc/drbd version: 0.6.9 (api:64/proto:62) 0: cs:Connected st:Secondary/Secondary ns:0 nr:0 dw:0 dr:0 pe:0 ua:0 1: cs:SyncingAll st:Secondary/Primary ns:0 nr:33168084 dw:33168084 dr:0 pe:0 ua:998 [>...................] sync'ed: 0.4% (49817/50006)M finish: 1:28:27 speed: 9,680 (19,305) K/sec ^cur ^avrg since device start
Assuming the /etc/ha.d/ha.cf file correctly identifies both nodes in your cluster, the next step is to edit the /etc/ha.d/haresources file. Add the following line to that file
10-0-1-31.linux-ha.org 10.0.10.10/16 datadisk::drbd0 httpd
The 10.0.10.10 address is the IP address attached to the web server. See the heartbeat documentation for more info on that. Basically, that line tells the heartbeat software to run the /etc/ha.d/resource.d/datadisk script with a paramater of drbd0. In the event of a failure, the datadisk script will run on the secondary node, and switch the drbd0 drbd device from seconday to primary state.
You also need to edit the /etc/fstab file on both machines. It is important to configure this device to NOT mount upon boot. The noauto flag will take care of that.
/dev/nb0 /mnt/disk ext2 noauto 0 0
At this point, you can start up the heartbeat software.
/etc/rc.d/init.d/heartbeat start
If everything is configured correctly, the heartbeat software will launch the web server and tie it to the virtual interface. At this point, you can power off thost1. If all goes well, in about 30 seconds, thost2 will take over and launch the web server. If that test works, go ahead and setup drbd and heartbeat to start upon boot, and sit back and enjoy the high availability!
This short example will demonstrate some of the things you need to take into account when failing over services which require state information to be kept. We will set up an NFS server which exports a filesystem stored on a shared drbd device. Again, we'll just consider the steps to do things manually
We're going to assume that you've set up the heartbeat package just as above, and just go over the changes required for setting up a failover NFS service instead of the web service. So, just as before, on thost1 issue the following commands:
$ insmod drbd $ /etc/rc.d/init.d/drbd start
Do the same thing on thost2:
$ insmod drbd $ /etc/rc.d/init.d/drbd start
At this point drbd should be running. Make sure drbd is the primary interface on thost1. We'll assume that you will be mounting the drbd device at /mnt/disk. At least on Red Hat and SuSE based systems, the state information for the NFS server is in /var/lib/nfs, and we will want this state information to be found on the shared device. So, on thost1, with the NFS server stopped, do:
$ mkdir /mnt/disk $ mount /proc/nb0 /mnt/disk $ mkdir /mnt/disk/var; mkdir /mnt/disk/var/lib $ mv /var/lib/nfs /mnt/disk/var/lib $ ln -s /mnt/disk/var/lib/nfs /var/lib/nfs
On thost2, ensure the NFS server is not running, and execute the following:
$ mkdir /mnt/disk $ rm -r /var/lib/nfs $ ln -s /mnt/disk/var/lib/nfs /var/lib/nfs
The last thing is to set up the part of the filesystem we will actually export over the NFS mount. This will be the hierarchy below /mnt/disk/export. Assume that the NFS client has IP address 10.0.20.1. So, on thost1:
$ mkdir /mnt/disk/export $ echo "/mnt/disk/export 10.0.20.1(rw)" >> /etc/exports
On thost2:
$ echo "/mnt/disk/export 10.0.20.1(rw)" >> /etc/exports
Assuming the /etc/ha.d/ha.cf file correctly identifies both nodes in your cluster, the next step is to edit the /etc/ha.d/haresources file. Add the following line to that file on BOTH machines
10-0-1-31.linux-ha.org 10.0.10.10/16 datadisk::drbd0 nfsserver
You need to make sure that the nfsserver script can be found by the heartbeat package; see the heartbeat documentation for more details.
The 10.0.10.10 address is the IP address attached to the NFS server. See the heartbeat documentation for more info on that. Basically, that line tells the heartbeat software to run the /etc/ha.d/resource.d/datadisk script with a paramater of drbd0. In the event of a failure, the datadisk script will run on the secondary node, and switch the drbd0 drbd device from seconday to primary state.
You also need to edit the /etc/fstab file on both machines. It is important to configure this device to NOT mount upon boot. The noauto flag will take care of that.
/dev/nb0 /mnt/disk ext2 noauto 0 0
At this point, you can start up the heartbeat software.
/etc/rc.d/init.d/heartbeat start
If everything is configured correctly, the heartbeat software will launch the NFS server and tie it to the virtual interface. At this point, you can power off thost1. If all goes well, in about 30 seconds, thost2 will take over and launch the NFS server. Test it by mounting the NFS mount from 10.0.20.1 and performing some file operation which will take more than 30 seconds; it should pause and then magically start going again, unconscious of the fact it's just swapped servers. Enjoy your high availability!