Ask HN: How do I failover the load balancer
I'm trying to make sure the servers for my product are available as much as possible, one of the problems I couldn't figure out how to solve was how to fail-over the load balancer. How do you do it?
Let's say you have two servers, primary and secondary -- primary is the one that's currently acting as your load balancer. When both servers are running Heartbeat, they exchange UDP pings every second. If the primary server stops responding, then the Heartbeat process on the backup server will automatically take over the shared IP. When the primary box returns, it won't cause any problems or steal back the IP address, until you explicitly instruct it to do so. (If you really wanted it to steal back the IP automatically, I'm sure you could configure it that way.) You can also initiate a failover with a single command, which is useful when you need to do maintenance on the primary box. Running "/usr/share/heartbeat/hb_takeover all" is a lot simpler than having to manually reassign the IP address.
My company's infrastructure is very dependent on heartbeat -- we use it to support our load balancers and all our back-end services. You need to be careful or it will do "surprising" things when you're rebooting or changing the network config, but on the whole it makes it cheaper and easier to provide highly-available services.
Of course, if you're using a hardware load balancer, it should have a similar failover protocol built-in. I'd recommend just using its built-in features, that's what you're paying the big bucks for.
The load balancers share one public IP address. Only one server at a time is assigned that address. To failover: 1. Physically isolate the failed load balancer by removing power and/or disabling its Ethernet port on the switch. 2. Assign the shared public IP address to the new active load balancer. 3. Broadcast an unsolicited ARP reply from the new active load balancer.