I'm excited for all the buzz going on around service discovery, it's often one of the most crucial design decisions for a SOA architecture. I'm really bummed, however, that so many of them rely on clients doing DNS lookups. I vastly prefer a system like airbnb/Yelp SmartStack, consul template, or really any system that reflects discovery state in real load balancers such as HAProxy, nginx, or even those new fangled service load balancers like vulcand.

Honestly just don't do DNS. Why?

1. Performance

Service discovery should not happen during the life of a HTTP request. The number of times that I've had to debug performance problem that ultimately ended up being DNS requests taking like 100ms (which for most internal applications is too damn long) is staggering to me. Yes you can mitigate this, but it is not easy because even local DNS has to miss the cache sometimes.

2. DNS propagation sucks.

This article talks about one side of the DNS propagation problem ("it's down" -> removed from DNS), but they seem to completely ignore the other side of the DNS propagation problem (removed from dns -> clients stop talking to it). At any kind of reasonable engineering scale one of your libraries (watch out for ones that cache DNS "for performance"), programming languages (java), or infra tools (nginx/haproxy) are going to not re-resolve DNS like you expect them to.

3. More Performance

Client side load balancing is legitimately hard, and if you're just doing round robin over DNS records you are likely screwing your high percentiles. HAProxy/nginx/vulcand/etc all support better load balancing strategies like least connection load balancing that really make a difference here. This can be the difference between a 500ms 99% and a 50ms 99%.