Your observation about clients requiring more and more logic (e.g. discovery, reconnecting, timeout, etc) is spot on. It is a pretty big overhead and teams that are building an architecture based on micro-services should be prepared to pay that cost.

I am not sure why people choose such an architecture, but for me the biggest reason to do this is to be able to run it on platforms like Mesos, Kubernetes, Nomad etc. Once you've aligned your system with such a platform you get plenty of benefits(improved workflow, reliability, fault tolerance, cloud mobility). But the price is pretty big.

The good part is that good tooling is becoming more widespread and open source projects like this make it easier for people to start up.