Modern network load balancing and proxying
blog.envoyproxy.io
blog.envoyproxy.io
This means that you can have your proxy "hold requests" for a few seconds while your system restarts in the background & slowly let off the requests out to the backend in "one more request per second" scale up.
This sort of stuff would work great with a proxy that actually understands something a kubernetes scheduler & know when a node is failing/restarting etc.
tl;dr -- ECMP means Equal Cost Multi-Path
* non-equal-cost loadbalancing (give the Skylake boxes 25% more traffic than the Broadwell boxes)
* ECMP doesn't know anything about service health, so what happens when one app server out of 10 gets wedged and stops responding to requests?
* TLS termination on the proxy means you limit what machines need to hold your private keys.
* what if you want to load-balance or direct traffic based on something other than a 5-tuple hash?
* ECMP doesn't scale very well. Not all that long ago, at $DAYJOB, we had scalability problems because a certain large network vendor couldn't do more than 8 next hops for a single prefix.
Envoy is an L7 proxy, so it's the moral equivalent of the Google GFE (which, conveniently enough, is the next layer of load balancing behind Maglev for most Google services).
ECMP still has some scaling challenges, IMHO, since each destination host still needs to peer with the upstream switch over BGP (unless you do some cleverness).