BTW, since the blog post doesn't make it explicit: we're down to ~40 microseconds round trip between VMs in the same zone. No placement groups or infiniband required :).
Disclosure: I work on Google Cloud.
Disclosure: I work on Google Cloud.
I do get impressive bandwidth (1.95 Gbits/sec) using iperf.
Regarding in-guest iptables -- there's not much we can do on GKE/Kubernetes about that. I serve about 1.2M requests per second on my GKE cluster through GLB and see this overhead very well.
FWIW I also look forward to IPVS in k8s 1.9 that should improve this slightly.
That reminds me though: you have to be communicating over the internal IP addresses, not the external ones (I hope the Services stuff does the right thing automatically). Firewall rules and such are different for internal versus external IPs (people often want a simple "deny all external traffic"), so that's sadly still a performance cliff.