It seems all this could be just done with traditional networking tech, like microservice endpoints just having real IP addresses and using normal application level auth/load balancing methods when conversing with internal services.
https://docs.projectcalico.org/v3.1/introduction/
http://leebriggs.co.uk/blog/2017/02/18/kubernetes-networking...
In a Calico setup, every container (or VM - they use the term "workload" to abstract over the two) has an IP address, and talks to other containers in the usual way. AIUI, Calico does a couple of things to make that work at Cloud Scale (tm): it pushes firewall rules into the kernel, to make sure each container can only communicate with the other containers it's supposed to, and it propagates routes around, so each host knows where to send packets destined for containers on other hosts.
The routing bit is important because Calico is designed to run as a flat IP space on top of a non-flat ethernet space, ie one where the access switches are connected by routers rather than more switches. That's useful because scaling a flat ethernet network up to a huge size is apparently hard (network engineers start telling horror stories about Spanning Tree Protocol etc).
Calico still has more moving parts than i'm personally comfortable with, but it seems broadly sensible.
Different orgs work at different scales and in different styles; some orgs are producing monoliths, others are producing "fat services" or "microservices" (without going too much into what that might mean).
Some orgs have template repos or base libraries (big difference!) that they use to produce services. Others just have standards and you can do it how you like but plz conform to the standard (have /healthz, use statsd or export for prometheus, etc etc).
Also, how do all the things auth to each other? Do you TLS all the things or do you have api keys and secrets n-way between all the things? Does stuff trust each other based on IP? Etc.
There are lots of "illities", but particularly various kinds of monitoring, metrics, circuit breakers, access control and so on that you can either bake in to each service independently or implement via shared code somehow or other.
Notice that the above generally implies some degree of language homogenisation (usually a sane thing to have when you take into account other illities like artifact repos, dependency analysis, coding style guides, static analysis tooling etc - adopting a new language is not "easy" at scale) or else you are rewriting all these things a lot.
Anyway one option in this whole rainbow of possibilities is that you pull some of this out of the service itself and push it into a network layer wrapper somehow.
And that is how you end up with a service mesh...
Broadly speaking, my estimation is that if your company doesn't have multiple buildings with lots of people who have never met each other, you probably don't need a service mesh. And maybe not even then.
At this point the solutions seem to be intro a service mesh or copy pasta.
The issue with microservices at scale vis-a-vis microservices is linear scaling of routing tables, very short-lived IP connections, and large update times for massive service maps.
I think networking is complex in general, but the promise of this kind of integration (from my limited understanding), would be a clear reflection of services at the routing level instead of a hodgepodge of ports and IP addresses that provide little context or meaning. Having lost a few half-days learning iptable nuances in our Kubernetes cluster I can see the benefits in having an integrated stack with less impedance between architecture and routing.
Generally we don't eliminate complexity, we just move it from one place to another. I imagine this is a case where a more complex service implementation could provide a simpler user experience.
From an ops point of view, this creates a nearly opaque wall you get to run up against when troubleshooting issues; a completely unnecessary wall.