Maybe I’m wrong but it almost feels like busy work for DevOps. Is my first impression wrong? Is this the right way to architect systems in some use cases, and if so what are they?
As a company grows sooner or later most of these features become pretty desirable from an operations perspective. Feature developers likely don't and shouldn't need to care. It probably starts with things like Auth and basic load balancing. As the company grows to dozens of teams and services then you'll start feeling pain around service discovery and wish you didn't need to implement yet another custom auth scheme to integrate with another department's service.
After a few retry storm outages people will start paying more attention to load shedding, autoscaling, circuit breakers, rate limiting.
More mature companies or ones with compliance obligations start thinking about zero-trust, TLS everywhere, auditing, and centralized telemetry.
Is there complexity? Absolutely. Is it worth it? That depends where your company is in its lifecycle. Sometimes yes, other times you're probably better off just building things and living with the fact that your load shedding strategy is "just tip over".
Even aside from a lot of the more hyped up features of service mesh, the biggest thing Istio solves is tls everywhere and cloud agnostic workload identity. All of our pods get new tls certs every 24 hours and nobody needs an API key to call anything.
Our security team is thrilled that applications running with an Istio sidecar literally have way to leak credentials. There’s no API keys to accidentally log. Once we have databases setup to support mTLS authentication, we won’t need database passwords anymore.
Just as it’s foolish to create dozens of services if you have a 10-person team, you don’t really get much out of a service mesh if you only have a handful of services and not feeling the pain with your traditional tooling.
But once you get to large scale with convoluted business logic that is hard to reason about because so many teams are involved, the search for scalable abstractions begin. Service mesh then becomes useful because it is completely orthogonal to biz logic and you can now add engineers 100% focused on tooling and operations, and product engineers can think a lot less about certain classes of reliability and security concerns.
Of course in todays era of resume driven development, and the huge comp paid by FAANGs, you are going to get a ton of young devs pushing for service mesh way before it makes sense. I can’t say I blame them, but keep your wits about you!
When it was just mongo installs are easy since they only needed a mongo desktop client
So, in my opinion, the questions are introspective:
- “Do I have enough context to know what problem those solutions are solving, and to at least appreciate the problem space to understand why someone may solve it like this?”
- “Do I have or perceive those problem to impact my infrastructure/applications?”
- “Does the solution offered by the use cases described appeal to me?”
If yes at the end, then one potential implementation is a service mesh.
A lot of these are solved out-of-the-box with Hashicorp’s Nomad/Consul/Vault pairing, for example!
Adding a service mesh onto Kubernetes seems to bring a lot of complexity for a few benefits (80% of the effort for the last 20% sort of deal).
I think the benefits are magnified in larger organizations or where operators and devs are not the same people. And the complexity is relative to which solution you pick. If you're already on Kubernetes, linkerd2 is relatively easy to install and manage; is that worth it? To me it has been in the past.
I am wondering does Nomad/Consul continue to scale after some level?
I'm not a big fan of this approach; the two teams need to have a meeting and need to have a shared goal to implement the business's selected security requirements together. But sometimes fixing the org is too hard, so there is a Plan B.
What's missing are a bunch of things you probably want to check before issuing keys; was the release approved, was all the code reviewed before release, is the code reading the foo-service key actually foo-service? That involves some input from your orchestration layer; i.e. an admission controller that checks all these things against your policies, and only then injects a key that the application can read. (Picking up rotated keys becomes more difficult, but this might be a good thing. "If you don't re-deploy your code for 90 days, it stops being able to talk to other services" doesn't seem like the worst policy I can think of in a world where Dependabot opens up 8 PRs a day against your project.)
This all has the downside that it doesn't really prevent untrusted applications from ruining the security; a dump_keys endpoint that prints the secret key to a log, nefarious code checked into source control but approved (perhaps due to a compromised developer workstation), etc. Fixing those problems is well outside the scope of a service mesh, but something you have to have a plan for. CircleCI didn't! Now you read 3 blog posts a day about how they got hacked.
Anyway, not sure where I was going with this, but application teams need to consider their threat model and protect against it. Security isn't a checkbox that can be checked by someone that didn't write the code. Sure, you can get all sorts of certifications this way that look nice on your marketing page, but the certifications really only cover "did they do the bare minimum to look kind of competent if it was 10 years ago". If you have sophisticated adversaries, you're going to need a sophisticated security team.
We want to automate everything not related to the code we want to write. Service meshes sound like a good way to do that.
You could use the service mesh thing from AWS, along with cognito jwts, for authenticatetion and authorization
To me, it's a fairly big decision to layer something that's complex in it's own right on top of something else that's also complex.
I'm not so sure. I suspect it'll follow the same roadmap as Gateway API, which it already kind of is with the Service Mesh Interface (https://smi-spec.io/)
Understanding these few CRDs give great insights on what do expect from a Service mesh and how thinks are typically articulated.