Containers, microservices, and service meshes
jpetazzo.github.io
jpetazzo.github.io
https://docs.google.com/spreadsheets/d/1OBaKrwR030G39i0n_47i...
From my own experience I've had some great success with LinkerD in the past on Mesos DC/OS.
Since moving companies, and switching to Kubernetes, we've yet to deploy any service mesh into production.
The blog from Jerome highlights many of the benefits already.
From my perspective the bigs ones in the past were:
- Tracing (with Zipkin)
- Retries which removed or fixed dodgy app reconnect logic
- Time series metrics in LinkerDViz showing realtime rates and latency and errors between services
Istio gets all of the cool press attention and blogs written about it. Yet, you also read a lot of warnings about it always being 6 months away from being really robust. Even at version 1 we read some horror stories about obscure bugs showing up in fairly standard use cases.
Connectivity between services is too scary to gamble on. It's a similar deal with CNI (we're still on Calico despite arguably cooler stuff being out there) and Ingresses (still on ingress-nginx).
AWS have a service mesh that is probably going to be the one we trial next at work.
Improved observability and retries would definitely be of benefit on our current platform. Another driving factor is also our security team wanting mutual TLS between services.
Message Brokers are the airplanes. Even if you can't understand how they work you will see that many companies use them who really rely on distributed systems working, eg telephone companies, banks, trading companies.
Service Meshes try to solve a problem that K8s was meant to solve in the first place. So K8s is the boat and Service Mesh is the attached helicopter. By itself it might be a good idea to use it, but the way things are taped together right now it's just an anti-pattern.
If you don't have a pointy-haired boss forcing you to use it, then it's probably better to avoid the whole thing.
I'd rather see how far the development around k8s-less pods with podman will go and take care of the distributed architecture of my systems myself.
At some point folks need to realize they arent google and they really dont need to abstract the development layer because they dont have a team of google SRE to manage what their pods / containers / mesh runs on
Message brokers are... not that.
These are apples and elephants you're comparing.
The reason to use a tool is to make a task simpler or take a task out of your hands in most cases but the rare edge case. For instance grep makes it easier to go to a file to find a certain line.
If you need to know networking, storage, microservice architecture yourself you don't need any tools.
In k8s you need to know all this, and on top you need to know k8s to achieve the same thing. And then you either need to write k8s plugins or also additionally know all the k8s plugins and wether any of them will actually solve the problem you have.
I don't think it is service mesh OR brokers. Both cater to different use cases.
https://activemq.apache.org/how-should-i-implement-request-r...
https://programmingisterrible.com/post/162346490883/how-do-y...
"A protocol is the rules and expectations of participants in a system, and how they are beholden to each other. A protocol defines who takes responsibility for failure.
The problem with message brokers, and queues, is that no-one does.
Using a message broker is not the end of the world, nor a sign of poor engineering. Using a message broker is a tradeoff. Use them freely knowing they work well on the edges of your system as buffers. Use them wisely knowing that the buck has to stop somewhere else. Use them cheekily to get something working.
I say don’t rely on a message broker, but I can’t point to easy off-the-shelf answers. HTTP and DNS are remarkable protocols, but I still have no good answers for service discovery."
I would not disagree with any of this but I don't know enough about service meshes to know if they mean that you don't need a protocol (meaning the edges still need to deal with failures after the mesh or broker has retried etc).
One thing a broker gives you is that if the problem is that the other service is busy or down then it will eventually get the message. So it avoids one specific kind of failure in the case where eventual consistency is acceptable.
The protocol has less to do with it, see grpc for example.
Having run an operationally critical microservice infrastructure with consul for service discovery I realized that a mesh makes sense.
When you start hitting a few hundreds of services you’ll want:
1. The observability. Scenario: Why is the application slow? The application depends on two different services each with individual sets of downstream dependencies.
2. Making the services discovery process more resilient. An outage of consul without other means of discovery sucks!
Message brokering as well as event sourcing seems to this layman like awesome patterns where eventual consistency is perfectly ok.
In most scenarios it probably is, but for some it is not.
Whatever works!
Service discovery is also inherent in the routing rules itself when you use something like AMQP for example.
Observability might be something that the broker system lacks of I agree.
But you might have scenarios where you want or require synchronous or continous direct service to service communication is what I meant.
Nick, long time no see. Hope thing are well. Amazing who you find browsing these threads :)
Introducing these things prematurely are just asking for trouble. They come with their share of complexity and overhead. However they do solve some problems. In a lot of cases, if you don't know about these things yet, you probably don't need to.
At its core, a service mesh is just a complicated, multi-tier, higher-level router and VPN. For TCP/IP routing there's lots of protocols, from the local level to LAN to WAN and beyond. Many of them are designed for specific use cases. But even so, traditional routers aren't centralized, they don't care what other routers they connect to, and they all speak common languages, transfer information, make independent decisions, and act in isolation. This is a really good design that I think service mesh should utilize.
I think what we really want from a service mesh is a router for applications with a standard protocol that doesn't share state (or at least, state is asynchronous and non-consistent, and only verified at the end of an operation, similar to IP routing). Additional features are needed, and those can be codified into the protocol. That would make the system more resilient to change and compatible with different products, rather than now where each service mesh software is basically a unique incompatible router and protocol.
Istio is used in Cloud Foundry without needing a Kubernetes master.
Those five teams iterate over there service and product. And maybe even the outside service iterates there product.
How do you make sure that everyone is on the same page and is using the right version and that the right service talks to the rights service in the right way? Especially the outside service for example?