Mizu – API traffic viewer for Kubernetes
github.com
github.com
* log of all traffic
* service graph
* automatic swagger doc generation
THANK YOU!!
Maybe it's a question of UX, more likely it's a question of user error (and therefore partly UX). In any case, I can't wait for it to get better!
I don't know what went wrong but the ecosystem prefers software presented as a black box and shrouded with mystery.
If you contrast this with Envoy, it has stellar documentation -- Envoy wants you to know how it works!
Envoy is used and abused by Istio (k8s wrapper) which has terrible documentation, history of horrific upgrade/update story, (& software quality).
In kubernetes, for calico CNI, the pod networking is done using veth pair. One interface remains on the host to be hooked to the host bridge and the other interface is sent to the pod namespace.
Mizu is deployed as a daemon set(one pod per node) and with enough privilege you can "tcpdump" the pod interface(the one where the envoy proxy puts back the packets unencrypted) (similar to "ip netns exec")
Basically, they tap the pod interface from host level. (They have a blog post about this as well: https://up9.com/gain-visibility-into-istio-mtls-traffic-with...)
This is my understanding so I might be wrong
That something like this is necessary is yet another testament to the complexity of Kubernetes.
I'm assuming that they didn't spend time authoring this for no good reason. It got 2700 stars on GitHub, which isn't nothing.
> The only complexity this calls out is that of microservice architecture done to an extreme, but not that of Kubernetes.
There are better approaches to dealing with tracking requests through a microservice architecture than a ham-fisted packet capture.
By default, the way networking is configured in Kubernetes is exceptionally complex. It certainly stands to reason that a packet capture may be useful at some point.
Kubernetes, as a technology, has support tooling being built around every interface it has.
Just to install Kubernetes you've got Kubespray, Kops, kubeadm, minikube, or you can choose from a smattering of managed Kubernetes options.
For performance you've got cAdvisor, and you probably need to export those metrics somewhere like Prometheus/Grafana.
Then there's Helm, which is yet another abstraction over deployments to Kubernetes, which is SUPPOSED to be the thing Kubernetes simplifies.
And let's not forget about Rancher. Can't run one Kubernetes cluster, because security and other reasons. So let's simplify running ten of them.
I can't think of a single other technology that was built to ostensibly simplify things, only to be completely enshrouded by support tooling.
Meanwhile, as a developer, I find that k8s does simplify a great deal with regard to deploying services and managing them once deployed - this, in turn, makes life easier for our infra folks, who no longer have to spend so much time splitting focus to provide direct dev support for deployment issues. Which is after all one of the reasons to run Kubernetes at all! It's built to simplify things, but not to simplify everything; there is a deal of irreducible complexity involved in the domain k8s addresses, and because it is indeed irreducible, that complexity can at best be managed and abstracted, not removed. That's what k8s intends to do, and in my experience it does that quite well - not without tradeoffs, true enough, but when in our line of work is that ever not the case?
[1] I mention this in particular because I used to take a view much like you do here, until an HN commenter pointed out in response to my own such fussing that it didn't have to be as complicated as I had understood it to be, and recommended trying k3s in that wise. It's been a year and I no longer remember who that was, but they were absolutely correct regardless!
Point taken, and it's a valid one. I'll throw some more of my perspective at you here (from a guy who mainly uses AWS managed services).
Let's talk scaling. Scaling is, and has been a solved problem before Kubernetes (see autoscaling groups). Imagine an autoscaling group whose instances launch and run one docker container -- your service (maybe also nginx, if you want). Maybe you even went the extra mile and baked your app's docker container into an AMI so you don't take a runtime dependency on Docker Hub or similar.
In an autoscaling group, you define where instances can launch and what type you want. You can get fancy about capacity by leveraging weighted capacities to increase instance type diversity. You set the amount of capacity you want, and it goes. ASG instances register with a target group behind a load balancer, and you're off to the races. Oh and by the way: if any of those things break, it's Amazon's problem, not mine (could be good or bad; I personally think it's good).
AWS has scaling figured out. Busy neighbors? Can't say that I have been impacted by that recently. Run out of disk? The impact of that is limited to one node that can be respun independently of everything else.
Kubernetes scaling is a similar story with a different interface... except now you have to worry about how services stack up on nodes. For example, suppose you have a node of size "4X" and a service of size "3X" (X can be whatever limiting factor you want - CPU or memory). You now have to deal with the potential issue of 1X being wasted on nodes that the 3X service is deployed to. Maybe there's a 1X service that can slot in there. Maybe the 1X service is not scaled as high as the 3X service, so some nodes just sit there with wasted utilization. You can deploy the 3X service to a correctly sized node group, but that's another thing to manage. You could go the route of oversubscribing resources, which would save money but it's another failure mode for your infrastructure.
If you run Kubernetes on AWS, it's probably using autoscaling groups anyway. And then on top it's doing its own orchestration. So to run Kubernetes on AWS, you have to understand both autoscaling AND how Kubernetes works.
How about security? The wisdom of the day is: don't colocate sensitive services on the same Kubernetes nodes due to container escape concerns (and probably some others). So now you're looking at running an entirely separate Kubernetes cluster, or at least segregating the sensitive service to a different set of nodes. That's more management that is simply a non-issue running a VM-focused workload.
How about load balancing? AWS can manage your load balancer and get you an SSL certificate that renews automatically. In Kubernetes, you can use the managed stuff or roll your own load balancing. If you elect to roll your own, you get the same functionality -- except if it breaks, it's your problem. If you roll with the managed stuff, you now have to understand not only how AWS load balancing works, but also how Kubernetes interacts with it. If the Kube interfaces breaks, it's probably your problem.
I could continue here, but I think you get the idea.
The central issue of Kubernetes is that you take on more problems than you otherwise have to versus using managed services. Amazon has entire teams working SPECIFICALLY on autoscaling, load balancing, SSL cert provisioning, etc. Any individual or team who thinks they can do better (in addition to their other job duties) probably has a bit of an ego issue.
That said, the largest production infrastructures I've worked with have all been hosted in EKS, so I'm not entirely without relevant experience, and that experience does give me still to think that k8s delivers value despite the added overhead and complexity you correctly describe.
The example that comes to mind is a multi-day downtime we took, a few years and one job ago. I don't recall the details in full at this point, but my interpretation of (what I overheard of) what our TAMs were saying is that we had scaled at a rate and to a point where their infra had failed to keep up - in any case, they and we were both clear that the problem was not of our making. Either way, though, none of our EKS-hosted infra was able to meaningfully operate, with nodes going unhealthy almost as soon as they came up.
But despite the EKS downtime lasting as long as it did, we only took a few hours' worth of revenue impact, because we were able quickly to bring up our critical-path services on GKS, cut over our LBs, and get back up and running with just enough functionality to do business. It wasn't perfect, and our CX folks in particular had a somewhat rough go of it, but we got back in business as quickly as we did because we hadn't gone all-in on AWS tools, and because one of the best infra teams I've ever had the privilege of working with had put real thought and planning into "never happen" DR cases like "what if EKS breaks?"
I've worked with some amazing teams, including the one I'm with now. Sure, I don't expect us to be as good at Amazon is in the things Amazon specializes in, and I certainly don't imagine that most teams are too likely to run into the kind of mishap I just described. But most of the stuff you're describing is almost table stakes these days - Amazon, Google, Azure, Tencent, no doubt a dozen others, hell, even DigitalOcean offers most of those capabilities these days, although they're newer enough at it that I wouldn't go straight there for a significant prod workload - I tried a side project on their managed k8s a couple years back and had a lot of trouble, and while I'm sure they're a lot better now, I'd still be a little nervous.
(They're a lot cheaper, though! Or were, last time I looked - cheap enough I didn't think twice about paying out of pocket, which I super would have to weigh carefully with one of the big players. And if my stuff is k8s native rather than platform native, it's a lot easier to think about starting cheap with DO and migrating to AWS when it's time to join the big leagues, right?)
Maybe AWS stays good forever, or maybe it doesn't. For a side project, I'd probably be fine with just using their own-brand offerings. But for a real business? I've never been a founder, and I don't suppose I can really imagine I know. But I do think that, at the point where I'd accrued enough runway to start thinking hard about the future - planning in detail for a horizon of not a month, not a year, but two or three or five years out - I think that'd be the point where I would want very much to prioritize making sure I could keep the doors open to customers even if AWS did one day catch fire, because I've seen that happen before. Sure, if everyone else is AWS-bound too, I'm probably not in worse shape for the outage than they are. But why settle for "not worse" when I can shoot for better? Why be born and die with AWS if, at the price of a little more work early on, I can buy myself a meaningfully longer runway by avoiding big-boy infra costs until I know I need big-boy infra?
I don't really know, I suppose. As I've been at pains to point out in all these conversations today, it's always a tradeoff no matter what you do, and the trick is all in getting the most upside out of whatever choices you make - especially the complicated ones. If I'd been more hands-on with AWS and knew more of what you know, maybe I'd hold an opinion much like yours. But I don't, and this is the thinking that my judgment and experience lead me to.
I think I'm not full of it, but who knows? Maybe I'll go scare up a seed round and we'll find out in the only way that really counts whether or not I'm talking nonsense. :) But in the meantime, kicking it around seems a fine way to pass an afternoon when I'm too ill to work.
You have really hit the nail on the head about risk, and what it means to go "all-in" on a particular provider.
There's a very relevant case to be made for electing to remain provider agnostic if your business is sensitive to outages. There is a balance between avoiding administrative overhead and mitigating risk.
I like to think about it like buying insurance (I'd put security posture in the same bucket, too). Pursuing a cloud-agnostic design has some cost per month. A cloud vendor outage has some chance of happening per month, and if it does occur it will cost you $X. The business needs to figure out when:
cost(cloud-agnostic design) < risk(cloud vendor outage) * cost(cloud vendor outage)
Figuring that out isn't trivial, especially because a large part of cost(cloud-agnostic design) will be in employee hours. At best you probably get a ballpark estimate.