Kubernetes and Kernel Panics | Hacker News Reader

Kubernetes and Kernel Panics

netflixtechblog.com

113 pointszdw27 comments

Is Netflix still using their Titus container platform? It’s mentioned at the start of the post but the rest of the post seems to be able Kubernetes. Also the Titus repos on GitHub were archived a year ago.

https://github.com/Netflix/titus

Titus used to be a framework on top of mesos. Guess it's now a framework on top of kubernetes

Yep: https://netflixtechblog.com/consistent-caching-mechanism-in-...

Titus is a federation layer that sits on top of many Kubernetes clusters

Interesting to learn about last gasp packets, seems like a useful mechanism for any system. I generally find that observability into kubernetes is not that great especially for analytical queries. Answering the question of “where where my pods running for a replicaset at time X” is surprisingly not easy to answer with just events. Also, events are designed to have a short ttl so you need to extract them out of etcd and into some sort of long term storage and indexed in a way which you can run useful queries. At larger scales kube-state-metrics will blow up about any prometheus server outside of a full blown billion metric stream scale mimir deployment.

Sloop is pretty good for helping you understand events over time. It has an embedded database so you need to be careful with resources (apparently it's possible to split separately if required). Filtering out extraneous events can help minimise resource utilisation - it's amazing how many events a real-world cluster can generate running things like Kyverno, Flux etc. https://github.com/salesforce/sloop

Curious why this isn’t published to Pod events, seems like that would be the more natural place. They mentioned not wanting to patch Status but that too seems non-standard?

For the problem of where to run this if this was upstreamed, it’s an interesting problem - seems like you could make the case for the kubelet node agent baking this behavior in and gossiping to neighbors when the node terminates.

Probably be way less work and more reliable to emit node event when it comes back and scan for that. Seems like a case of google cargo culting. Remember kids, Google runs real machines and you probably dont

Does the AWS EC2 network filters UDP packets with a spoofed source? Is the firewall well configured?

Because it sounds like something spamming UDP packets with IP spoofing could take down all kubernetes nodes.

Maybe signing the kernel panic message could help too.

If you mean traffic from outside of the cluster, there is no reason for their cluster nodes to have public IP addresses, so they wouldn't even be routable from outside the VPC. I would expect this controller to run on the control plane and probably not be Internet-accessible by any means, but even if it were, I'm sure the security group only allows UDP port 6666 from inside the cluster, presumably based on source security group, not IP, so even if you could somehow spoof a private IP address and get it into the VPC, it would still not be accepted.

If you mean malicious traffic coming from their own pods, they could simply add Kubernetes network policies not allowing UDP port 6666 egress from any pod since the traffic they're actually hoping for would always come from a kernel, not a pod, and thus egress from a different interface.

So now you're left with the remaining attack vectors being spoofed IPs, or potentially even the correct IP but a spoofed packet, coming directly from EC2 instances in the same VPC, either by launching separate instances or compromising clusters nodes at the OS level. An insider with that level of access is always a threat pretty much no matter what you do because they could just delete your nodes through the EC2 service. You mitigate that with thorough audit logging, no shared or long-lived credentials, deep vetting of a very small number of personnel you give that level of access to, and the threat of near-certain jail time if someone does it on purpose anyway.

> Luckily the netconsole-setup command makes the setup pretty easy. All the configuration options can be set dynamically as well, so that when the endpoint changes one can point to the new IP.

There’s no spoofing happening here…

I think OP is highlighting that spamming spoofed UDP packets to this service is an attack vector for a denial of service attack.

in that case, yes aws ec2 (and other cloud providers) disable ip spoofing by default for VMs in your VPC.

Thanks. That’s a good feature from AWS.

Yes, but what if there was? A safer alternative would be to simply do the annotation and leave k8s to do the deletes on timeout as usual.

From the description, I’d think this is all in cluster, not public.

I'm not sure I understand the purpose of using netconsole in virtual machines. The serial console is easier to use, has fewer moving parts, and doesn't rely on a functional network stack.

Shouldn't ec2 kernel panic be handled on the ec2-level instead? It's not like they're running hardware nodes, after all. And surely kernel panics on k8 nodes shouldn't really be a thing? Are they loading custom kernel modules at random times in there?

EC2 it appears does not do this (capture a kernel panic reason and put it somewhere that an orchestration system can find it later).

Kernel panics are a thing. Whether a kernel is hosting k8s processes seems not terribly relevant to the probability it will panic? k8s seems neither likely to induce panics more often, nor to reduce their probability, imho.

This is probably a case of "at scale, everything happens", including kernel panics in VMs that most people assume never panic.

This is a really creative solution to orphaned pods.

I wish the article had some examples for things that caused a panic, I didn't hear of let alone have one in half a decade

Kernel panics are more common in Kubernetes because the container memory limitations mean the kernel is more often trying to reclaim memory, or failing to allocate, and that exercises little-tested error paths that are full of bugs.

Happens all the time especially on bleeding edge kernels. 4.1x was especially bad. Usually driver problems but sometimes other bugs too

Tl;Dr: use netconsole to send console messages via UDP to the k8s controller, where they are scanned for panics.

This is great info, and looks to be usable even outside of k8s.

looks like doesn't support 802.1Q?

how come?

Reply on news.ycombinator.com