A visual guide on troubleshooting Kubernetes deployments
learnk8s.io
learnk8s.io
The article does contain a lot of useful information. I don't think it was intended to ride the "k8s is too complicated compared to vanilla X" bandwagon, despite the huge flowchart at the top.
Our CI/CD was leaking "review" deployments, I forgot about them until one day I upgraded a node and the entire site went down, even though everything was green. Turned out there is some sort of naximum amount of nginx entries in ingress and we were hitting it. That was some frantic debugging, solution was just to delete the spurious review deployments.
If your pod's `cmd` is the cause of the crash, simply replace it with `sleep 50000` which will allow you to then get a shell to debug the crashing cmd (granted your container image has a shell available).
If your pod is getting killed by its Deployment (because of a scale down for example), simply "edit" the pod (`kubectl edit pod [pod-name]`) and change its deployment name to some inexistant deployment prior to exec-ing a shell. This basically orphans the pod from its deployment; you will need to manually delete it afterwards.
If your pod is getting killed by k8s because it's exceeding its resource `limits`, edit the pod manifest and remove the limits (again, using `kubectl edit pod`)
In the end, it basically mostly comes down to editing the pod's manifest file and removing the things that could instruct k8s to restart your pod. Another thing would be to remove the `livenessProbes`.
I'd like to see some means of keeping a pod running even if the application is crashing. Sometimes logs are being sent to a file, sometimes running the application manually gives a clue. Lacking that - well I've found various tricks such as changing the image to 'ubuntu' and replacing the command with 'tail -f /dev/null' so a crashing pod will stay alive and I can exec into it and run the real command by hand. But it's a hack and doesn't always get the info you need.
In OpenShift (I'm a Red Hat Consulting Architect), we have the ability to debug Pods (Failed, Running, or otherwise), DeploymentConfigs (Deployments), some other things, but not BuildConfigs. You just do something likes `oc debug pod-name-hash` or `oc debug deployment/deployment-name` [1]. What this does is start up the Pod with all the configuration (ConfigMaps, Env Vars, Mounts, Secrets, etc.) but replaces CMD/ENTRYPOINT with a shell using `/bin/sh`. This might be magic bits we've added to our `oc/kubectl` (they're one in the same in OpenShift) but I don't see references for similar functionality in Kubernetes. Useful for identifying where a configuration may be slight off what you or CMD/ENTRYPOINT are expecting.
[0] https://kubernetes.io/docs/reference/kubectl/cheatsheet/#vie...
[1] https://cookbook.openshift.org/logging-monitoring-and-debugg...
Our project is a bit unconventional and I'm usually dealing with applications that have been minimally and reluctantly containerized. Pods that crash with no logging and no tracing is probably something I have to deal with more than others.
With the customization you can do to k8s the debugging can get a bit more weird when dealing with operators and web hooks.
Props to the author for the chart. He's actually providing real value to all the suckers, err, SREs stuck dealing with this stuff.
Most of the diamonds listed here aren’t things you have to remember, they’re statuses that are explicitly called out in the responses of those commands. This is an over-complicated chart that in reality is summed up as: look at the pod, look at the service, look at ingress.
That's one of the strengths with K8s, at least there is documentation, unlike a 1000 line perl script with no tests which does service discovery.
I fully agree on that being a major strength of k8s. Looking at the resource manifests gives a ton of insight into how a system is supposed to function.
I went through your problem before on a practice cluster and ended up giving up on that one.
Realistically the hard way doesn't actually explain much of the commands you're running, and much of it covers the tedium of getting the many many certs you need to bootstrap everything (which will then expire and not rotate automatically). It's much better to just use kubeadm, and take some time to read all the args to kubelet, kubeadm, kubeproxy, etc...
Sure but I am not comparing it against manually done / bash script powered / Ansible deployments. Even if I was the CI/CD pipeline would capture many of the problems early and than I have an immutable AMI with autoscaling groups that is very easy to debug, networking setup is super simple and load-balancing is also straightforward to troubleshoot. Cramming all of these into a simple project that is not as easy to debug is not worth the effort for us.
A complete flow chart of all the possible states for any non-trivial thing appears complex. In practice the actual problems you run into are part of a much smaller set of possibilities.
- using ping
- sshing into a box
- using ssh port forwarding
- using ps aux
- understanding system load
- understanding oom killer messages
- is this systemd? use systemctl, otherwise figure it out
- is this systemd? use journalctl, otherwise figure out where an app is logging
- is nginx running? what box is it running on?
- is the nginx config okay?
- what about haproxy? are health checks going through?
Plus dealing with whatever configuration management system you're using, credential store you're using, etc.
Kubernetes, while complex, does take over a huge chunk of responsibilities from existing deployments. As such, a large part of any debugging will be done via kubectl and on a much different level of abstraction than if you were dealing with individual machines and services deployed on them via Ansible.
Yes, there's new things to learn. But for many cases (not even just scale!) it's quite worth it.
For those that do need a solution like Kubernetes, charts like this are helpful, and the knowledge requirements certainly aren't too steep relative to comparable platforms.
What about Kubernetes makes it especially worth going in and commenting that you don't have a use case for it, in a thread dedicated to Kubernetes? Would you make similar comments about other platforms/libraries/technologies that you, for whatever reason, don't have a use case for?