This is a pretty bad mistake from the customer if this is true. If not done already it would probably be good to expose Prometheus metrics on CPU/Memory usage per node.
Yes, this is true on it's face: it's bad to deploy containers to k8s without appropriate resource limits. However, this should in no way affect the operation of the node, so the implied transfer of responsibility for this incident from AKS to the customer is invalid imo.
It does if you don't have enough system reserved. It causes the kubelet to not function well and eventually the node goes sideways due to oom killing.
Right, I think we're saying the same thing. If the node is properly configured an end-user pod should not be able to take down the kubelet.
lol so aks forgot to provision enough resources and possibly setup enforcement and you are blaming the user? the user should be able to run as close to edge of "allocatable" as possible or even go over it and be oom kill'ed without bringing down the entire node. this functionality is even built into kubelet already. there's no way you can twist this to make it into user error.
More generally I should be able to choose to run an interruptible workload that I know to leak memory. I should expect that if I don’t, one of my coworkers will, and the node will stay up. Not leaving enough RAM for the node’s core resources is a mistake, but far from the worst thing in the world.
We are indeed working on more convenient container monitoring and logging on Azure portal.