Actually Kubelet should already be adjusting OOM scores to make sure that user pods (containers) get killed over Kubelet or the Docker daemon. Why didn't that work here?
I see OOMScoreAdjust=-999 for kubelet being used but why not -1000. -999 seems like it would be equally likely to be evicted as -998 unless the for_each_process(p) macro always goes first to last processes?
[0]https://github.com/kubernetes/kubernetes/blob/master/pkg/kub...
[1]https://elixir.bootlin.com/linux/latest/source/mm/oom_kill.c...
>unless the for_each_process(p) macro always goes first to last processes?
It seems it would usually go first to the first processes - the macro below - i.e. it would get to the "top" processes like kubelet, docker, etc before the containers.
#define next_task(p) \ list_entry_rcu((p)->tasks.next, struct task_struct, tasks)
#define for_each_process(p) \ for (p = &init_task ; (p = next_task(p)) != &init_task ; )
Given that "chosen" is updated only "if (points > chosen_points)" it seems that the first listed process with score 1 will stay the "chosen" in that situation, ie. it will be one of the top processes like the [-999] kubelet, not a [-998] container.
From a provider of an Azure class i'd have expected that they wouldn't rely on that machinery and would instead go the way of disabling the killer for the top processes outright.