The number of times I’ve seen CPU limits kill off pods during even mild spikes and causing pretty much downtime and “disaster” is just as surprising. Work on autoscaling nodes instead, don’t use cpu limits.
If your service allows horizontal scalability, you can use autoscaling of pods with Horizontal Pod Autoscaler (ideally also with a cluster autoscaler) to increase pod count for a given service when some percentage of the requested CPU is exceeded, whether or not you set a CPU limit. Setting the cpu_request appropriately for your pods is critical to ensure that node CPU is not oversubscribed by the Kubernetes pod scheduler.
Pods where mem & CPU requests = limits are given the highest class of service ("guaranteed"). For your most critical and latency sensitive services, this is the best approach when also coupled with HPA. Assuming a 4.19 kernel or later, I suppose.
https://medium.com/better-programming/the-kubernetes-quality...
Memory limits however will kill the pod if the pod uses more than the limit.
In general, the problem is that people don't understand how these complex systems interact -- what do limits do, what are the consequences of limits, how do you decide on correct limits, what do liveness and readiness probes do, what is the kubelet's role in the probes, wait what's a kubelet, etc.
Kubernetes changes how you create the infrastructure but you still have the same problems you had in a distributed system before.