Are you sure that BestEffort QOS do not disrupt the entire node? I remember in the past a single pod would freeze the entire VM.
Prometheus scrapes of the kubelet have slowed down a bit, but are still under 400ms.
Prometheus scrape latency for the node kubelet has increased, but not it's still sub-500ms.
Note that this cluster (which is on EKS) does have system reserved resources.
[root@ip-10-1-100-143 /]# cat /sys/fs/cgroup/cpu/system.slice/cpu.shares
1024
[root@ip-10-1-100-143 /]# cat /sys/fs/cgroup/cpu/kubepods/cpu.shares
8099
[root@ip-10-1-100-143 /]# cat /sys/fs/cgroup/cpu/user.slice/cpu.shares
1024