OOM relation to vm.swappiness=0 in new kernel
mysqlperformanceblog.com
mysqlperformanceblog.com
First, setting vm.swappiness to 0 does not have the beneficial impact that people say it does, and yet it's religious dogma to set it that way on MySQL servers. To my knowledge there has never been a single article published that clearly demonstrates its benefit under a controlled test. The evidence of any benefit is anecdotal at best.
Second, you can easily exclude any process from being killed by the kernel's OOM killer: "echo -17 > /proc/PID/oom_adj"
Doing such with a large memory process is a sure way to cause a server to shut down and reboot when it runs out of memory. OOM killer is bad to large memory processes, but the alternative is worse.
Ideally you don't want to swap, you definitely don't want to preemptively swap, but even swapping is better than rebooting a server.
So much time is wasted by not enough swap space resulting in rebooted or hung machines. The way linux memory allocation works, it is really not a good idea to constrain swap. It is really easy to trigger the OOM killer by using up too much virtual address space. Doing a `fork/exec` from a process near your physical memory limit can easily get you over virtual memory limit with not enough swap, even though under this condition no paging would occur.
swap on and have great week.
"avoid swapping out pages of important process or process groups while there is a reasonable amount of pagecache on RAM so that we can satisfy our customers' requirements."
But apparently it's deciding to not swap even when there isn't RAM available, failing to allocate, and not falling back to swap.
It's also possible the bug is only in the redhat kernel, due to the correctness of this commit depending on some other commit not backported.
Deleted comment
I'm not saying it's not a valid feature to avoid swapping at all and just use it for hibernation, just that this does not appear to be the intent of this commit.
Looking at the patch, it looks like swapiness=1 now does what the old 0 did. Would that not cover your use cass?
Yeah, it seems, but the commit message, in my incomplete understanding, contradicts what you're saying. Also, I wouldn't feel so confident making claims about what the changes do, not without looking at the context and at least some understanding of that code.
I'm able to toggle between plain text and syntax highlighting via the '<>' icon @ the top of the code blocks.
They used to release RHEL kernels with hundreds of patches that could be applied in line, or removed/fixed individually. Now they keep one tree that they manage privately and release only the fully-patched source, without any documentation about what's changed or why. Your only hope of figuring out what has been changed or getting help in fixing it is to pay for support, or browse the commits at Oracle's RedPatch (https://oss.oracle.com/git/?p=redpatch.git). Basically, a RHEL kernel is an obscure product designed to make them money.
The primary reason RHEL kernels aren't stable is the bug this thread is about. There is no valid reason to backport a change to swappiness behavior into a 'stable' kernel. The whole idea behind 'stable' is it works, so don't change it. But RHEL has other ideas.
RHEL kernels are more likely to contain security holes than a vanilla kernel of the same version, partially due to the backporting of new or changing functionality. They also don't get as much testing and scrutiny as the vanilla tree. Besides internal testing, changes that aren't included in stock linux kernels are introduced into Fedora for their users to beta-test. Over time we've seen plenty of security holes make it into otherwise 'stable' RHEL kernels and tools by RHEL backports. It's usually a feature they're importing that brings along with it additional attack surface, or introduces a hole in pre-existing functionality.
It's also a big pain in the ass to get a RHEL kernel to support something a normal kernel would support, due to the major changes. You also have to modify whatever patches you want to apply on top of RHEL since most people create patches targeting the vanilla tree and not RHEL. In general if you have a problem with the RHEL kernel you have a lot better chance of success by throwing it out and downloading a vanilla kernel.
At this point RHEL's kernel is a nightmare hybrid of changes going into a tree that nobody else uses because they've moved to 3.x trees for new enterprise releases. Between vanilla 2.6.32 and RHEL's 2.6.32-431 there are 9144 files changed, 3,563,927 insertions, 699,721 deletions. The diff is 45MB compressed. The RHEL sources are 102 megabytes larger than the vanilla source. And in case anyone asks, no, this swappiness change was not introduced into the vanilla kernel's long-term support tree (which is still maintained for this same kernel version). (http://www.kroah.com/log/linux/2.6.32-stable.html)
As for your specific contention about the stability or security of backported patches -- can you provide a citation that RHEL kernels have any more fixes per year than, say, Ubuntu kernels? (Which BTW have their own backported patches.)
I'm not making an argument about who has more changes. RHEL just has bad changes. Bad because they're changing expected functionality. Bad because they're introducing unnecessary features. Bad because they increase exposure to security and stability bugs. Bad because their changes aren't documented publicly. Bad because they keep trying to support an antiquated platform that every other enterprise has moved on from. Bad because it's much more difficult to modify, and includes less support for modern hardware (unless it's patched in, again increasing problems). Bad because they have less testing and review. Bad because it creates vendor lock-in. I can't actually think of a single good reason to use a RHEL kernel, other than the vendor-locked-in support contracts and industry certifications that use RHEL as a standard.