Open sourcing oomd, a new approach to handling OOMs | Hacker News Reader

Open sourcing oomd, a new approach to handling OOMs

code.fb.com

154 pointskiyanwang39 comments

The "pressure stall information" looks interesting. It's exponentially weighted moving averages of the fraction of time "some" or "full" (all non-idle) tasks are stalled on each of CPU, memory, and IO. Also a counter of absolute time. [1]

Printing this seems like a good addition to "Linux Performance Analysis in 60,000 Milliseconds". [2] More useful that the load average printed by uptime, which hasn't aged well. Linux mixes cpu-bound and io-bound stuff into one bucket [3] which is a terrible idea, and the number of tasks is confusing at best when you have a mix of thread-per-CPU and thread-per-connection stuff running.

Looks like it's also exported on a per-cgroups basis, which is great.

[1] http://git.cmpxchg.org/cgit.cgi/linux-psi.git/tree/Documenta...

[2] https://medium.com/netflix-techblog/linux-performance-analys...

[3] http://www.brendangregg.com/blog/2017-08-08/linux-load-avera...

I was excited at first, and then realized this is a kernel patch and not a userland tool. Boooo. FB says "PSI tracks three major system resources — CPU, memory, and I/O — and provides a canonical view into how the usage of these resources changes over time." It should have been possible to do this in userland and make this (somewhat) kernel version-independent.

I'm sure I'll use this once there's OSes running a stable kernel with these patches, but I don't run non-vendor kernels in production [unless I'm forced to], and depending on kernel features makes it hard to support legacy systems.

how would that work

edit: you seem to be implying that FB is stupid and likes writing kernel code for no good reason.

This PSI data could be the holy grail of autoscaling metrics.

That looks like an interesting way to intercept vmscan's oom killer. I prefer to avoid it all together.

I prefer to set vm.overcommit_ratio=0 and then increase vm.vfs_cache_pressure to somewhere between 400 and 10000 depending on the server role, then set vm.min_free_kbytes, vm.admin_reserve_kbytes and vm.user_reserve_kbytes higher based on the amount of ram in the system using a simple formula.

For systems that are ephemeral, I also set vm.panic_on_oom to 2 so that they self heal. Avoid doing this on databases.

Of course, THP should also be set to madvise and memory defrag disabled unless you really need it. That will also free up some leaky memory and artificial pausing.

Beyond these things, I agree that CGroups is another way to set memory constraints around applications to further protect the server. There are many that won't have CGroups for a while however, as many systems are still on older non systemd distributions.

One does not need systemd in order to have control groups. Control groups are a kernel mechanism. However note that the headlined software needs version 2 of Linux control groups.

Agreed. I tried to use it on centos 6 but it was very incomplete. The kernel is just too old. I started using kernels from elrepo to work around this, but not many will do that in a production environment.

> The kernel is just too old

For my curiosity, is there a technical reason for the distro to use an old kernel? I would think one would wish to have all the bug fixes available, and I don't think the kernel has a bad backward compatibility story, so from the outside it seems like a sure thing -- leading me to believe there must be more to the story.

Enterprise distros such as Redhat fork the kernel, then patch for bugs and vulnerabilities, to avoid adding too many new features to a major version release of the OS. This actually makes a lot of sense from a stability and predictability perspective. It would be significantly harder to support the upstream kernel releases, as new features and behavioral changes occur rather often.

>I prefer to set vm.overcommit_ratio=0 and then increase vm.vfs_cache_pressure to somewhere between 400 and 10000 depending on the server role, then set vm.min_free_kbytes, vm.admin_reserve_kbytes and vm.user_reserve_kbytes higher based on the amount of ram in the system using a simple formula."

I understand vm.overcommit_ratio=0 but can you elaborate on what the net effect of setting all the rest of of these tuneables to these values is? What does this achieve and how do these work together?

>"For systems that are ephemeral, I also set vm.panic_on_oom to 2 so that they self heal. Avoid doing this on databases."

I'm guessing this triggers a reboot for you when the OOM evicts a process? Do you do this in conjunction with the above setting? Or is this an alternative?

Each of the memory settings are described in the kernel sysctl [1] documentation. The goal is to create a larger margin for user error or lack of memory budgeting. This often precludes ever needing swap and forces user space applications into the memory that is actually available.

[1] - https://www.kernel.org/doc/Documentation/sysctl/vm.txt

All of the settings should be calculated per server role and memory capacity after extensive testing of your application settings and performance.

> I'm guessing this triggers a reboot for you when the OOM evicts a process?

It triggers a panic if you hit OOM, rather than kicking in OOM killer which often leaves a system broken, sticky and confused. This should only be used for ephemeral systems or systems that do batch work and can handle reboots. This requires people to find the root cause of their over-allocation and fix them.

>"All of the settings should be calculated per server role and memory capacity after extensive testing of your application settings and performance."

So you're basically turning every node/instance into a special snowflake of sysctl settings all in the hopes of preventing an OOM? And you do this every time dev push a new rev of code? This doesn't seem very agile or sustainable at any of kind of scale or release velocity.

> Of course, THP should also be set to madvise and memory defrag disabled unless you really need it. That will also free up some leaky memory and artificial pausing.

Keep an eye out for patches making their way to distribution kernels around THP / memory. There has been some major work done upstream and THP is no longer the alloc-stall causing monster that it was before. It used to cripple some fleets around here (and so we'd have it completely disabled), but once Oracle backported the fixes in to UEK4 it has been smooth as silk with it turned back on.

Oh nice! Do you happen to know if those patches have been accepted into RHEL7?

I haven't been tracking that, can't honestly say.

If you're interested in experimenting and seeing if the patches fix any THP problems you see, UEK4 is completely free, and can be installed on RHEL7/CentOS7 etc. (http://public-yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_...).

Could maybe use it to provide details in a bugzilla report?

If you set THP to madvise without also having some hack that calls madvise on your program's text, you'll be throwing out significant amount of CPU rate ... 5-15% in my experience. That's something to weigh against the cost of the extra space.

For sure, any system level changes should be tested extensively to determine pros and cons. Performance testing should ideally be a part of the QA process. We found THP to be nothing but trouble on our systems across a myriad of applications. Most of our systems have 512GB+ ram so the problems induced by it are exacerbated quite a bit. The real issue in our case is the associated defrag. It exhibits nearly identical intermittent system lag spikes to Java FGC's.

There needs to be a signal apps can be sent when resources are low... there are quite a few apps that could do things like empty various caches under ram pressure for instance.

Similarly for disk space being low, or under IO pressure, some apps could hold off checking certain files / throttle things for a while.

This has been invented long since. On AIX it is SIGDANGER.

This is a good point, in Go all instances of sync.Pool (Go's generic object pool with thread local storage) are registered with the allocator/GC so that under heap pressure they can be drained, it would be great if the OS could request this draining too.

On android this Is done using LRU cache for background app.

If your app has a cached process and it retains memory that it currently does not need, then your app—even while the user is not using it— affects the system's overall performance. As the system runs low on memory, it kills processes in the LRU cache beginning with the process least recently used. The system also accounts for processes that hold onto the most memory and can terminate them to free up RAM.

This might become useful in the future:

https://lwn.net/SubscriberLink/759781/1ebdd9d9518b8f14/

I would be interested to see this compared to what I consider to be the null hypothesis: disabling overcommit altogether and handling out of memory at the application level.

poor, because 1. fork exists, and 2. the application incurring OOM is statistically unlikely to be the one that should be killed on multi-application servers, and 3. 99.9% of applications do not correctly handle malloc returning NULL in 100% of cases. some make an effort, but except for stuff like SQLite, almost nobody tests it, so it's usually broken at least 1% of the time.

> 3. 99.9% of applications do not correctly handle malloc returning NULL in 100% of cases.

0% of applications correctly handle being unable to write to a page they were given as writable. It's certainly true that few applications correctly handle a failed malloc, and in many cases there's not really a correct way to handle it, but at least it's available to handle.

>the application incurring OOM is statistically unlikely to be the one that should be killed on multi-application servers Could you please explain this?

Not convinced that has any advantage compared to a naive approach of:

- introspection at the application level: exit if memory usage goes way above usual (say by 2x)

- management by systemd to gracefully restart the application when it exits (dependancies etc), and also monitor memory use in case the application doesn't monitor it well (say exit when 2.5x the usual)

I could be made better, but at the cost of more complexity.

I guess this would be a good way to operate dunning-kruger.com, but if you want your site to stay up this seems problematic. If you introspective application forms some belief about "usual" memory usage when it's idle an unloaded, and you suddenly shift all of your traffic from another datacenter to this one, and the heap usage on that application increases by 10x, then all your replicas will suicide.

If the memory usage can increase 10x while this is not reflected in the assumptions, both the measurements and the assumptions are flawed

To give more context: I run many copies of a daemon with known memory leaks, on machines each with a load close to 0.7 and about 200M of free ram left. I consider that an heavy load.

This approach restarts one of the copies of the daemon when its own leaks become dangerous to the rest of the system.

It is not perfect, but good enough when the numbers are measured at peak. The 200Mb free RAM are all the extra leeway I care to leave.

Interesting approach. I'm curious to try it out.

After playing around with vm.overcommit_ratio, different swap sizes, earlyoom[1], and a few other variables, I still haven't found a happy medium between high memory utilization and low risk of swapping to death. vm.overcommit_ratio=0 is safe, but on systems where occasional swapping is tolerable and memory is limited (e.g. my laptop), I'd rather allow some overcommit.

The risk is that if many cold or unallocated pages get touched while the system is under high memory pressure, the system can become totally unresponsive. At the moment I use "Magic SysRq"+f to manually start the oom_killer, when possible. Obviously it's not a great solution. Is there some kernel tunable to keep the system responsive that I'm unaware of? What do you guys do for desktop/laptop systems?

1. https://github.com/rfjakob/earlyoom

Android also has its own low memory killer: https://android.googlesource.com/platform/system/core/+/mast...

Does doing this in user-space have any advantage beyond they could dev a PoC faster? It seems like building a better OOMK and having it live in user-space are orthogonal.

It's because this supports plugins which are intended to interact with other user-space programs in order to allow for cooperative resource management. These plugins would be much harder to write if they had to be implemented in kernal-space (you couldn't use standard user space libs for making RPC calls, would be easy to introduce kernal corruption if you don't know kernal programming). So yes it's for development reasons but not to allow for a faster POC. It's to allow non kernal devs to easily write plugins.

It would be nice, but for complicated systems in the kernel you sometimes have to provide evidence that the old way sucks and the new way is better. With this work we can easily point at our production workloads and say the in kernel oom killer sucks and this new way is obviously better and hopefully work towards a better in kernel solution. That being said the user is always going to know what they care about more than the kernel, so being able to say “kill chef first, alway” can be a nice thing to have.

That's neat but I question if many of us are at the level of Facebook w.r.t. the need for this. To wit, who here runs their own data center?

What does running their own data center have to do with it? As I understand it, this daemon could be deployed to a VM instance.

Great question! You only care about this if you care about cost utilization. If you don't care about cost then you will just statically pre-allocate the space on a host, waste the slack, and never overcommit. If you do care about cost then the pessimistic pre-allocation of space will look wasteful and you'll want something like this.

The other aspect of your question is about how "many of us" are working at places with huge datacenters. If you count up all the folks at Amazon, Google, Microsoft, Facebook, and Apple how does that compare to the long tail of people operating smaller datacenters at smaller companies?

OOM issues have arisen at every at-scale operation I've ever been involved in. OOM killer does its job, but this seems like it could make it even more manageable.

There are companies out there who prefer to be in control of their own data and core business, but their numbers are shrinking.

Reply on news.ycombinator.com