Handling out of memory conditions in C

The "segfault policy" is an extremely bad idea.

Over the past couple years, several major security vulnerabilities have been caused by programs that took the "segfault" path making attacker-controlled array references to NULL pointers. It's not hard to see that if you index something with a standard native integer, 0 + index can mean any word in memory. If you write to that, that's game over.

It's for a similar reason that I don't recommend developers explicitly check malloc, or use xmalloc wrappers. Both are opportunities to make an error that shouldn't be possible in properly deployed code. Adobe Flash had explicit error checking with a convoluted O-O-M policy, forgot a check, and Mark Dowd was able to use a Flash header field to create a 32 bit offset to a NULL pointer, and:

http://chargen.matasano.com/chargen/2007/7/3/this-new-vulner...

xmalloc addresses this, except if you forget to use xmalloc, or if you forget that "strdup" needs an "x" too, or if you use a library that calls malloc.

The safest approach is just to configure malloc to abort (or invoke your O-O-M handler via longjmp or signal). Most C libraries have explicit support for this, but every development platform will allow you to preload a malloc wrapper if they don't.

For what it's worth, I came by this opinion not because of security, but because as a systems programmer I was tired of dealing with useless conditionals on a function that's called so often; at one job, I shrank my codebase by 30% (!) by losing malloc checks and the associated "this function should return -1 to indicate malloc failed inside of it" detritus it carried. The "just preload a malloc that aborts" rationale I got from a friend (then) at Juno.

Perhaps I'm missing this, but what about the "return failure" policy (for libraries)? If malloc fails, log an error and return a failure to the calling application. It isn't recovery, since it doesn't attempt to continue working, but it doesn't abort() the whole program (including calling application) either.

The calling application can then close the current handle to our library if it chooses (thus freeing up all of the memory again).

(This is what our library, x264, does, and IMO makes the most sense when dealing with situations where it would be insane to try to recovery, but in which one doesn't want to kill the calling app as well. We added malloc checking when it became possible to set options that would hit 2GB allocation on a 32-bit system.)

Checking the malloc return value doesn't always guarantee that you'll be OK.

From the malloc man page (under BUGS):

"By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory really is available."

It goes on to describe how this behavior can be turned off.

Very good article, I like the examination of various open source project's use of memory allocation and failure checking.

For me personally it depends on the project. If there is nothing to clean up in the rest of the code an abort in the malloc function is fine. But if there are other things to deal with at shutdown you have to bit the bullet and check your return values all the way back up the call stack.

Allocate 1MB of memory upfront. When OOM happens, free that memory and run recovery mechanism.

Similar policy is used in Linux for disk space: 10% of disk space is reserved for root.

PS. Linux OOMKiller already do something similar: it just kills a process to free some memory.