You can turn off overcommit, in which case you'll get a null return instead of OOM killing.

But also, you can still get a malloc failure without actually be running out of memory if the allocator can't find a big enough contiguous chunk of address space.

This is highly unlikely on a 64 bit system, but if you try to malloc gigabytes on a 32 bit machine you might see it.