That depends on what you mean "more memory than you have". If it's more physical RAM than is currently available, not necessarily a problem - some other processes can be pushed out of RAM into swap. If it's more than that, some pages can be discarded because they're backed by binaries on disk. If it's more than that, some processes can be killed. If it's more than that, really bad things happen.

The problem is that there's no good way to differentiate between a process that merely wants more VM that it'll never touch and a process that wants actual RAM. In theory, every fork() should double your VM requirements - in practice, almost all are immediately followed by an exec() and the previous allocations are irrelevant. Do you fail fork() because you can't guarantee you can back every modified page in the new process? If not, why should you fail malloc() because you can't guarantee you can back it? There's no correct answer, and Linux lets you control which answer you want via /proc/sys/vm/overcommit_memory .