Hm, do you have any test/references on that? I always thought that in such cases system simply swaps memory to free some RAM.
It's really annoying, your application may get killed without any way to react, even to just print a "Low memory" error.
You can run out of both physical memory and swap space. Or your system is swapping is so heavily that, for all practical purposes, the system becomes unusable.
I should probably add this to my list of non-security reasons (or an angle at least) for using Nizza-style architecture [1] where critical functionality is taken out of Linux part to run on microkernel or small runtime. Maybe even potential to go further and have such apps that monitor for this sort of thing with a signal to apps that works across platforms. Not just this design flaw, but potentially others where app doesn't have necessary visibility. What you think?