this is not really true and it's dangerous to think this way (IMO).
one of the awesome things about virtual memory is you can "use" more memory than you actually have. if you only have 4GB of physical pages on your system and you allocate 18GB of virtual memory, things will be fine until you write to more pages than you have.
at this point, things are still fine! the memory manager will pick some pages (usually ones that haven't been accessed in a while) and page them out to a paging file. this behavior is transparent to your application. when your application tries to access those pages, the memory manager will bring those pages back in. if it can't bring the pages back in because all the pages are taken, it will choose some pages to put back into the page file. and so on ...
"so what happens when the page file is full?" well, modern operating systems have a page file that grows. so really, "what happens when the page file is uncomfortably large" is the question, and the answer there is the kernel will start killing tasks, and the first on the chopping block will probably be yours.
some platforms allow you to register for notification from the memory manager that there is a lot of pressure for physical pages or virtual addresses. the problem with trying to gain awareness like this is that the pressure on memory is generally totally outside your control. if the memory manager tells your app 'hey, free any memory you're not using' you are going to say 'oh yeah right i just had these hundreds of megabytes just sitting around but now that you mention it i probably should release them'. uh huh.
imo short takeaway:
1. check if malloc is null because the behavior of the allocator is platform dependent.
2. if you get a non-null pointer from malloc, treat it is valid. there is nothing else you can do. perhaps touching it will make the kernel kill your task perhaps not! it could be because your app is using too much memory. it could be because the app next door is using too much memory! you can't know.
3. you should take an operating systems class from a university if you want to really explore this topic in depth