So, the general point about fragmentation, non-moving memory management, and process heaps versus what can actually be returned to the OS is true and common, but your “single contiguous block” model is somewhat off:
On at least modern GNU/Linux, malloc can and does still use brk/sbrk, but will also use mmap for particularly large allocations, as can (on my system) be shown by stracing this C program:
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
int main(void)
{
printf("%p %p\n", malloc((size_t)1 << 14),
malloc((size_t)1 << 20));
return 0;
}
On Windows, HeapAlloc is at least presumed to use VirtualAlloc-or-equivalent underneath, and there is no brk/sbrk.
Regarding the other two factors you mention: firstly, if you make a large number of allocations on a brk-based heap and then free most of them, a page from that heap can still only be paged out if none of the live allocations intersects it. (If you make a single large allocation, see above re automatic use of mmap.) If the heap is highly fragmented, you may not be able to page out many pages. If the heap is a brk-based heap and is not highly fragmented after the free phase, it's more likely there would be a bunch of space near the top, and then malloc could shrink the heap at the OS level too. One allocation pattern I can imagine where what you say would have a higher chance of happening, though, is “allocate many times for intermediary data during an operation, then allocate for the result, then free all the intermediary data”, in which case the result might well be stuck at the top of the heap with a bunch of dead pages underneath. But you still take lazy disk I/O doing the paging, and you still take a reservation across system virtual memory as a whole. On the whole I'm not all that convinced that this is a “mitigation” that's worth paying attention to from a developer perspective; it might be more useful from a sysadmin perspective.
For the second, terminology-wise, I've never heard “memory-mapped file” used to refer to anonymous mmap; I think I've only ever heard it called mmap in a Unixy context. On Windows, VirtualAlloc and MapViewOfFile are separate API functions, and VirtualAlloc is the equivalent of anonymous mmap.
Windows allows the creation of additional heaps for HeapAlloc such that a heap can be destroyed all at once, and I presume these use their own VirtualAlloc regions underneath. There are various C libraries that allow arena-based allocation on Unixy systems as well. These can even both reduce manual effort and increase performance if your memory usage patterns involve a large number of allocations for some operation followed by freeing everything, because then you don't have to free every block individually; you just throw away the entire arena, like an in-process equivalent of what exit() does to whole-process memory resources. Your main option in C if malloc is causing fragmentation/performance problems for you isn't usually to jump to raw mmap unless you're writing your own allocator on top of that; it's to jump to picking a more advanced malloc and maybe giving it some extra context about what you're doing with the memory.