Does malloc lazily create the backing pages for an allocation on Linux? (2009)
stackoverflow.com
stackoverflow.com
mmap will go and create the page table mappings (entries) for that address space (~262144 of them with 4k pages)
unless someone's been messing with overcommit: the physical memory allocation is normally lazy
this is done by setting the "faulted" bit on the allocated page table entries
when the application first reads/writes from a page not yet allocated: a fault will occur, the kernel will step in, (hopefully) find an unused physical 4kb of memory, update the page table entry to point to it, unmark the faulted bit, then return to the application (which will be blissfully unaware that anything happened)
It does not, actually[1]. Page tables are a hardware datastructure with significant overhead. All the kernel does at mmap() time is create the a virtualized record of the mapping. It's not until the process takes a page fault to a mapped address that the hardware page tables are allocated and filled out. In practice this is a win because (especially with use cases like malloc) most processes tend to map large chunks of which they'll only use a subset in the short term.
[1] By default. There is a MAP_POPULATE flag you can pass that will allocate the page tables and allocate/prefault all the mapped pages before the system call returns.
I haven't looked at the mmap implementation in detail, but I believe it doesn't even do that: it just creates a new VMA (struct vm_area_struct in the kernel) for that address space range, and the page table is updated lazily.
It does. Last time I dug into that I think there was a 128KB threshold. Smaller allocations would use sbrk while larger once would use mmap.
The obvious answer is just looping through it and setting it to zero would be replaced with a call to `calloc` by an optimizing compiler, and doesn't calloc also do some kind of similar "lazy" trick? So what do you do? Mark the pointer as volatile and loop through it and set all the memory to 0 (or some other value)? Call memset, I guess?
EDIT: apparently malloc + memset compiles to calloc as well [0], so if you want to allocate, zero out memory, and make sure it's committed, using volatile seems like the best bet to me...
Therefore, you must use OS-dependent primitives to force eager allocation. On linux this can be achieved by two flags for mmap: MAP_POPULATE initializes the page table entries eagerly (but doesn't guarantee a backing store allocation), and MAP_LOCKED for touching all pages (just like the calloc, but at kernel level and more deterministic). Note that MAP_LOCKED doesn't raise error for touching errors due to low memory, so you still need to touch it again at user-level.
Are you really saying that this is not possible with malloc, you have to drop down to platform-dependent system calls like mmap?
This means that writing zero is unlikely to cause a real allocation to occur. Writing a non zero value may be more likely to cause a page allocation.
That said: it can be entirely legal for a platform to do more page deduplication than just special casing the zero page. Linux has some kernel tooling called KSM (kernel same page merging).
What do you want to do? Bake in the virtual memory semantics of the Intel architecture into C somehow? No, you need to use tools out of the scope of C for this and that’s how it should be.
“does it actually create 1 GiB worth of swap pages? Or does it mprotect the address range and create the pages when you actually touch them like mmap does?”
And how it is implemented can be seen, as an example, here:
https://sources.debian.org/src/glibc/2.28-10/malloc/malloc.c...
Note both mprotect and mmap references.
> does it actually create 1 GiB worth of swap pages
Create? No; probably never.
> does it mprotect the address range and create the pages when you actually touch them like mmap does?
Both mmap and mprotect are syscalls, and syscalls do not (directly) call other syscalls. So mmap does not do that. And malloc can't because it does not run anymore after it has returned.
On top of all of that, malloc(insanely_big_value) can still fail immediately by returning NULL even if you have overcommit activated. Granted, 1GB is probably never an "insanely_big_value"? Not even sure on 32 bits systems though.
The question was clear, the answer on the site was clear, by the answer here to which I responded was by somebody who I suspect simply hasn't read what was written there.