And by zeroing pages, you prevent lazy memory allocation policies which do not allocate the pages until they're read or written for the first time. This can have a significant impact on memory usage, data locality etc.
So I'm quite skeptical that there is no difference between calloc and malloc. Is there more evidences of this somewhere?
But the OS/MMU doesn't distinguish between a regular write, and a write of zero. Thus, if you manually zero every page you get (And thus write zeros to the read-only zero-page), it'll page-fault back to the OS and the OS will have to allocate a new page so that the write succeeds - Even though if you didn't do the zeroing of memory you would have gotten the same effect of having a bunch of zeros in memory, but without having to allocate any new pages for your process.
Since malloc/calloc are generally used for smaller allocations, the chances you can actually avoid allocating some pages you ask for is pretty slim since a bunch of objects get placed into the same page (And thus writing to any of them will trigger a new page being allocated). There's also no guarantee there isn't headers for malloc to make use of, or similar surrounding your piece of memory, which makes the point moot - Just using malloc triggers writes to at least the first page. So while calloc/malloc are kinda compatible with lazy-allocation, you really shouldn't rely on it being a thing, and it probably won't matter.
It's worth understanding, but the chances it actually comes into play aren't huge. If your program does lots of small malloc's and free's, then it basically won't matter because you won't be asking the kernel for more memory, just reusing what you already have.
If you care about taking advantage of lazy-allocation for one reason or another, the bottom line is probably that you shouldn't be using malloc and calloc for that then. Just use mmap directly and you'll have a better time - more control, you have a nice page-aligned address to start with, and you can be sure the memory is untouched. malloc and calloc are good for general allocations, but using mmap takes out the guesswork when you have something big and specific you need to allocate.
E.g. imagine /etc/passwd is read and later on the page it occupied is put back in the free pool. Another process comes along and asks for memory. It gets that page and can now read /etc/passwd .
A few things though:
- Last time I ran this, in the memory dump, every second character was the letter "f". Like, in the dump, instead of saying "Words" it would say "Wfofrfdfsf". It changes each time, probably to do with my char onechar variable. (Actually now that I think about it, it should probably only be onechar[1] instead of onechar[2]. It was called onechar because I was trying to printf 1 char at a time)
- Before I changed "p" to &memcpy, it was "p = &p", and it would only read my environ before segfaulting.
- This program, on linux, dies almost immediately. It only spits out a few bytes.
So, no, you aren't printing out bits of files. You're printing out functions that your program has the ability to call.
He's starting somewhere in /usr/lib/system/libsystem_platform.dylib (where memcpy lives) and printing the rest of the executable code his program has mapped in.
It doesn't surprise me that some of this contains public keys and certificates. They are probably used by some OSX networking library.
On linux you can dump a processes address space mapping from file "/proc/<pid>/maps". Or to see an example just run: cat /proc/self/maps That cat command will dump the address space mapping for that instance of the cat command. If you run it multiple times it will show different address ranges because of a security feature (ASLR, address space layout randomization).
Also you shouldn't use fprintf or printf to print 1 char. printf will try to look for format characters in the "string" you are passing it. It would be better to use fputc(*p, stdout), then you don't even need to call memcpy() or the "onechar" array.
If you set "p" to a block of allocated memory, then your program should segfault shortly after reading past the end of the memory. How many bytes you can read past the end of the allocated block of memory depends on how the memory allocator manages pools of memory and it's own bookkeeping information is stored.
I can't get it to print 3g of stuff on OSX. It prints about 300m , which makes a lot more sense.
If you're sitting in a loop calling malloc(), filling it with data, processing it, and calling free(), changing malloc() to calloc() is just wasting cycles.
I guess one can rely on such behavior when writing a platform-specific libc. But in portable code? Yuck.
2. One could imagine this being configurable for embedded or HPC systems which don't care about security, though I'm not aware of people actually doing that.
3. The memory could equally well be overwritten with 0xdeadbeef and you have no control over that.
4. I once heard that Windows may sometimes fault-in nonzero pages if it runs out of pre-zeroed ones, but I couldn't locate this in MS docs so I'm not quite sure.
Thanks.
Heap allocation can easily contain data that have been previously freed in the same process and hence calloc has to perform some extra work to wipe them.
IIRC FreeBSD will zero freed pages in the background when it has idle time, and calloc can draw from that pool of zeroed pages "for free".
Of course it does.
> That only works when the pages are actually freed. Since actually asking the kernel for more memory is expensive, most malloc/calloc implementations will hold-on to some pages since it assumes you're going to allocate more. Since it reuses this memory without ever giving it back to the kernel, the kernel won't zero it for us.
It's what the standard system malloc does, possibly in cooperation with the kernel but not necessarily so (why would it need the kernel involved after all?)
Just as OpenBSD's malloc framework can do way more than trivially request memory from the kernel: http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/.... In fact, since OpenBSD 5.6 it'll junk small allocations on freeing (fill them with 0xdf) by default. And as usual that uncovered broken code…
> When we started junking memory by default after free, the postgres ruby gem stopped working.
To be clear though, the parent commenter (and I) are talking about the pages that malloc keeps around in the processes memory to be reused (And thus are never released back to the OS). Are you saying FreeBSD/OpenBSD/others have a system to tell the kernel when a user process has pages it plans to zero, and then a system for the kernel to notify the process when/if it does? That would be pretty interesting to see, but I've never heard of that being a thing.
I was talking about https://www.freebsd.org/doc/en/articles/vm-design/prefault-o... but after actually re-checking the docs rather than going from bitrotted memory pre-zeroing only happens for the initial zero-fill fault, so the original calloc is "free" (if there are zeroed pages in the queue) but freeing then re-calloc'ing will need to zero the chunk (unless the allocator decides to go see if the kernel has pre-zeroed space instead I guess). My bad.
> To be clear though, the parent commenter (and I) are talking about the pages that malloc keeps around in the processes memory to be reused (And thus are never released back to the OS).
Yeah so was I, but I was misremembering stuff.
> Are you saying FreeBSD/OpenBSD/others have a system to tell the kernel when a user process has pages it plans to zero, and then a system for the kernel to notify the process when/if it does?
Turns out no.