Comment by AceJohnny2 | Hacker News Reader

Comment by AceJohnny2 | Hacker News Reader

Parent Full thread

Quoting another user, jcranmer [1]:

> "The fundamental operation of mmap is to add new entries to the page table of a process, and the precise properties of those entries are heavily dependent on what the arguments to mmap are.

> When you mmap a regular file, you're essentially adding an entry to the page table that shares the data with the kernel's filesystem cache."

Now, understanding this requires some understanding of an OS's virtual memory function, and what "page tables" are. Those are what the OS use to track a process' memory, whose granularity is in "pages" (historically 4kB on Linux, though others use larger granularity such as 16kB).

mmap() has a a lot of flags [2] that affect the properties of those mapped pages. It is the swiss-army knife of memory management on Linux.

Some of those properties allow you to share memory with other processes (MAP_SHARED | MAP_ANONYMOUS), or just allocate memory (MAP_ANONYMOUS) or, by default, map an (open) file specified by the `fd` argument.

(fun fact! On linux, when you malloc(), you don't actually get memory, just an IOU from the kernel. Only when you access that memory, ie accessing those memory pages, does the kernel actually make the effort of allocating you that memory.)

[1] https://news.ycombinator.com/item?id=35412842

[2] https://linux.die.net/man/2/mmap

One thing I don't understand is that if I read a gigabyte from a file with the read call, why the kernel can't see that it's unused allocated memory, create copy on write pages, which could just be shared with the disk cache as long as the data is aligned (which it should be after a malloc + read call).

It seems the intuitive way to implement read if there's a complex memory subsystem that does all these great things anyways.

malloc() and the kernel would need to conspire to know this before you call read, when malloc() gives you back the memory in the first place

I guess you haven't read my parent: malloc doesn't give you memory, just reserves pages to be used in the future.

malloc(3) does not know or care about whether the OS actually commits the VM allocation that it got from somewhere (be it mmap(2) or sbrk(2)), apart from the fact that most implementations of malloc would write some kind of book-keeping header before the returned object (observe that on amd64 glibc, malloc(3) for large allocations returns pointers that are offset +0x10 from page alignment, that is where the two word allocated block header lives). The fact that the memory might not really exist is mostly invisible to userspace (as in, there is no portable way to find out that it does not exist).

The OS somehow detecting that argument to read(2) is page aligned block of uncommited meory and transparently creating MAP_PRIVATE mapping from that seems like nifty idea, but userspace tends to not have page-aligned buffers that often (see the aforementioned 0x10 offset for one reason why) and applications that intentionally allocates IO buffers with some particular alignment do so in order to bypass the block cache for particular IO patterns and are exactly the kinds of applications that tend to use mmap(2) for IO where that does not matter.

> most implementations of malloc would write some kind of book-keeping header before the returned object (observe that on amd64 glibc, malloc(3) for large allocations returns pointers that are offset +0x10 from page alignment, that is where the two word allocated block header lives).

Hmm. You're right about glibc, but macOS for instance reliably returns page-aligned pointers for large allocations. I wonder how Windows behaves.

> it's unused allocated memory

Well, first of all, it's not: malloc may have previously stored its internal structures in that memory so it may be dirty. Second, AFAIK if you ask malloc() for a gigabyte, it'll internally call mmap() on an anonymous file and will give you its result.

I see, that's how it's implemented.

But from what I understand there may be nothing in POSIX forbidding the kernel to do this optimization, just kernel authors decided against it (which is understandable).

That's right. Indeed, if the optimization is implemented properly it should have no observable behavioral difference except for performance (better in some cases, worse in others) which is not really something POSIX concerns itself with. So POSIX doesn't forbid it.

I can think of one edge case not mentioned by others, though.

One of the main benefits of mmap is that the data can be paged out while not used. But when you try to page the data back in, the attempt may fail, or it may return different data than was originally there. This is particularly likely with network filesystems or external storage. With mmap, these situations result in, respectively, the process receiving SIGBUS, or the contents of memory changing out from under it. The former is usually a crash (unless the program installs a signal handler), and the latter can be worse than a crash, if either program logic or compiler optimizations make assumptions about memory staying consistent when not written to. A program using mmap is opting in to that risk. A program using malloc and read is not.

That said, it may be reasonable to trust that the system’s main disk(s) (however you define that) won’t misbehave this way, and limit the optimization to files mapped from them. After all, if swap is enabled, the same potential risks exist for any memory that’s swapped out. Even without swap, if the root partition starts failing, the system is not going to stay up for long.

Or, the optimization could be enabled for other disks, but with the caveat that the pages would be swapped to the swap file/partition rather than paged out normally.

I see, the main disks should be probably defined when mounting (allow memory mapping), but it needs a lot of kernel work to know whether it all is worth it or not.

Still, the current problem with mmap is that it's buggy on some operating systems according to https://www.sqlite.org/mmap.html, so I guess it will cause more frustrations until AGI takes over the world :)

Sounds like an argument for liburing more. That also is supposed to let the kernel share buffers and reduce syscalls, but does come with a mechanism for handling errors without terminating the process

That's what mmap is for :)

When a read call completes, you know you've successfully read the data from the underlying device. If you have swap disabled you know you can read and write from that memory quickly.

With mmap'd data, a memory read/write can trigger a page fault and I/O, and if you encounter an error like a network filesystem not responding in time, the program sees a segfault.

> if I read a gigabyte from a file with the read call

`mmap` signals the kernel that you want to use the data someday. You `mmap` it, it automatically handles fetching and throwing unused bytes away.

`read` signals the kernel that you want to do something with the data now.

You `read` it, you need it in the memory now.

> why the kernel can't see that it's unused allocated memory

Only a in-process garbage collector would really know what part of your memory is unused. The kernel can't really know for sure.

Reply on news.ycombinator.com