Comment by AshamedCaptain | Hacker News Reader

Comment by AshamedCaptain | Hacker News Reader

Claiming "mmap is faster than system calls" is dangerous.

I once worked for a company where they also heard someone say "mmap is faster than read/write" and as a consequence rewrite their while( read() ) loop into the following monstrosity:

1. mmap a 4KB chunk of the file

2. memcpy it into the destination buffer

3. munmap the 4KB chunk

4. repeat until eof

This is different from the claim in the article -- the above monstrosity is individually mmaping each 4KB block, while I presume the article's benchmark is mmaping the entire file in memory at once, which makes much more sense.

After I claimed the "monstrosity" was absurdly stupid, someone pointed to a benchmark they made and found that the "monstrosity" version was actually faster. To me, this made no sense. The monstrosity has triple the syscall overhead vs the read() version, requires manipulating page tables for every 4KB block and as a consequence had several page faults for each 4KB block of the file. Yet it was true: their benchmarks showed the monstrosity version to be slightly faster.

The idealist in me couldn't stand this and I reverted this change, using for my own (unrelated) experiments a binary which used the older, classic, read() loop instead of mmap.

Eventually I noticed I was getting results much faster using my build on my single-socket Xeon than they were getting on their $$$ server farms. Despite what the benchmark said.

Turns out, the "monstrosity" was indeed faster, but if you had several of these binaries running concurrently on the same machine, they would all slow down each other, as if the kernel was having scale issues with multiple processes constantly changing their page tables. The thing would slow down to single-core levels of performance.

I still have no idea why the benchmark was apparently slightly faster, but obviously they were checking it either isolated or on machines where the other processes where running read() loops. I guess that by wasting more kernel CPU time on yourself you may starve other processes in the system leaving more user time for yourself. But once every process does it, the net result is still significantly lowered performance for everyone.

Just yet another anecdote for the experience bag...

Your anecdote doesn't follow your warning.

Using mmap in an unusual way (to read chunks) on presumably legacy hardware doesn't generalize to using it in the obvious way (mmap entire files or at least larger windows) on modern hardware.

I think the story is to always benchmark first, and also to make sure your benchmarks reflect real-world use. What's dangerous is assuming something is faster without benchmarking.

I think many people reading that anecdote may come away with the idea that mmap is bad (and a monstrosity even) and read is good rather than your interpretation that you should benchmark better.

I dislike this kind of muddying the waters and I hope my comment provides another perspective for readers.

The whole point of the story seems to be that you shouldn't just read a story like this or the linked article and take away a simplistic data point like "mmap is faster" or "mmap is bad"

Best place to start is to have a good mental model of how things work and why they would be performant or not for a particular use case. Otherwise you're just taking shots in the dark.

Indeed, my warning is about being cautious when making generalized claims.

If someone claimed running was faster than walking and then I told a story about how I once saw someone running in snow shoes on the grass and it was slower than walking then that would just be muddying the waters.

I remember a quote, I cannot find the source for now, but it basically says "A book can either be completely correct, or be readable, but not both."

Yet the article is literally is also showing a current peculiarity (user-space memcpy being faster than kernel-space memcpy) that may be or may not be true for your platform _now_ or in 3 years from now.

It seems a more apples-to-apples comparison would be to open a file, seek(), read() a block, then close() the file. Just as bizarre as the repeated mmap, of course.

Regardless of how bizarre it is, I've seen this in real code in embedded applications before. It's a workaround of buggy serial port drivers (flow control or buffering is probably broken): You open the port, read/write a line, close it, and open it again...

Hah I came here to say pretty much the same thing! Recently ran into it and coding that workaround on a resource constrained system felt absolutely bonkers.

I already got 5 upvotes now. Looks like a real headache for many embedded developers ;-(

Even old operating systems like Windows NT4 could easily memory map a window into a file as large as 256 MB at a time, and could map multiple segments concurrently. It could do this even if there was much less physical memory available, because this is a virtual memory mapping only.

Well-written code would start with a sliding window of some reasonable size such as 64 MB, and if that failed would try halving it repeatedly down to some lower threshold.

Unfortunately, the 64-bit era has lead to a "pit of failure" where many programmers incorrectly assume that this means that 2⁶⁴ bytes can be mapped reliably in a single call. This is never true, because of all sorts of operating system and hardware limitations.

I've seen "modern" code written with this assumption, such as a Rust library and a couple of C# libraries. They fail on older Xeons, some hypervisors, and 32-bit platforms.

Even in 2021 server applications run as 32-bit surprisingly often. For example, Azure's App Service defaults to 32-bit on the low-end tiers to save memory.

If someone hears "mmap is faster than system calls" and then mmaps and munmaps 4KB chunks at a time in a loop, not realizing that mmap and munmap are actually system calls and that the benefit is not about calling those functions as much as possible, there is no saving them.

That's not the fault of a 'dangerous' claim, that's the fault of people who go heads first into something without taking 20 minutes to understand what they are doing or 20 minutes to profile after.

Yep, there's no substitute for a qualitative understanding of the system.

You'd need significantly more time then 20 minutes to form an informed opinion on the topic of you don't already know basically everything about it.

The only thing you could do in that timespan is reading a single summary on the topic and hope that this includes all relevant information. Which is unlikely and the reason why things are often mistakenly taken out of context.

And as the original comment mentioned: they did benchmark and it showed an improvement. They just didn't stresstest it, but that's unlikely doable within 20 minutes either.

In 20 minutes you can read what mmap does and see that you you can map a file and copy it like memory.

In another 20 minutes you can compile and run a medium sized program.

Neither of those is enough time for someone to go deep into something, but you can look up the brand new thing you're using and see where your bottlenecks are.

But without going deeper you are likely to have preconceived notions like "mmapping a gigabyte file into a memory actually uses a gigabyte of memory, oh noes!" I can totally see the thinking behind this case.

I don't think an unsubstantiated general predication about people making a wild assumption has much to do with the point that nothing is going to protect people from performance pitfalls who don't learn the bare basics about what they are doing.

But they did learn the bare basics, enough to implement something somehow working. Or I don't get what you're trying to say?

Who is 'they'? Is it still the made up future scenario? The bare basics of mmap on a file is that it maps the file into a processes memory space instead of copying it into memory.

Was it perhaps a multi-threaded task? Because that would almost definitely crawl.

In general, unmapping expensive, much more expensive than mapping memory, because you need to do a TLB-shootdown/flush/whatever to make sure a cached version of the old mapping is not used. A read/write does a copy, so no need to mess with mappings and TLBs, hence it can scale very well.

It was multi-processing. I guess mmap(or at least munmap) also needs to send an IPI even if no other processor currently has the same VM, to avoid race conditions.

The fact that you once met some idiots who wrote code that wouldn't pass the review of any experienced Unix programmer doesn't really invalidate the fact that faulting in a mapped page is generally faster than getting the same data with a page-sized read. The one thing doesn't have anything to do with the other.

Right, especially if the alignment within the file is a multiple of the page size, and it's a shared mapping, many OSes can make a zero-copy read directly from non-volatile storage into the process's memory (buffer cache pages mapped directly into the process's address space).

The kernel can, in theory, estimate what page you're going to load next, so loading 4KB at a time may not have the page faults you'd expect.

On Linux mmap_sem contention is a well-known concurrency bottleneck, you may have been hitting that. Multiple efforts over the years have failed to fix it, IIRC. I guess one day they'll find a good solution, but until then, take care.

mmap is a more ergonomic interface than read too. How often are people copying a file to a local buffer, or the whole file a chunk at a time, in order to use the file like an array of bytes. mmap gives us a range of bytes right off the start. Even if not optimal, the simplicity in usage often means less room for bugs.

Be mindful that can give you Time Of Check/Time Of Use (TOCTOU) security bugs though. The buffer can change out from under you.

That can sometimes be a feature, not a bug. I've worked on systems that used memory mapped files as a form of persistent shared memory between processes.

Thats the open/lock mode too. Same as without

I'm also puzzled as to why the benchmark of the monstrosity would have been faster. How is it not doing the exact same work as read()'ing 4KB, and then writing out the same to the destination file? And, as you said, 3x the system call overhead doesn't seem good, either....

"munmap" is actually expensive as it has to flush the TLB

Isn't the whole point of mmap to randomly access the data needed in memory? Did they think memcpy is a totally free operation or something, without any side effects?

Out of curiosity, what was the use case where they were trying to get these marginal gains out of their program?

Reply on news.ycombinator.com