Hacker News Reader

Comment by searealist | Hacker News Reader

Your anecdote doesn't follow your warning.

Using mmap in an unusual way (to read chunks) on presumably legacy hardware doesn't generalize to using it in the obvious way (mmap entire files or at least larger windows) on modern hardware.

View on HN

Even old operating systems like Windows NT4 could easily memory map a window into a file as large as 256 MB at a time, and could map multiple segments concurrently. It could do this even if there was much less physical memory available, because this is a virtual memory mapping only.

Well-written code would start with a sliding window of some reasonable size such as 64 MB, and if that failed would try halving it repeatedly down to some lower threshold.

Unfortunately, the 64-bit era has lead to a "pit of failure" where many programmers incorrectly assume that this means that 2⁶⁴ bytes can be mapped reliably in a single call. This is never true, because of all sorts of operating system and hardware limitations.

I've seen "modern" code written with this assumption, such as a Rust library and a couple of C# libraries. They fail on older Xeons, some hypervisors, and 32-bit platforms.

Even in 2021 server applications run as 32-bit surprisingly often. For example, Azure's App Service defaults to 32-bit on the low-end tiers to save memory.

I think the story is to always benchmark first, and also to make sure your benchmarks reflect real-world use. What's dangerous is assuming something is faster without benchmarking.

I think many people reading that anecdote may come away with the idea that mmap is bad (and a monstrosity even) and read is good rather than your interpretation that you should benchmark better.

I dislike this kind of muddying the waters and I hope my comment provides another perspective for readers.

The whole point of the story seems to be that you shouldn't just read a story like this or the linked article and take away a simplistic data point like "mmap is faster" or "mmap is bad"

Best place to start is to have a good mental model of how things work and why they would be performant or not for a particular use case. Otherwise you're just taking shots in the dark.

It seems a more apples-to-apples comparison would be to open a file, seek(), read() a block, then close() the file. Just as bizarre as the repeated mmap, of course.

Regardless of how bizarre it is, I've seen this in real code in embedded applications before. It's a workaround of buggy serial port drivers (flow control or buffering is probably broken): You open the port, read/write a line, close it, and open it again...

Hah I came here to say pretty much the same thing! Recently ran into it and coding that workaround on a resource constrained system felt absolutely bonkers.

I already got 5 upvotes now. Looks like a real headache for many embedded developers ;-(

Indeed, my warning is about being cautious when making generalized claims.

If someone claimed running was faster than walking and then I told a story about how I once saw someone running in snow shoes on the grass and it was slower than walking then that would just be muddying the waters.

I remember a quote, I cannot find the source for now, but it basically says "A book can either be completely correct, or be readable, but not both."

Yet the article is literally is also showing a current peculiarity (user-space memcpy being faster than kernel-space memcpy) that may be or may not be true for your platform _now_ or in 3 years from now.

Reply on news.ycombinator.com