Hacker News Reader

Comment by butterthebuddha | Hacker News Reader

mmap is great. I worked on a project a couple months ago where we had ~20 threads concurrently fetching data over the wire and dumping it into a file. We just ftruncated the file to ~4TB (bigger than we'd ever need it to be) and mmapped'd it all into memory, initializing the first 8 bytes to be an atomic integer that kept track of the actual size. The data needed to end up on disk anyway, and we were more than happy to let the OS take care of that us. We also had very little random access (mostly sequential writes) so it worked really well.

View on HN

Did you ever down-truncate it? Did your backup program take forever to crunch through non-existent 4TB (and take as much space to restore?)

Atomic access wise, that’s fine as long as you don’t map it through a network file system if any sort; also, $DIETY help you recover a consistent state if the power goes out or you kernel panic

Mmap is awesome, I use it all the time. But writing into mmaps robustly is hard. Best example to learn from I am familiar with is symas LMDB source code.

Unwritten pages in files that are extended don't actually take up space or appear as blocks; a good backup program should respect that. See https://en.wikipedia.org/wiki/Sparse_file

Indeed, but many programs, backup or otherwise, or unaware of it. IIRC gzip/bzip/zstd don’t (will spend time reading and writing all the missing pages) whereS gnu tar does.

Zstd by default will write sparse files when it detects runs of zeros during decompression, but isn't aware of it when reading a file.

We down-truncated on process exit.

I'll take a look at the symas LMDB source code, thanks for pointing that out.

Beware of multi-host filesystems (e.g., GPFS). The required locking and sync can be far slower than you might get on a local filesystem.

If nothing else, it's very useful to have a "don't use mmap" option in your code.

Another risk is that if the file system on which the file resides can get unmounted for any reason, access to the mmap'ed data will fail in nasty ways.

Reply on news.ycombinator.com