Hacker News Reader

Comment by karel-3d | Hacker News Reader

mmap is fine when you know the file fits in memory, and you need random file reads/writes of only some parts of the file. It's not magic.

It's also quite hard to debug in go, because mmaped files are not visible in pprof; whe you run out of memory, mmap starts behaving really suboptimally. And it's hard to see which file takes how much memory (again it doesn't show in pprof).

View on HN

random reads are ok. writes through a mmap are a disaster.

Only if you are doing in-place updates. If append-only datastores are your jam, writes via mmap are Just Fine:

  $ go test -v
  === RUN   TestChunkOps
      chunk_test.go:26: Checking basic persistence and Store expansion.
      chunk_test.go:74: Checking close and reopen read-only
      chunk_test.go:106: Checking that readonly blocks write ops
      chunk_test.go:116: Checking Clear
      chunk_test.go:175: Checking interrupted write
  --- PASS: TestChunkOps (0.06s)
  === RUN   TestEncWriteSpeed
      chunk_test.go:246: Wrote 1443 MB/s
      chunk_test.go:264: Read 5525.418751 MB/s
  --- PASS: TestEncWriteSpeed (1.42s)
  === RUN   TestPlaintextWriteSpeed
      chunk_test.go:301: Wrote 1693 MB/s
      chunk_test.go:319: Read 10528.744206 MB/s
  --- PASS: TestPlaintextWriteSpeed (1.36s)
  PASS

Interesting. We use it to mmap a big file that just-enough fits in memory and we mostly read randomly (around 1/1000 of the file) and sometimes sparringly write and it works great. I haven't tested how would it work more fast with Seek/Read/Write, but mmap is very ergonomic for this; it just acts as a slice and it's basically invisible; and you can then take the slice of bytes and unsafe.cast it as slice of something else. Very easy.

Reply on news.ycombinator.com