Hacker News Reader

What happens if the program unexpectedly terminates during an update to these persistent data structures? Is the mmap'ed file corrupted, or still in a usable state?

View on HN

I don't have a good answer to this question yet. For now I only create the heap in swap, write it to disk, and then use it as read only mmap. To ensure the written file is valid, one can write it to a temporal file first, once confirmed the file is written, then mv the file to desired file name and location. This works for immutable data, but is a big blocker for me to make OPIC work on mutable back store.

This problem is generally hard. See [Ensuring data reaches disk](https://lwn.net/Articles/457667/)

You should have looked more closely at LMDB, which already solves this problem. Also you could look at LMDB's API for using fixed-address mmap, which lets you store pointer-based structures without any deserialization step at all.

The operating system will dutifully write back to disk whatever bytes your program left on those memory-mapped pages at the moment it terminated. If your data structures were in an inconsistent state -- pointers to nowhere, say -- then your file will get in the same inconsistent state, i.e. corrupted.

Reply on news.ycombinator.com