Hacker News Reader

Comment by lucian1900 | Hacker News Reader

lucian1900

Actually, using mmap-ed files is a great idea. It's precisely what Varnish does too.

Yes, using mmap is a good idea and not because it is easy. The problem is that mmap-ing is simple, so a lot of first-time database engine developers use it, but a naive usage of mmap is a recipe for poor operational behavior. A competent database engine that fully leverages mmap is actually pretty complicated internally. Things like managing write back scheduling behavior are important.

mmap() is a good way of doing things but for different reasons than some people assume. Like all tools, you have to learn how to use it well.

The Redis project abandoned its VM implementation, to focus on in-memory operation, while suggesting that datasets > RAM should be sent to an RDBMS. This has resulted in a few success stories where the Redis+RDBMS combo has been successful.

The old-school DBA in me immediately thinks that the "70's tech" of the RDBMS is still widely used because the time to develop sufficient memory management is hard work and takes time.

That was exactly what my thoughts were too: letting the OS do all the memory management, caching, is a strategy many great projects use, among which PostgreSQL and Varnish.

However, I do feel there is something "wrong" about the approach MongoDB is taking. They need to allocate new files in huge buffers, which completely take up all I/O while being filled with zeroes. There is no logical hierarchy in the files, and it just feels a bit weird.

Perhaps they should've taken the approach PostgreSQL did, which is to simply use files and read from them instead of using mmap. The whole reason they went for a global lock instead of more granular lock is because the whole mmap'ed area is one big blob, and it was the most "obvious" approach.

Thanks for the insight on the global write lock. I've searched and searched and wasn't able to find anything on why they have the global lock.

Out of curiosity, is there a simple way to explain why someone would mmap instead of just reading files directly (I've never done any programming with mmap, so I'm a bit ignorant of its use cases)?

This SO entry seems to answer your question fairly well: http://stackoverflow.com/questions/258091/when-should-i-use-...

mmap also blows-up the TLB cache by taking up so many page addresses.

It is a widespread myth that postgresql lets the OS do all the memory management and caching. I don't understand why it is so prevalent though considering it is so trivial to look and see that it is nonsense. Postgresql reads all data into a shared buffer cache. The data is stored in files, so of course the filesystem buffer cache is also used, but the idea that postgresql leaves it entirely up to the OS is totally false. It has its own cache as well on top of the filesystem cache.

Reply on news.ycombinator.com