Hacker News Reader

Comment by gavinray | Hacker News Reader

Database engineer folks:

I recently got to talk to someone who had written a query engine on the JVM

One thing I thing they said is that:

  > "The buffer manager should probably be rewritten to use the native OS page cache. When we wrote it originally this functionality wasn't easily available and so we used Java DMA (Direct Memory Access) instead."

I'm not familiar with what this means, would anyone be willing to explain more about OS page cache and how you'd implement something like that?

Would really appreciate it

View on HN

This article is a decent introduction to the page cache:

https://lwn.net/Articles/457667/

You typically have to go out of your way to _not_ use the page cache.

This looks super useful -- thank you!

I’m not familiar with JVM things and what DMA is there, but usually when people talk about the OS page cache, they mean the in-memory file cache stored by the kernel. This means that if you read or write to a page in the cache, you’d be accessing memory instead of disk.

The alternative is to open your file with O_DIRECT, which makes your reads/writes always interact with the storage system and bypasses the page cache.

I don't understand what that means unfortunately. I would think the second part of the comment refers to direct buffers in NIO that were added in java 1.4. One possibility is that he means directly interfacing with the operating system I/O system calls and using sun.misc.Unsafe to read or write to buffers.

If it helps, here's the docs and source for what I'm referring to:

https://docs.jboss.org/author/display/TEIID/Memory%20Managem...

https://github.com/teiid/teiid/blob/master/engine/src/main/j...

Reply on news.ycombinator.com