Is Memory Mapped File This Fast in Java? | Hacker News Reader

Is Memory Mapped File This Fast in Java?

sherxon.com

19 pointsTulik10 comments

Memory mapped files are great fun in C, but Java can't safely unmap files! Refer to this bug that's been open since 2002: http://bugs.java.com/view_bug.do?bug_id=4724038

Quote (December 2016): "I have a plan to implement this, but it requires changes to the Java Memory Model and to the compilers. I didn't get it done in time for JDK 9 because of being distracted by a ton of other things, but I hope I'll get it done by JDK 10."

If you find ByteBuffer's API awkward/insufficient, you might want to consider one of the replacements:

* Netty's: https://netty.io/4.0/api/io/netty/buffer/ByteBuf.html as used by Netty

* Agrona's, as used by Aeron and SBE: http://insightfullogic.com/2015/Apr/18/agronas-threadsafe-of...

You should never read() one byte at a time unless you like waiting. The C idiom is still true for Java:

byte[] buf=new byte[4096]; // this matters

while(true) {

  int num=in.read(buf,0,buf.length);

  ...

}

Also MemMapped files in Java have an int index so you can't really use them for large files without using another method as well.

Finally typically one has to parse the files to a string of a certain charset. This poses still more issues in terms of how you read files; esp multi and variable byte charsets.

When was MappedByteBuffer introduced in Java?

Furthermore, in what situations is mmap() (or MappedByteBuffer, if the situation is Java-specific) worse than read() et al?

I ask because Java is on my "I should probably know at least a little bit" list and I'm interested to pick up all the tidbits and tricks that I can - and this article screams "just use MappedByteBuffer everywhere you need I/O!", so I want to know if there are any exceptions.

map is worse if you get I/o errors: you will get a default or other fatal signal. It is also often slower than properly used read().

Is this a POSIX signal? How easy is it to handle from Java?

Also, I'm genuinely curious about what you mean by "properly used read()". Could you give an example?

(i tried to type "segfault" instead of "default" before autocorrect mangled it).

The most common signals you get in this situation are SIGSEGV and SIGBUS. AFAIK you can't catch it in Java.

You can get these errors in circumstances that don't involve physical IO device errors, you can also get them in cases where the OS decides you can't get to that file (eg another process truncates it, or you run out of space writing to a sparse file, etc).

By proper use of read() i meant being performance conscious. read() will usually have to copy the data into your buffer, whereas mmap() can just give you a virtual-memory assisted view of the kernel buffer cache. To eliminate this as a source of slowdown, you should size the read buffer so that it fits in CPU cache but still results enough work per read() call that the system call overhead doesn't get you. Around 128k is often close to the sweet spot.

> (i tried to type "segfault" instead of "default" before autocorrect mangled it).

(Ah. If only the dictionaries were bigger..... sigh)

> The most common signals you get in this situation are SIGSEGV and SIGBUS. AFAIK you can't catch it in Java.

I just had a look around, and it seems you're unfortunately right.

- http://stackoverflow.com/questions/39183517/java-8-applicati...

- https://forums.macrumors.com/threads/java-catching-stack-ove...

- Apple appears to have taken responsibility for one, but I don't think mmap was involved: https://lists.apple.com/archives/java-dev/2002/Aug/msg00129.... (found in the above link)

It kind of makes sense, the JVM pretends it's the "world", so something platform-specific affecting that world is generally going to be treated as a literal end-of-the-world (pun genuinely not intended... lol).

But then there's JNI. So Java does acknowledge that the OS is kind of there. And not all signals are fatal! This is mildly stupid.

> You can get these errors in circumstances that don't involve physical IO device errors, you can also get them in cases where the OS decides you can't get to that file (eg another process truncates it, or you run out of space writing to a sparse file, etc).

Oooh, riiight. Because you're not doing read()/write() I/O and mmap() is memory-centric, the kernel treats failures as memory errors. Makes a lot of sense!!

> By proper use of read() i meant being performance conscious. read() will usually have to copy the data into your buffer, whereas mmap() can just give you a virtual-memory assisted view of the kernel buffer cache. To eliminate this as a source of slowdown, you should size the read buffer so that it fits in CPU cache but still results enough work per read() call that the system call overhead doesn't get you. Around 128k is often close to the sweet spot.

Oh, it's because of cache consistency! I get it now!

I remember reading an article on here very recently about exactly this, but I can't remember any terms it had in it so I can't find it :(

I was slightly confused by the claim of the article saying:

"Memory mapped files are new feature in Java nio package"

They were introduced in Java 1.4, 15 years ago.

Thanks.

Reply on news.ycombinator.com