Author here, I'm away from my computer atm, but I can cook up a repo with each test in a few hours when I get home.
I designed the tests as a drag race because that mimics my real world usage.
I designed the tests as a drag race because that mimics my real world usage.
I tried recreating your DataInputStream + BufferedInputStream (wrote the 1brc data to separate output files, read using your code - I had to guess at ResultObserver implementation though). On my machine it roughly in the same time frame as yours - ~1min.
According to Flight Recorder:
- ~49% of the time is spent in reading the strings (city names). Almost all of it in the DataInputStream.readUTF/readFully methods.
- ~5% of the time is spent reading temperature (readShort)
- ~41% of the time is spent doing hashmap look-ups for computeIfAbsent()
- About 50GB of memory is allocated - %99.9 of it for Strings (and the wrapped byte[] array in them). This likely causes quite a bit of GC pressure.
[1] https://github.com/openjdk/jmc
edit: better link to JMC
It doesn't seem like it can be true that 90% of the time is spent in string parsing and hash lookups if the same operation takes 10% of the time when reading from a filechannel and bytebuffer.
var buffer = ByteBuffer.allocate(4096);
try (var fc = (FileChannel) Files.newByteChannel(tempFile,
StandardOpenOption.READ))
{
buffer.flip();
for (int i = 0; i < records; i++) {
if (buffer.remaining() < 32) {
buffer.compact();
fc.read(buffer);
buffer.flip();
}
int len = buffer.get();
byte[] cityBytes = new byte[len];
buffer.get(cityBytes);
String city = new String(cityBytes);
int temperature = buffer.getShort();
stats.computeIfAbsent(city, k -> new ResultsObserver())
.observe(temperature / 100.);
}
}I would guess at least some of the bottlenecks are in hardware, the operating system or in native code (including the JVM) in this case.
Please add https://github.com/apache/fury to the benchmark. It claims to be a drop-in replacement for the built-in serialization mechanism so it should be easy to try.