Hacker News Reader

Comment by detaro | Hacker News Reader

> - heaps beyond 32g can be beneficial but you’d have to go for 64g+, 32-48g is a dead zone

I'm curious why that is the case?

If the heap is under 32GB, the JVM can use "compressed oops" with 4 byte "pointers" to 8 byte offsets inside the 32GB heap instead of 8 byte pointers. This makes everything more compact. Since Java doesn't have structs, everything is a pointer and arrays of things are arrays of pointers. So a 48GB heap won't fit a lot more data than a 32GB heap.

See

https://shipilev.net/jvm/anatomy-quarks/23-compressed-refere...

https://wiki.openjdk.java.net/display/HotSpot/CompressedOops

Java compressed object pointers.

Shipilev (JVM performance engineer) explains it in great detail here: https://shipilev.net/jvm/anatomy-quarks/23-compressed-refere...

But the short of it: Java uses 32-bit numbers for object references if it can address whole heap with it. Given default 8-byte alignment of object we have 2^32 * 8 = 32g of addressable heap.

Once we are beyond that number 64-bit references are used and suddenly we are wasting a lot of heap space (Java is very much objects everywhere). Usually up to around 40% of heap is references.

> Usually up to around 40% of heap is references.

This blows my mind. I figured it would be non-trivial, but never expected that much.

Thanks, I somehow thought that was at smaller sizes but makes sense of course.

Si it’s better to have 2 nodes with 31gb each than one with 64gb?

See my other reply on big iron vs small instances (as in 31g of heap “small”).

In short there are many variables at play, but without any context generally I’d recommend trying many small instances first. Now depending on your hardware, scale and application things may easily change in favor of big iron.

well, I was thinking more in using more than one ES instances on the same machine listening on different ports, but your points on the other reply still apply.

I actually did that once - splitting one NUMA machine into 2 ES instances each isolated to its own socket (numactl etc).

Sadly at the time there was a big perf problem with indexingdocuments with 20kb+ binary blobs (non-indexed field) and so the change to 2 nodes vs one did have little noticeable effect on the throughput benchmarks I did with our data back then (~2015 ES 1.4) In the end I dropped this split and focused on the other bigger problems.

Would love to revisit this exercise with more of free time and better tools.

Reply on news.ycombinator.com