JVM Garbage Collectors Benchmarks Report | Hacker News Reader

JVM Garbage Collectors Benchmarks Report

ionutbalosin.com

119 pointsSindisil37 comments

Why isn't the benchmark code public ? For people to adopt/convinced, they should be able to run the benchmark themselves, and see a similar pattern in the numbers as well.

Some theories:

(a) laziness

(b) wanting to avoid discussions about whether the exact way something was written was fair to some GC or another

(d) the code doesn't matter since these microbenchmarks are not the actual application you are interested in

Seriously, if you are interested enough in this that you would be willing to run the benchmarks, you presumably have some JVM workload you care about. (Or you could just grab a standard benchmark suite.) If you are interested enough, you can run your workload with the different GC flags and see what they do for you, in your context. Being able to reproduce the author's microbenchmark numbers on your laptop would not tell you anything about how your workload of interest runs on the big servers (or whatever).

Has anyone experienced a significant performance improvement just by switching GCs? We have an Elasticsearch cluster with G1GC and I’ve been wondering if it’s worth the time to replace a small number of nodes with different GCs. We could swap out 10% of the G1GC nodes for ZGC nodes, for example.

Yes, we moved to G1 (before it was the default) and then on to Shenandoah in a stream processor doing a lot of in-memory processing, lots of small allocations. As we reached a certain threshold of in-memory objects, Parallel GC would choke up and we’d see GC pauses which could reach 1s every few seconds, totally unacceptable.

No doubt we could have spent time optimising this but moving to G1 brought this down to rarely exceeding 100ms, Shenandoah brought this down further to typically <10ms.

Yes, we had an app that would hold a large amount of data in memory as a cache, about 16GB out of a 24GB heap. Before G1GC we'd see brutal stop-the-world collection pauses. Switching to G1GC we'd get pauses usually under 100ms, occasionally around 200ms. Although if anything ever caused a full GC it would cause a 30+ second pause to process the whole heap.

Never got around to trying any of the newer GCs before it was rearchitected.

I know that the Shenandoah people have tested with Elasticsearch and seen great improvements.

https://d3i71xaburhd42.cloudfront.net/605efbc0fed86899dcbf87...

Christine Flood is pretty great and she has some wonderful talks on YouTube if you're interested in how Shenandoah works. The image above comes from one of her papers on GC (https://www.semanticscholar.org/paper/Shenandoah%3A-An-open-...) and shows order-of-magnitude improvements in pause times with average pause times 1/10th that of G1 and total pause times 1/27th that of G1.

Shenandoah and ZGC are also made for larger heaps and often datastore have larger heaps.

Thanks for that, I’ll definitely check out the paper! I don’t know much about the JVM, or GCs, but it’s always interesting to learn.

Performance is not precise enough. Throughput, latency, peak memory consumption. Those characteristics can also vary by workload, as TFA demonstrates.

As far as latency goes the new collectors do perform better in my experience, but you should try and measure.

By "latency" do you mean "how long an Elasticsearch query takes to return results"? Or do you mean "how long the GC pauses the application for"? I suppose they are related, to some extent, but it's probably the ES search query time we'd be interested in improving.

Latency is not a single number, it's a distribution. Your 99.9%tile latency will suffer when your GC has a 2s pause every 20 seconds, even when the mean latency is low.

But yes, they also affect mean latency.

GC latency, which is the commonly used term for the duration of GC interruption.

> The period of time during which the garbage collector is active [and threads are stopped] is referred to as its latency. [1]

This does of course affects the distribution of response latency. High GC latency causes more varying response times (though not necessarily higher average response time).

[1] https://docs.microsoft.com/en-us/dotnet/standard/garbage-col...

On CLR we moved our server from server GC to client GC. It reduced efficiency, but fixed the problem we were having with long GC pauses. So yes, and it depends what you want to do.

A good start but it’s really a brief survey and I wouldn’t call it actionable.

The greatest challenges for Java gc is when heap sizes are big and when data is turned over slower than pure transactional. In other words allocate and retain for a while and free, with a 32gb heap. That’s where gc tuning becomes a dark art.

The bottom line is allocation rapidly and keeping up with that is easy enough. Mix in hundreds of parallel and live app threads, large memory structures that live for minutes or hours, and add a transactional gc turnover.

I think go does so well because if you want a cache in go you get smart and use the c escape route.

For large data structures I sometimes move them off the Java heap using the NIO/ByteBuffer APIs, which keeps my normal working set smaller. Only works for certain use-cases though.

Java Unsafe can be great for moving things off heap too. And this is also the kinda place where pooling in Java can help more than it hurts. But all these solutions have a few sharp edges and aren't universally helpful.

If/when Valhalla lands it'll help soften a few of the rough edges here.

What's the performance of Go's GC like against the JVM's? Especially since both languages are about the same abstraction level and roughly the same speed (3x slower than C for general unoptimized code excluding startup time).

I don't know of any direct comparisons of JVM and Go GC efficiency.

However, the biggest difference is that there's only one Go GC, while there are several supported ones for Hotspot. With the exception of CMS, which is deprecated, those collectors occupy different niches, meaning there are times you'd use each of them.[0]

The Go collector is optimized for relative simplicity and low latency, and currently tolerates significant overhead to hit that target.

[0] Maybe Shenendoah and ZGC are close enough that they're just direct competitors? Perhaps they're even designed to supercede G1? I'm not quite sure about that, but even so, it would leave at least 2-3 options, depending on whether you include Serial/Epsilon as serious options.

Go's collector is similar to OpenJDK's deprecated (about-to-be-removed) CMS (applied to the entire heap, rather than just the old generation, as in OpenJDK). They're both concurrenct non-moving mark and sweep collectors.

It's fundamentally difficult to compare them, because the features of the language itself change the workload the GC faces in a typical program. For example, Java boxes almost all values, so can't store very much on the stack. That means that conventional Java programs (typically) generate many more short-lived heap allocations than Go programs.

For more, see the excellent blog post here: https://blog.golang.org/ismmkeynote

I haven't put real thought into this, but it seems that in principle, given enough understanding of the runtimes, you can create good benchmarks of the collector. You would have to find patterns that force it to allocate/collect similar amounts of data, but that's doable.

However, to your point, so far as the languages tend to allocate more or less data on the heap, that might change which benchmarks look good, the same way that you'll care about different automotive benchmarks if you want to tow a trailer or race.

It's also a little difficult to benchmark Go GC the way you can the JVM, because you can't set a maximum heap size in Go. So, if you create objects too fast you'll just run out of system memory.

I would wager that Java's GC is better just for the simple fact that it has a whole bunch of knobs you can turn based on your workload. I've investigate an issue with a very time Go program, and the performance of it was being harmed by the Garbage Collector.~600ns GC doing 2k GC/s per second- ~12% of its time was spent garbage collecting. The solution was to turn the GOGC to 37- this yielded the best results but only increased performance 5%.

My coworkers were saying how they had 100s of knobs they could turn and try to try and tune the GC to either reduce the number of pauses, or do one big one every 10 seconds. . Java's options are more then I wish to list here. Go has `GOGC=` and that's it.

As one to one performance, that would be hard to measure, but my guess is Java would win in the end.

I would guess the same. Go's GC is narrowly tuned towards server workloads with pretty low latency. That means batched workloads (such as the Go compiler itself!) is hurt by GC.

Here's a nice presentation transcript that summarizes the design decisions behind Go's GC: https://blog.golang.org/ismmkeynote And as you can see, it's basically entirely driven by low latency, but not necessarily high throughout.

Another point to consider besides GC is that Java at least has JIT which really benefits hot paths; Go does not.

“At least has a jit”, go is compiling to native code, Java is to a jvm bytecode instruction executed effectively by a interpreter, a jit is critical to good performance in the latter. In the former, ie. go you have control over memory layout and are already in machine code at execution the constructs of the language aka runtime are effectively a linked library instead of a virtual machine.

Golang prefer one simple way over playing with runtime knobs. That said the gc characteristics of go, are partly reactionary to java at that time (10yrs ago) unpredictable latency and pauses, and they optimized to address that also in response to early prod users. All gc languages have gc overhead from a throughput perspective 15-20% afaics

> I would wager that Java's GC is better just for the simple fact that it has a whole bunch of knobs you can turn based on your workload.

Tuning the GC is a challenge in and of itself. When you get so many knobs to turn, it can get hard to rationalise the overall impact of any particular one. It's part of why skilled DBAs are worth their weight in gold, for example (there was a fun paper & tool from CMU a couple of years ago, OtterTune https://github.com/cmu-db/ottertune, which tries to apply machine learning to what is almost a black-box optimisation problem)

This is an interesting article on the topic:

https://blog.plan99.net/modern-garbage-collection-part-2-1c8...

https://www.reddit.com/r/golang/comments/5j7phw/modern_garba...

> "Go: 67 ms max, 1062 pauses, 23.6 s total pause, 22 ms mean pause, 91 s total runtime

> Java, G1 GC, no tuning: 86 ms max, 65 pauses, 2.7 s total pause, 41 ms mean pause, 20 s total runtime"

What I got from this is that Go has faster GC pauses at the cost of throughput (i.e. more pauses resulting in higher total pause time).

The new JVM Shenandoah GC also makes this tradeoff by adding another layer of indirection ("Brooks forwarding pointer") which has a throughput performance cost but gives you really low GC pauses.

With the BurstHeapMemoryBenchmark, I wonder whether the effect is driven by the concurrency of G1/Shenendoah/ZGC, not greater efficiency.

If the code doing allocation is single threaded, then parallel/serial GCs could have stop the world pauses, while the other collectors would continue allocating while the GC runs.

That's a real effect. It means that sometimes you can improve the throughput of a process with a lot of sequential work by allocating extra-CPUs and using concurrent collectors. However, that's something of a niche case--more often, you choose between throughput and lower pause times.

I think the rock-paper-scissors dance between throughput, latency, and efficancy that a GC plays is a pretty facinating topic. It really brings out the our biases from the kinds of work we do.

If a service has reduced it's latency that only increases the throughput the overall system. CPU/Memory efficiency of a single process often much less important because there's often headroom and GC can help leverage underutilized that cpu/memory.

Do i read correctly that in the ConstantHeapMemoryOccupancyBenchmark, ZGC delivers two operations per second? Two? Per second?!

The test is just:

  void test(Blackhole blackhole) {
      blackhole.consume(new byte[_8_MB]);
  }

So it's not like each "operation" is actually a large number of elementary operations, either.

No, the test is more than that. The preallocated objects matter because they are pathologic cases for certain GC operations. Large allocations compound this further.

I guess CMS manages to perform fine here because it can flexibly resize the young generation and isn't region-based while G1 has special optimizations for what they call humongous allocations.

It's a test that checks how collectors perform under two combined deviations from the assumptions for which they're optimized (i.e. that object graphs have a high branching level that allows parallelization, short-lived objects are small on average)

Does anybody know the author of the article? Where does he get his content from? Is he making a living as a speaker or is/was he also a coder?

> Does anybody know the author of the article?

You know this article is on his website, don't you? His name is the domain name.

https://ionutbalosin.com/about-me/

> is/was he also a coder

Seems unlikely anyone would be writing an article about Java GC, including having coded up benchmarks, without being able to code.

I am just curious. In my 'likely' world, people doing this kind of work have a reputation or a visible history. A live q/a session on the internet could do wonders.

> have a reputation or a visible history

He does - it’s on his website.

Thats not enough for me. I write stuff all day on my or other websites. Who vouches for his work? The guy is secretful. One can not reproduce the benchmark. Why should i trust his results?

> Who vouches for his work?

It's a blog, not a peer-reviewed research paper. The guy is widely known in the JVM community. I'll vouch for his work if you want.

May have been added in the last 29 minutes, but the title of the page says:

Ionuț Baloșin

SOFTWARE ARCHITECT AND INDEPENDENT TECHNICAL TRAINER

Reply on news.ycombinator.com