JVM Performance Comparison for JDK 21 | Hacker News Reader

JVM Performance Comparison for JDK 21

ionutbalosin.com

105 pointsmfiguiere32 comments

I often wonder if there should be a suite of tests that deliberately dont use well-tuned algorithms, or optimal ones, as the basis for benchmarks.

In the real world, people dont choose optimal paths in code very often. Or what was once optimal becomes suboptimal because the context changes.

Supposing that a benchmark chose to do, say, bubblesort on an insanely large list. Would that lead to more insight around how the compiler would act in the real world? Or are the pitfalls so well understood that they are universally optimized away by the compiler's earliest stages or processing?

If I were selling, say, Azul or some other alternative JDK, I might make a point of noting that it does a superior job of dealing with the code as it exists today, rather than as one might wish it to be.

I like to use naive recursive fibonacci to bootstrap performance testing of my interpreters. It's a pretty lousy algorithm, but it does a good job of stress testing instruction dispatch and function calls.

Anecdotally I've seen much bigger gains on my team's Spark jobs that use complex business logic (i.e. our own hand rolled algorithms in idiomatic Scala) compared to plain ETL jobs that use the vanilla Spark API, which is now very mature and optimized.

GraalVM does a really impressive job with mediocre code :-)

It would be useful to see memory utilization while performing the benchmarks. Historically GraalVM used ALOT of memory.

I never really understood why the Oracle/Enterprise version of GraalVM has better performance (also advertised) compared to the CE version. Is there any fundamental difference?

Yes, it is similar to C and C++ commercial compilers with different prices levels, written as modular compilers where the set of available optimization passes, among other features isn't available.

For example, it used to be so that stuff like OpenMP was only available in professional licenses for C, C++ and Fortran compilers.

"Oracle GraalVM" (the former "Enterprise Edition") is now free, but (unlike "GraalVM Community Edition") not open source.

It is free to try. But may not be free for commercial use. Some random blog post says check it with Oracle if you want to sell your products/services with it.

"The GFTC is the license for GraalVM for JDK 21, GraalVM for JDK 17, GraalVM for JDK 20 and later releases. Subject to the conditions of the license, it permits free use for all users – even commercial and production use."

"What is the new “GraalVM Free Terms and Conditions (GFTC) including License for Early Adopter Versions” License?"

https://www.oracle.com/java/technologies/javase/jdk-faqs.htm...

Seems like there's a different license for GraalVM 22.

Diffing the licenses shows that the only change was to add clarity around graalvm as opposed to a generic program:

"For clarity, those portions of a Program included in otherwise unmodified software that is produced as output resulting from running the unmodified Program (such as may be produced by use of GraalVM Native Image) shall be deemed to be an unmodified Program for the purposes of this license."

Other than that, it appears the same. The thing I'm wondering about is how this matches with everyone saying it's "free to use" when the following is in both versions of the license (underscore _emphasis_ mine):

"(b) redistribute the unmodified Program and Program Documentation, under the terms of this License, provided that __You do not charge Your licensees any fees associated with such distribution or use of the Program__, including, without limitation, fees for products that include or are bundled with a copy of the Program or for services that involve the use of the distributed Program."

This to me reads as if:

- You cannot charge for licensing fees on graalvm built distributed pieces of software (licensing for a desktop application, cli program, game, etc)

- You cannot charge for licensing fees on services that involve the use of graalvm built software (SaaS software that uses the graalvm built binary in a k8s container, lambad, etc as part of the service topology)

My gut reaction (since this is Oracle) is that this was an ambiguity oversight in the language that is now being corrected to lock down this "free" usage of the premium version, but I may be overly pessimistic.

Is anyone using this in prod based on it being "free" and has any info from their legal team or otherwise?

Mostly looking to be corrected in my likely misinterpretation.

Market segmentation to justify charging companies for the the compiler with more optimizations. If its not your money and can convince your boss you need it...

The primary performance benefits with Enterprise are access to the G1 garbage collector and profile-guided optimization

? both of these things are available in openjdk distributions

Apparently those are not available for native executibles produced by graalvm community edition.

Profile guided optimisation for native executibles sounds cool. Too bad Oracle wants to keep it closed for monetisation.

PGO (both static and dynamic) is something that is offered to you in the FOSS form in .NET land.

Meanwhile .NET land requires writing reflection free specific code for AOT, while GraalVM can handle reflection just fine.

ART also does PGO for free, as does OpenJ9. Both of which also handle reflection.

The thing with Java ecosystem is choice of implementations, instead of Microsoft's single implementation, specially now since most of Xamarin's work has been consumed into modern .NET.

This is simply not true. AOT requires code that is free of uses of JIT-specific features like runtime reflection-emit or assembly loading, which is not possible without build instrumentation for AOT implementations on other platforms either[0], because it by definition needs a JIT compiler. Reflection itself not only works but is sometimes faster because closed world guarantee by AOT allows for a more streamlined implementation.

With that said, trimming, also known as tree-shaking, which is required to not have massive binary size, has to know at compile-time which types, members and metadata tokens are reachable. This means that certain patterns of unbound reflection are by definition not statically analyzable and require the user to annotate (with [RequiresUnreferencedCode(params)]) the corresponding types with details of what they need the trimmer/linker to preserve (the entire assembly can also be rooted but, well, that's bad for size).

Other features like compiled expressions have received proper interpreter fallback which makes them just work(tm) without developer intervention.

You can also see that GraalVM Native Image is subject to very similar restrictions[1].

[0] Like pre-generating generic types/method bodies for Unity with special tooling. NativeAOT currently has (planned to be deprecated) rd.xml to manually instantiate certain generic methods and root specific types, replace their bodies, etc., it will probably be fully superseded by further expansion of attributes-style annotations API or obsoleted by improving compile-time analysis in the future.

[1] https://www.graalvm.org/22.1/reference-manual/native-image/L...

p.s.: quick search shows that Java Swing is plagued by similar issues with Native Image (not mentioning having to generate metadata annotations) to making .NET's GUI frameworks work with NativeAOT save for certain blessed templates/workloads introduced in .NET 8. Java AWT appears to be faring somewhat better on hello world examples but I suspect it experience the same fate as the templates for WinForms that the community has come up with so far (let me know if that's not the case).

The difference is that GraalVM, OpenJ9 and ART, have the capability to feed PGO data into the AOT toolchain, as a means to automatically generate a rd.xml like archive, thus preserving the existing language ecosystem.

In ART's case that metadata is even shared across devices via PlayStore, which gathers and uploads it, packaging it for download along the APK for new installations. This metada is dynamic, as more devices take party into executing newer code paths.

This is the beauty of Java ecosystem versus .NET, the plethora of implementations, with different approaches solving the same set of problems, allowing us in many cases to pick and choose.

From Real Time GC with AOT compilation used by US and French military in embedded systems, factory automation robots, phones, infotainment systems, M2M gateways, runtime snapshots between executions, or crazy ideas like having PGO data from cloud based JITs sharing executable code.

I feel like I keep reading the same comment from your profile every time except the order of words changes slightly. So much for expecting a meaningful response.

(please, please read the documentation first before making confident assertions, like in this case, and at least google discussed topics to actualize the knowledge on the points in favour of which you're arguing)

The sentiment goes both ways, the big difference is that I am not invested in any specific ecosystem to let that influence my arguments.

This is great work, although benchmarks that do not explicitly include analyses of each JVM’s GC options and their behaviors don’t help me much in practice. I’m not doing scientific computing; I need to know what kind of GC pauses I get and how often.

GC pauses are dependent on the code and its workload so general benchmarking is rarely useful. You can have a code that works really nicely with GC at a given throughput and request pattern, but not at all at a different load profile. Minor implementation change can also dramatically alter the GC workload. A generic work-queue, where one thread is enqueuing tasks (allocating memory) and another is consuming them and deallocating memory can make it hard for GC to do its work, while minor changes so that queue is fixed-sized and does not allocate can dramatically change the GC workload.

Even really simple optimizations where the `enqueue` may just copy to its queue-owned buffers can make allocations happen exclusively from TLAB with bump allocation which are essentially no-ops for GC & the allocator. Generic benchmarking happens but is more the domain of GC developers, and not very useful for developers – because usually, your workload is different from mine.

> Each benchmark uses 5x10s warm-up iterations…

How do I read “5x10s”? 10^5?

5 warm-up iterations of 10 seconds each.

The Oracle GraalVM JIT compiler performs the same optimization as the Oracle GraalVM JIT compiler.

Makes sense.

You did notice the "CE" in one of them? Community Edition. Evidently a dumbed-down variant so that there is some incentive in buying the better one. I have they give up on that at some point.

No I did not, there is no CE to be found in that sentence and I grabbed straight off the page.

"The Oracle GraalVM JIT compiler performs the same optimization as the Oracle GraalVM JIT compiler."

whew lucky

I think there is a CE somewhere missing :)

TL;DR: real-world performance between the three will be highly dependent on just how bad your code is in different metrics. Also: the CE version shouldn’t be considered representative of the non-CE version performance.

Java, still not debt.

Reply on news.ycombinator.com