GraalVM at Facebook
medium.com
medium.com
Disclaimer: You guessed it, I work at Oracle.
I don’t work at Oracle but have lots of customers who use their/your products and they all feel your sales staff and motion are best described as predatory.
It's a bit ridiculous when, being an Oracle employee and using Oracle products, there's only one member on our team with access to a product's support pages and you have to message him and ask for a PDF printout whenever you run into an obscure issue.
Yes, but that's one of the reasons Public Clouds are thriving.
Maybe you can answer this question I had below: https://news.ycombinator.com/item?id=27786051
If at any given second of the course of a month I have somewhere between 500 and 2000 vcore’s worth of AWS vms running with Java SE, how do I figure out how much I owe Oracle?
Amazon, a competitor to Oracle, publicly provides enough information to figure this out.
I don't know anything about Oracle. Maybe they're awful. But I've interacted with the folks on the GraalVM team on a semi-regular basis and they're all genuinely friendly, helpful people.
I can understand bringing up "Hey the parent company is no bueno in my book"
But I don't think it's necessarily fair to the humans behind the product, who don't embody the company itself and are (for the most part) just genuinely interested in working on this problem and building a product -- to bring that kind of criticism.
Imagine you got your dream job working on something you were passionate about, and you're just doing your honest-to-god best, but every time someone brings up your work the only thing you hear is "Hey, fuck your company."
=(
Just my $0.02 though
This should be a massive warning to anybody who cares about what they create: don't do it for an unethical company that will use your work to fuck over their customers.
Every time I get to the download page, and despite all the claims around the Community edition OS licenses, it just reminds me who is behind it.
I think the trick is that they might just drop their efforts behind the Community Edition anytime. The Open Source license its not a guarantee. Any Open Source team that would decide to pick up the engineering effort would have them coming around and claiming some kind of license infringement.
Really a shame, actually.
To be honest, I would be much more afraid of google touching an open source project for that matter - they have a much worse track record.
And sued a company for doing their own implementation of Java...
It was simply open up or die.
Google has thousands of open source projects, including the Chromium web browser, the Chromium operating system, the V8 Javascript engine that Node.js depends on, Kubernetes, Golang, the Android OSP, gRPC, Tensorflow, Authenticator (2FA), etc.
The track record you're apparently referring to, of commercial products being discontinued, is not really that relevant to this. The issue in that case is that they are services that users come to depend on, and then are forced to switch away from. There's not really a comparable scenario in open source. The worst case is that the original sponsor stops paying for new development. If the project is viable enough, forks and other sponsors are likely to arise.
The second thing there doesn't seem like a fair criticism of Oracle or any company really. Firstly any company can abandon stuff. There is no expectation that open sourcing something means funding it in perpetuity forever. Look at Google and Facebook, they abandon open source projects at a way higher rate than Oracle has ever done.
And secondly Oracle have actually been very steady hands on the tiller of Java. GraalVM is a very long term project. More than a decade old now. They have been funding long term research programmes like Valhalla. They're still maintaining JavaFX and other large libraries, even when they didn't become super popular. Their attention span is way longer and more stable than the average tech company.
Apache contains a patent grant.
> still maintaining JavaFX and other large libraries, even when they didn't become super popular
They kept JavaFX proprietary for years before opening it up, and they only did so reluctantly—after it was obvious that it was a failure.
There are also some potential patent encumbrances? I know less about that.
I also don't know for sure if IcedTea is still necessary to get a 100% open build of newer versions of Java (I'm still one of those incorrigible elements who's still on Java 8), but, at least historically, there was an appreciable chunk of the official OpenJDK distribution that was not, in actual fact, open.
It died the moment the OpenJDK was released 14 years ago? The issue wasn't the TCK itself, as far as I understand Sun just didn't want to license any Java implementation that could cut into its embedded market and dragged its feet when the community requested an out for the Apache licensed Harmony project. The TCK now seems to be available for any OpenJDK derived implementation[1] in case you want to use official Java trademarks for your implementation.
> I also don't know for sure if IcedTea is still necessary to get a 100% open build of newer versions of Java
I can't actually find anything about IcedTea for any recent OpenJDK, but its wiki page[2] seems to hint at the IcedTea patches were merged back into OpenJDK. Debian and other free software distros also seem fine with distributing current OpenJDK versions instead of reviving gcj to torment their users.
[1]https://en.wikipedia.org/wiki/Technology_Compatibility_Kit#T...
Sure, you wouldn't want to share the answers to the drivers license exam...are you saying that they should sell the answers instead? What the actual fuck?
Compare it more, perhaps, to housing codes. . . and imagine if the specification of the housing codes were covered by an (at least ostensibly) democratic process, but the actual housing inspectors all worked for one company, and that company also happened to be the biggest player in the construction business. So the inspector therefore had a clear incentive treat outside construction firms with less than complete fairness.
The whole marketing angle behind GraalVM is that it's a drop-in replacement of your JVM. If things go wrong down the road, what stops you from switching back to an OpenJDK distribution from a 3rd party vendor? Nowadays there are enough companies involved in maintaining OpenJDK and Hotspot... including some that aren't exactly friends with Oracle.
I work in a very slow moving domain where once we get customers, they tend to stick with a product unless you seriously screw them over. This has moved me to evaluating everything I do on at least a 10-year time horizon.
So in the case of GraalVM, what do I think the risk of Oracle wanting to charge me for every CPU in Amazon's data center is in the next 10 years at the worst possible time to have to drop real work and move everything to OpenJDK? I think it's unacceptably high. YMMV.
For GraalVM in particular there's no risk because the EE is basically just the open source CE version + more compiler optimizations. So it truly is a drop in replacement.
-- https://en.wikipedia.org/wiki/History_of_Linux
Also if the same morals apply to the remaining contributors of the Linux kernel, better not use it. There are a few other ones that seat at Oracle table, so to speak.
Your parent said: "[Oracle has] built every trap door that they can into it". Oracle doesn't control what goes into Linux; they can only propose, it's Linus who controls what goes in. So your claim that Linux suffers from a similar risk because Oracle et. al. contribute to it is invalid. Contribution != Control
There's a reason you have to give Oracle joint copyright ownership if you contribute to GraalVM. They want to be able to close the source and sell it.
What a completely disingenuous argument without any concrete facts. Also, you can’t close source something with a permissive dual license.
Oracle wanted to do it with MySQL, but their lack of copyright ownership for some parts of a GPL-ed MySQL codebase made it impossible. They fixed that mistake with their new contributor agreement, ensuring that they always have control over your code if you contribute.
$18 per processor per month to use GraalVM Enterprise.
--
x86 has a 0.5 core multiplier. So if you have a 16-core server, you need to purchase “8 processors”.
And before people start spewing Oracle hate, this is standard enterprise on-premise licensing practices used by Microsoft, IBM, SAP etc.
https://www.oracle.com/ae/a/ocom/docs/graalvm-price-list.pdf
https://www.oracle.com/us/corporate/contracts/processor-core...
I don't see why we can't be annoyed at all the large enterprises with their "negotiate your own per-core price and commitment" sales.
Ed: ah, no. See sibling comment
Worse, if your VM platform supports migration (vMotion et al), they will then insist that you buy licenses for every core the workload could theoretically run on. For modern hypervisors, this often times means licensing every core in your virtualization estate. I've had Oracle licensing tell me this exact thing with a straight face.
There are more than one law firms in the US which do nothing but sue Oracle on their client's behalf. When your licensing terms are so odious that multiple law firms can make a living fighting your terms.... something isn't right.
"One Rich Asshole Called Larry Ellison"
See https://www.oracle.com/assets/cloud-licensing-070579.pdf
—-
> “ Amazon EC2 and RDS - count two vCPUs as equivalent to one Oracle Processor license if hyper-threading is enabled, and one vCPU as equivalent to one Oracle Processor license if hyper-threading is not enabled. Microsoft Azure – count two vCPUs as equivalent to one Oracle Processor license if hyper- threading is enabled, and one vCPU as equivalent to one Oracle Processor license if hyper- threading is not enabled. When counting Oracle Processor license requirements in Authorized Cloud Environments, the Oracle Processor Core Factor Table is not applicable.”
The key is that last sentence.
We've had the case where they threathened to double per-core licensing cost of their products just because our developers wanted to use PostgreSQL in their new project. Once they get into your system they'll bleed your time and money with their licensing bullshit. You have better things to do with your company time.
> This policy applies to cloud computing environments from the following vendors: Amazon Web Services –Amazon Elastic Compute Cloud (EC2), Amazon Relational Database Service (RDS) and Microsoft Azure Platform (collectively, the ‘Authorized Cloud Environments’).This policy applies to these Oracle programs.
So, only for AWS and Azure.
But the price list is here, it seems:
https://www.oracle.com/us/corporate/pricing/price-lists/java...
So price can vary between $25/processor/month and $12.50/month with further discounts available if you negotiate, probably. But ... this has to be rounded up to whatever the physical processor has, then multiplied by some "core factor" which can actually reduce the number of cores you need to buy from the physical number. Which can be found here:
https://www.oracle.com/us/corporate/contracts/processor-core...
For Intel and AMD chips, the factor is 0.5 - so the actual prices are just half of those numbers!
The rules about virtualization are kinda weird especially because Oracle Cloud presumably is virtualized itself. Presumably it's because if your jobs can "flex" at will then it becomes impossible to say how many cores you're actually using. Like, what do you do, integrate the area under the curve of second-by-second CPU usage or something? That would be very complicated. If they say, OK, your workload can use up to 100 cores then you pay for 100 cores, it's very simple. All you have to agree on with them is how many cores the maximum amount is. I can't imagine it would matter if the physical underlying hardware changes as long as the workload can only run on one machine at once. Otherwise it'd be impossible to run any Oracle product in any cloud at all, but people do it!
You only get them when lacking negotiation skills, or failing to understand the whole political process from enterprise sales.
As a lighthouse client, FB is pretty amazing in this space.
https://aws.amazon.com/ec2/dedicated-hosts/pricing/
Wow! An M5 costs $2300 a month, which is almost three times as much as Graal licensing would cost, assuming you paid for all 48 cores ($864). Frankly, if a minimum wage job is a larger revenue stream than your "app", then you should probably do the minimum wage job instead. There's also a community edition. It's free. I think the moral outrage about anything Oracle-related is a hollow fashion statement, like a Che Guevara t-shirt.
There's no moral outrage over here either, just "I dont want to do business with assholes who keep fucking everyone over."
...no, you're still reading it wrong. It's not the number of substrate CPUs on the compute node your workload is running on. It's the number of substrate CPUs on all the compute nodes your workload could potentially ever be re-scheduled or migrated onto. I.e., all the CPUs of every single (non-reserved) compute node in the datacenter. (Or, presuming certain service-transitioning technologies, multiple datacenters!) We're talking about e.g. "the entirety of AWS EC2 us-west-2." Hundreds of thousands of machines. Millions of CPUs.
By analogy: imagine there's a car-sharing service, that rents out its fleet of cars to people — but because, when you rent a car, you could theoretically end up driving any car in the rental company's fleet — they set their premium to be $18/mo * [the size of the rental company's entire rental-car fleet.] Because in theory, if you set things up just right, you could manage to drive all their cars, in one day. And so they want to charge you, at all times, for that basically-impossible worst-case upper-bound price.
Do you see how insane that sounds?
Or, as the op suggested, their pricing is just straight up insane
If you set up your own virtualisation environment, then you're subject to those rules. However, you can work around it by separating out the machines that runs Oracle into its own groups of physical machines.
It does mean that deployment of Oracle software is much more complicated, since you can't use the regular environment. You also need it to be as separate as possible unless you want to have a long discussion with Oracle's lawyers about how to interpret those clauses.
If you want to rebut those facts, go ahead, but don’t axiomatically accept the facts (by not arguing with the GGGP commentor directly, now that you understand the point that they were making) and then try to deny their implications.
[0] https://openjdk.java.net/jeps/410 [1] https://mail.openjdk.java.net/pipermail/metropolis-dev/2021-...
So you can continue to use Graal as your last tier compiler or to compile Truffle languages on OpenJDK.
This isn't true - you can use Graal from a JAR just fine.
I remember a contrat with Kong API Gateway , they offered generous deal if we agreed to sign an agreement to attend their conference to promote their product.
Very common in the industry to have those terms "legally binded".
https://jeeconf.com/program/performance-tuning-twitter-servi...
https://docs.spring.io/spring-native/docs/current/reference/...
We open-sourced some babashka code at https://github.com/staticweb-io/staticweb-open-wp/tree/maste... One major caveat: when I wrote that code, babashka didn't have any MySQL support, so I shelled out to the MySQL CLI. Later, I figured out how to compile the MySQL JDBC drivers with native-image and it's now available at https://github.com/babashka/babashka-sql-pods along with HSQLDB, SQL Server, Oracle, and Postgres drivers.
I wonder if FB bought the Oracle enterprise license or is using the community edition version.
What happened to writing C/C++ in the first place if speed matters? Would you write Stockfish in Java or Python and hope for a sophisticated, bug free VM?
And Scala is, for both better and worse, an extremely compact language too. I would expect about 5x more lines of code just from that.
I wouldn't necessarily just assume that it's 5x more code. I've spent a fair bit of time digging around in the Spark code, and it's not clear to me that it's as efficiently designed or cleanly implemented as one might like. And, while I can't speak to the influence of Google's style guide, I don't personally feel like modern C++ is 5x as verbose as modern Scala. A bit more, maybe? But it's not like Scala (2, at least) doesn't have its own bizarre sources of unnecessary verbosity. It mostly looks great by comparison to Java.
As much as we would like the modern C++ from books and conference talks to exist in real life, that is not how the large majority of the world actually writes C++.
Heck, just check the AOSP source code to see how much modern C++ it has, from one of the companies that contributes to ISO C++.
Or if that fails to impress, see how much modern C++ exists in Microsoft Github sample projects.
Naturally it is possible to write static classes that hide the internals of a data structure, and ensure they are properly used.
And if that isn't enough, Valhalla is eventually here, Java 17 already includes the preview for new native memory primitives, and polyglot developers can always write a bunch of native methods, no need to throw safety and productivity away for 100% of the code.
> Naturally it is possible to write static classes that hide the internals of a data structure
Which limits you to reusing a single instance which is not only very limited in what you can do with, but also inherently error-prone compared to passing structs.
Valhalla does not exist yet, had been in development for a decade, and even once they release it, it won't be equivalent to C++ structs - it is by design much more limited.
Finally, productivity of Java is overrated, I've been writing in many languages and I'm not more productive in Java than in C++. Productivity is mostly a trait of a developer and libraries, not a language.
I know C++ since 1993, have delivered C++ into production at CERN and Nokia Networks, and also taught it at the university to first year students.
So what do I know a bit about C++ productivity versus other languages.
Now if we start talking about GPGPU programming, HPC or writing device drivers then it is another matter, there is where C++ shines and will keep doing so for the foreseeable future.
CERN is definitely not going to rewrite HLT in Java, in fact for some units not even C++ was good enough, and the code is actually implemented in FPGAs.
Java's garbage collector, although the source of many performance headaches, is in fact very sophisticated and "does the right thing" for most OOP-style code. Yes, it takes some understanding and tuning (and yes, there are multiple garbage collectors you're supposed to test out as you tune your software...) but that's the sort of thing that's available.
In C/C++ world: you can't just swap out versions of "malloc" or "new" as easily (ie: without recompiling everything, to tune / test your code on different systems). Restarting your JVM with slightly different garbage collector settings over-and-over again is absolutely a thing in the Java world to help tune your applications to the machines they run on.
---------
Java's high performance concurrency libraries are very well written as well. And there are some real speed demons (Azul) out there.
Its really a different world and model than the C/C++ world.
LD_PRELOAD=libjemalloc.so MALLOC_CONF=narenas:1 /your/unmodified/appYes, it's easier with modern c++, but it's still significantly harder than with, say, rust or ada (or java for that matter, which is not slow, contrary to popular belief).
Furthermore, in the Facebook case, it's about cost : you save money by being faster, but you don't want to lose money because you're tracking overflow bugs in c++. If you can have both execution speed and development/maintenance speed , it's much better.
I'd be willing to bet that lots of the core ranking/ads stuff at most of the major tech companies is written in C++.
When we think C++ performance, we're thinking of something like a video or audio compression i.e. discrete functionality. Note a massive, enterprise level solution with myriads of kinds of business functions.
It would never get complete in C++ as that language has complexity issues which overwhelm everything else at some scale, more often than not.
If there are performance issues with a Java application, where after all attempts, there is still some room left, there is no need to throw everything way.
Those areas can be ported to C, C++, Rust, whatever, and then called via native methods.
There's also no reason to believe that C or C++ will be faster, because you have to 1) know how to write performant C++ code (not obvious) and 2) content with the Java-JNI-C barrier which is non-negligible.
I don't see too many scenarios where taking a module and porting to C++ is really ever a thing frankly.
Usually, it'd be for integrating already existing systems - or - for integrating things which might naturally be written in C/C++ in the first place.
For something like Graal, it's one of those broad improvements wherein everyone gains, hopefully magically without having to do anything like a big improvement in V8 that just 'happens'.
On managed languages FFI should never be done 1:1, as you very well point out it isn't negligible.
Rather they should be thought as in-process RPC, that actually do a bunch of work per call.
It is still faster than doing IPC across processes.
Right now their only use for green field projects is on ecosystems where there are no alternatives.
Unfortunate, because I'd love to build some prototypes based on graal features, but Oracle's apparent disinterest in supporting community contributions, and the questions around licensing of parts of the graal ecosystem leaves me worried enough to avoid it for now.
This is about the former.
Use a zero-based chart, and show a longer time period. Then the gap is clear, even though not as dramatic.
Also, the article probably shouldn't talk about the general benchmark, and probably better to omit the performance of the Enterprise edition, because Facebook is probably not using it. (It's not clear from the article which version of GraalVM FB uses, but I don't think they pay for it.)
We all know 10% CPU reduction is superb, especially on the Facebook scale. And we also know GraalVM is great technology. So please don't try to fool the readers.
By the way, it also mentions that they moved Presto to GraalVM and saw an improvement. It can be as significant, and would be nice to show the number if there is a latency improvement because Presto is interactive tool and the improvement can end up productivity gain, in addition to saving the CPU cycles.
unless the point of this post is to attempt to sell the Enterprise edition to FB
Agreed, but that visualization is even worse because it lacks a y-axis scale entirely.
(Think about user growth as a particularly good example of a variable like this).
To be honest, this plot is pretty useless either way; who cares about the trend it took downward since we have little or no context on what happened at any given point?
Why would I think this and not that Oracle would want to make money off their unique IP?
Oracle Java is best understood as two competing teams at this point. In theory GraalVM started out as a research project to feed improvements back to HotSpot and some of that they've done, like with the JVMCI interface that was added. But it didn't work out that way for whatever reason and GraalVM is now effectively a competitor to the 'classical' Sun Java product guys. They have different brand names, commercial models, feature priorities, price points, they are different organisations within the company. Fortunately GraalVM is not a fork of OpenJDK - it's a bunch of plugins and extra components, more or less. But at some level the difference is semantic because if you swap out the whole JIT compiler then maybe it actually is a fork? Not sure.
Anyway the original research vision was that C2 would be replaced by Graal indeed. The politics of merging such a total rewrite would be difficult at the best of times and it seems Oracle hasn't managed it, or perhaps they like having some internal competition for the OpenJDK guys. Certainly Graal team cares a lot more about modern marketing, about getting their tech adopted into other Oracle products and things that presumably Oracle management care about. Like the graalvm.org website vs the OpenJDK website is no competition. The former feels like an actual product, the latter is a bunch of wiki pages more or less.
The information you are trying to convey is the variability of the change over time.
0% improvement (1.0) is 5/8th of the length of the Y axis, making a 42% improvement look barely interesting. Probably should have opted for a different chart here.