Mystery meat OpenJDK builds strike again | Hacker News Reader

Mystery meat OpenJDK builds strike again

mail.openjdk.java.net

261 pointshocuspocus49 comments

A few emails later in the thread[0], it is noted...

> ...who actually maintains the "official" Docker images for 8u and 11u? Github history for Dockerfiles points to no one I recognize from around OpenJDK.

That, I think, is the main problem.

[0]: https://mail.openjdk.java.net/pipermail/jdk8u-dev/2019-May/0...

Edit: Actually linking to source (which I should've done from the beginning…)

Jikes

Yeah I’ve had to switch away from using these docker images, too many issues. They also stopped shipping openjdk12 because Debian stopped shipping them: which raises the question, why do the official openjdk Docker images use Debian’s version at all, rather than the vanilla openjdk builds?

Main problem or not, having the "release" version number on anything other than an actual release version is begging for problems.

As an observer, not any kind of authority:

The release methodology of OpenJDK from Oracle is obscure at best, and it is hard to imagine how package managers are supposed to cope. I notice both of these examples come with Debian version strings attached, and they likely build from tags in the version control repository. As example, in these two links:

https://hg.openjdk.java.net/jdk8u/jdk8u/tags

https://hg.openjdk.java.net/jdk-updates/jdk11u/tags

The "-ga" tags do pretty clearly delineate official releases, but they also get pushed to the public repository well (often weeks) after Oracle posts official binaries for these releases. It is confusing and makes it not all that mysterious that distribution packaging will take the best that they can see for doing a clean build, as they like to do.

It would help quite a lot if the +1/+2/+3/etc releases were instead delineated with "-alpha1", "-beta1", "-rc1" tags. Much clearer that they aren't meant to be production.

I think this has lees to do with the OpenJDK team and more to do with the part of the Docker team and/or community that maintain this repo and not pulling correct release versions.

I knew someone who worked at Gradle and noticed how RedHat did all their own gradle packaging that broke everything they had in their offical rpms and spec files.

Some projects like Elastic Search maintain their own v2 repos on their own domain (all there's are mirrored on Docker hub but their docs point to their own repo) and other projects have official Docker hub accounts (like Jenkins).

I'm guessing the right solution might be Docker hub working with the OpenJDK guys so they maintain their own organization on Docker hub and integrate building and pushing to it from their own CI.

It's an age old problem. Upstreams were complaining (legitimately) about how Linux distributions would (re)package their software and break it along the way 20 years ago. Broken packages in Docker being pulled from Debian is no surprise, it was ever thus.

Note that Oracle has nothing to do with these releases or builds. Red Hat is now the maintainer of OpenJDK 8u and 11u release families. But this isn’t Red Hat’s fault either.

By default, the OpenJDK builds put something obvious like “-internal” into the version string, and you have to override it explicitly if you are intending to produce an official build. Whoever made these builds overrode the version string to make them look like official releases, when in fact they don’t correspond to official release tags in the source.

In other cases there may have been a delay between the appearance of an official binary and the appearance of the release tag in the source tree, but that’s not the issue here.

This is indeed not the first time something like this happens and a good reason to get your openjdk builds from someone that has licensed the test suites and supports their builds.

I remember having weird issues a few years ago with builds for both Centos and Debian/Ubuntu where they first took ages to get release builds out and then made it worse by basing their builds of pre-release versions rather than actual stable releases. This was several months into the JDK 8 cycle. In their defense, JDK 8 had some nasty bugs early on. But still, a lot of this seemed accidental to me and not intentional.

You can get decent openjdk builds from Amazon, Azul and Red Hat. They each put in quite a bit of effort to ensure these builds are solid and tested and they each offer these builds for free they also each offer Docker images with their builds and make it very easy to use them. I've been using Azul docker images for quite some time.

Amazon particularly seems to have made an effort with their openjdk builds. If you read their commit log: https://github.com/corretto/corretto-8/commits/develop, you'll see a few backports of fixes from openjdk 9 and lots of efforts to get the packaging right. The last bug fix they added was from just a few months ago. I'm currently considering switching to this.

BTW. Oracle seems to have stepped away from this and only offers limited time support for their own JDKs which also cost money. So, not including them in this list on purpose. Unless you must use them (for legal/business reasons) and fully understand the legal situation that implies, you should avoid using them.

I'm thinking of moving us to the Azul images. I'm not actually that concerned about this myself, especially with the attention this is receiving, but I know I'm going to get my ear bent about it for the next few weeks.

https://hub.docker.com/r/azul/zulu-openjdk-alpine

> Oracle seems to have stepped away from this and only offers limited time support for their own JDKs which also cost money.

This is inaccurate (I work at Oracle on OpenJDK). Oracle provides the JDK under two different licenses: the commercial OTN, for support subscribers, and the free OpenJDK license (here: http://jdk.java.net/). Moreover, Oracle's OpenJDK builds are the official ones in the sense that bugs reported against OpenJDK must be reported against those (as those are the builds that the developers use). For 8u and 11u, the similarly official builds are these ones by Red Hat: https://adoptopenjdk.net/upstream

The thing about support is misleading becuase Java no longer has major releases (9 was the last one ever), and now has two upgrade paths: a gradual one: 9->10->11->12->... and a non-gradual one: 11->11.0.1->11.0.2.->...11.0.x->17 (neither of these is the same as the old model). The difference between upgrading from 11.0.2 to 11.0.3 and 11.0.2 and 12 can be very small (12 is not a major version), the gradual path is the recommended one as it will be the cheapest overall, and OpenJDK development is built around the gradual path. As far as OpenJDK builds go, Oracle supports only one of those paths for free (the recommended gradual one), while the other is for subscribers only. Note that in the recommended upgrade path, you will be supported by Oracle builds forever at no cost, and will never have to do another major upgrade again. So all the bug fixes and security patches for 11.0.2 are provided by Oracle, completely for free, in 12, which is not a major release.

As to the paid support path, this is how Oracle funds OpenJDK development. Oracle is the company that contributes >90% of the resources to OpenJDK (Here are the recent contribution stats for 11 and 12: [1][2]. Note that the stats do not consider contribution size.) So it's important to remember that all OpenJDK JDKs are developed primarily by Oracle, and it is Oracle's support subscription that funds OpenJDK development. And if, for example, you find a bug in Amazon Coretto, they will reproduce it on the Oracle (or Red Hat, depending on version) builds and submit it to the OpenJDK project for Oracle (or sometimes Red Hat) developers to fix.

[1]: https://blogs.oracle.com/java-platform-group/building-jdk-11...

[2]: https://blogs.oracle.com/java-platform-group/the-arrival-of-...

That's the Oracle pitch. The practical implication is that you don't get your openjdk builds from them if you want to avoid the audits, licensing, etc. Paid support is also offered by others so they don't do this exclusively. This has been true since they acquired Sun. I find this odd because that was arguably the most valuable thing they acquired from them.

BTW, v9 was a non LTS release, just like 10. v8 was the last traditional release. V11 is a relatively short lived LTS release. Hence Amazon and others stepping up to fill the void left by Oracle no longer offering long term support for free.

> That's the Oracle pitch.

Well, like the vast majority of OpenJDK developers, I work at Oracle :) (but I speak only for myself)

> The practical implication is that you don't get your openjdk builds from them if you want to avoid the audits, licensing, etc.

The OpenJDK builds from Oracle (http://jdk.java.net/) -- the "official" OpenJDK builds for the current JDK version -- are under the same license as any other OpenJDK distribution, and the download page for Oracle customers clearly says download our OpenJDK builds if you're not a support customer.

> the void left by Oracle no longer offering long term support for free.

The "no longer" here is misleading. There are two completely new upgrade models now. Oracle offers one of them (the recommended gradual path) completely free, and the other (the non-gradual path) as a subscription. Neither of those is the same as before. For example, JDK 11 is a feature release marked LTS. It is about the same size as 7u4 or 8u20. Oracle never offered LTS for 8u20 or 7u4; it was always only 6 months. In the past we had major releases that were supported with "limited update" releases and patch releases. Those major releases no longer exist, so it doesn't make sense to say that Oracle no longer offers something which is now irrelevant. On the other hand, the LTS releases now include only patches, whereas the feature releases are like "limited update +".

So Oracle no longer offers free LTS (in the form of limited update releases and patches) for major releases, and no one else does, either, because major releases no longer exist.

Pretty much every enterprise vendor has an OpenJDK build. I work for Pivotal. We've been building and shipping the OpenJDK from source for years (mostly Java buildpack, but we use it for anything that we ship that requires a JVM). Like our peers at other enterprise software vendors we spent a lot of engineering making the process smooth and traceable.

I think more of the industry will move towards AdoptOpenJDK builds. There's a lot of value in combining forces to produce the same byte-identical binaries for everyone.

Jetbrains do as well, they store theirs on bintray.

It's all a bit of a mess tbh and particulary frustrating from the outside where you aren't a full time java dev.

Not completely related, but this is one of the reasons why I prefer to add commits bumping the version number when a release is made instead of before–it means that any builds made from source don't have version number they shouldn't.

It took me a long time to understand why Maven uses version strings like 4.2.1-SNAPSHOT but this is exactly why it's useful.

(The standard practice is to strip the "-SNAPSHOT" off for the release commit, then stick it back on for the next commit, but with a bumped minor version. Both commits are then pushed at once, so stuff built locally always is marked as a SNAPSHOT.)

The problem here is that Debian's OpenJDK maintainer removed the "work in progress" indicator string manually. No approach protects against this.

The AdoptOpenJDK community (a consortium of vendors and users that produce business ready OpenJDK binaries) will work with the Debian maintainers to course correct this going forwards. If you’d like to get involved see https://AdoptOpenJDK.net

I'm still astounded how much of an (intentional?) mess Oracle have made of Java and OpenJDK. If you search for Java you get to a Oracle page with a big download button and a cryptic yellow warning about license changes, with a little link at the bottom directing people to OpenJDK. The official OpenJDK builds they send people to have no Windows installer - they're just a ZIP of the JDK. There is no way for an ordinary Windows user searching the internet for a way to install Java on their work PC to end up with a working Java install that doesn't leave their employer on the hook for Oracle licensing fees. AdoptOpenJDK have working builds with a proper Windows installer, but unless you already know about them somehow (for instance, by reading HN regularly) there's no way to find them.

The changes to the process were absolutely necessary for Java to keep growing and remain competitive, but the ecosystem will take a few years to adapt (and I don't think the warning is cryptic).

As to an installer -- the JDK does not require any installation (just point JAVA_HOME at it). In addition, the JDK is for developers only. The days of the "system desktop JRE" are over, and it is against the recommended practice to have the JDK on non-developer machines. Instead, the recommended practice is to have the developer create a small custom runtime with jlink and bundle it with their application (or perhaps a runtime shared for all of the developer's applications, in case the users are all in the same company).

Yes, the current practice is such because Oracle wants to tax all creators of java applications. Java is not run anywhere anymore.

The JDK is for "developers only" because it is open and java is not. Oracle cannot tax and legally threaten an open system the same way it can one where you need a license to actually run java.

I don't know what you're talking about. Oracle has recently, for the first time ever, open sourced the entire JDK. It is 100% open source. No one is threatened, and the license is the same one as that used by Linux.

Just to be clear: Java is a standard; OpenJDK is the name of Oracle's implementation of the Java standard.

That the JDK is for developers who then build their own custom runtimes using the JDK tools is a result of shifts in software distribution -- on the desktop, the shift to app stores, and on the server, the shift to containers -- that made the "desktop JRE" irrelevant (and unpopular). But if you prefer that old model, the JDK contains all you need to create such a runtime easily.

The current thinking is that end users are not supposed to install Java - it will be bundled with every application. That kind of solves the problem, but I guess we'll have to forget about writing small utility programs in Java.

Also, complicating things a bit if you want a traditional shared Java installation, AdoptOpenJDK has apparently been refused a license for the test suite, making it not quite properly offical Java.

> but I guess we'll have to forget about writing small utility programs in Java.

Developers will continue using the JDK (for which no installation is necessary; just setting JAVA_HOME). A tool suite intended for users will share the same runtime. And when it comes to small, user-oriented, standalone utilities, Graal's native image makes that more compelling than ever.

Ubuntu Java packages are a joke as well and I stopped using them when I was having issues with the OpenJDK-11 package from apt. It turned out the the package wasn't packaging Java 11, but Java 10 instead, with a fake name [0]. I didn't check but they might rely on bad Debian packages just like Docker.

---

[0]: https://twitter.com/d53r/status/1055403252286717952

There was a deliberate decision, not a random mistake: https://askubuntu.com/questions/1037646/why-is-openjdk-10-pa...

TLDR: Ubuntu Bionic came out before Java 11 was officially released, they did not want to have Java 8 as their default, but Java 10 would not fit their LTS strategy since it is not itself an LTS release.

Having Java 10 in a package that claims to be Java 11 was a stopgap measure, and it has been fixed now, i.e. if you update that package you now get Java 11.

Unfortunately by doing this they made another mess.

When they updated OpenJDK to 11 [1] they didn't realize that some things have been dropped from the core project, and others have moved; as a result (among other things) JavaFX is now broken in Ubuntu Bionic. [2]

I think this is something inexcusable for an LTS.

[1] https://blog.ubuntu.com/2019/04/19/announcing-openjdk-11-pac...

[2] https://bugs.launchpad.net/ubuntu/+source/openjfx/+bug/18250...

TLDR; I agree, but I understand how this happened.

The logic seems to have been:

* Ubuntu 18.04 is a LTS release.

* LTS releases should only incorporate software that is supported on a similar lifecycle.

* OpenJDK 8 was moving out of its support window early in the lifetime of Ubuntu 18.04.

* OpenJDK 11 (the next long term supported JDK) was not going to be released until after the release of Ubuntu 18.04

* OpenJDK 9 and 10 were short term releases so they weren't options

* The move from Open JDK 8 to Open JDK 9 included a lot of breaking changes (from project Jigsaw/modularisation)

So a compromise was necessary. From a purist packaging point of view it makes sense; release JDK 10 as "JDK 11" and silently upgrade under the covers when possible. In theory this would only involve minor changes and the 'wrong' version would only be there for the first few weeks of the release. In practice there were some breaking changes - enough that the backporting of OpenJDK 11 into Ubuntu 18.04 actually took rather a long time. It's also, of course, really confusing when you "sudo apt install openjdk-11" and something totally different happens.

The problem with staying on OpenJDK 8 (perhaps the obvious approach) is that in an unsupported state it would lack security fixes. In practice of course several alternative upstreams are now available supporting it.

Hopefully it will all go with fewer surprises when we get to the next LTS edition (Ubuntu 20.4).

I saw earlier in this thread of blaming Debian, but all I'm seeing is Ubuntu snafus

I can understand their reasons, but they effectively LIED to the users, creating problems and headaches to them.

It’s a ends justified the means situation and they got it terribly wrong.

> There was a deliberate decision, not a random mistake

Debian's mess was not a random mistake. "Whoopsie, I somehow deleted the EA tag from the version string - actually it must have deleted itself. Anyway, I've started uploading it to the package index, it's too late now."

debian is not a trustworthy source of software if you expect it to be close to the upstream version.

They will change things around completely (MySQL/MariaDB), cause terrible bugs (OpenSSL), create crazy/renamed/weird versions (apache httpd -> apache2), etc etc.

Frankly you are lucky to get something this close to real java and not a shell script wrapping GCJ or whatever.

>And any day Debian decides to stop using Foo, is always a happy day for Foo's upstream. Finally, the stream of bugs matching the below template will stop spamming Foo's bug tracker:

>> Debian X.x (stable) >> foo <5-10 years old release> >> >> <XXth duplicate of a 7 year old issue that has been fixed 3+ years ago | 5 year old Debian-only bug caused by a Debian patch>.

>The only appropriate reply to such bugs IMHO is:

>> Fuck off. And don't come back before switching to a distribution >> with a non-broken definition of stability.

>Unfortunately, most upstreams always try to stay civil.

https://lists.debian.org/debian-java/2019/05/msg00008.html

> I can only comment on the OpenJDK 11 backport in Stretch since I'm the one who uploaded it last month. Debian 9 "Stretch" is far from being usable with Java 11 (we poured a lot of work into Debian 10 to make this possible) and this backport can only be reasonably seen as a technology preview aimed at gathering feedback.

More at link, but that’s concerning. We’ve been using the “OpenJDK” images but maybe we should look at using a different set of them.

There don't seem to be any debian emails about it: https://lists.debian.org/debian-java/2019/05/threads.html

The place where you'd want to look is the Debian Package Tracker entry for the source package, openjdk-8.

Here's the URL: https://tracker.debian.org/pkg/openjdk-8

That has the current status of the package, and links to the bug tracker, which is where I'd expect things to appear!

Yeah, Moritz Mühlenhoff really should have stayed a movie critic, and should not touch important debian packages anymore.

That seems an odd place to look, considering this is at the top of the page:

> package is gone

> This package is not in any development repository. This probably means that the package has been removed (or has been renamed). Thus the information here is of little interest ... the package is going to disappear unless someone takes it over and reintroduces it.

So it is an obsolete package?

Debian does not pull things from random Docker containers.

The random Docker containers are using Debian's own packages.

This is the norm for Debian, including their own mystery meat in every package without asking upstream, and then making it look like an official release. I remember there was a post about that from the Redis guy a few days ago.

The guy who made those Docker images is not to blame in my opinion because he just trusted Debian, which is a renowned distro; he expected them to do the right thing. I believe Debian has a good reputation that they don't deserve anymore.

To be fair, I wouldn't have known what "WE" meant either. Why not "WIP" instead?

Unrelated, but one thing that's confused me, as someone who doesn't really use Docker.

I don't think of a programming language as being the thing upon which I would base my layers of containers. My expectation, as someone coming from not-containers, would be that I should first start with a core-libraries container, _then_ a language container, and then packages on top.

Having this clarity of composition, I think, is important if I need to start bringing other stuff into the environment, particularly stuff that requires compilation, so that I know what to target it against. It also makes it easier to know when a layer needs to be upgraded, for example because of a security issue.

The downside I see is that maintainers of layers higher up in the stack would possibly have to build multiple variants of their layer. But maybe that wouldn't be too big of a problem?

Core library of what programming language, if you haven't picked a language yet?

I think the OP's talking about stuff like Qt, OpenGL, BLAS/LAPACK.

I'm not sure what kind of deployment you are envisioning where this makes sense. If you are running a java application, you likely want a bare bones OS + jvm of appropriate version (like the ones being discussed).

The openjdk container is based on a distribution container and publishes multiple versions for different distributions (debian, alpine, windows, slim debian, old debian version, etc.). Users then base their containers on top of an openjdk container and add their packages dependencies to their docker files. The issue here is that the language container seems to have pinned against a specific non-release version of openjdk.

> It also makes it easier to know when a layer needs to be upgraded, for example because of a security issue.

This doesn't really apply. You rebuild when one of the dependencies needs to be updated. It didn't matter how deep it is into the stack or which position.

Reply on news.ycombinator.com