Removal of Unsafe in Java 9 – A disaster in the making | Hacker News Reader

Removal of Unsafe in Java 9 – A disaster in the making

blog.dripstat.com

92 pointspdeva1136 comments

The quote stops short, and for someone like me who doesn't follow Java very much, it feels like it's misquoting via omission:

"Let me be blunt -- sun.misc.Unsafe must die in a fire. It is -- wait for it -- Unsafe. It must go. Ignore any kind of theoretical rope and start the path to righteousness _/now/_. It is still years until the end of public updates to JDK 8, so we have /years /to work this out properly. But sticking our heads in the collective sands and hoping for trivial work arounds to Unsafe is not going to work. If you're using Unsafe, this is the year to explain where the API is broken and get it straight....

"Please help us kill Unsafe, kill Unsafe dead, kill Unsafe right, and do so as quickly as possible to the ultimate benefit of everyone."

The quoted engineer understands that people use it, claims that there's no immediate risk (since Java 8 will stay around), and very very strongly implies that there is a plan to move from Unsafe to something better than Unsafe providing equivalent functionality in a stable way, and asks for help with that. Is this correct?

Yes you are.

Here is the link to the full quote.

http://mail.openjdk.java.net/pipermail/openjfx-dev/2015-Apri...

The post directly links to the full quote. If the intention was to misquote that would not be the case.

> there is a plan to move from Unsafe to something better than Unsafe

It is this plan and the implications of it that the post is discussing

Actually I don't see the post discussing anything, is just feels like "Unsafe is going away, the sky will fall!"

The intention may not have been to misquote, but that is the effect that I saw. I had a similar reaction that geofft did when I read the whole email: it sounds like a reasonable effort to start the process of moving to something better than what is currently available. The submission here does not seem to take that into account.

I disagree that the article is "misquoting via omission." The article claims that "this engineer hates the Unsafe class for no real reason at all." The full quote which you reproduced doesn't offer any reasons for hating unsafe.

I also disagree that there is "no immediate risk." The end-of-life date for Java 8 is 2017, only 2 years from now. It can easily take a year or two to develop a product or internal tool. Would you sign off on a big new performance-critical project in Java, knowing that you might not be able to get the performance you want when these low-level APIs are removed?

Java has had a bunch of security problems in the last few years, but most of them have not been related to Unsafe at all. For example CVE-2013-0422 related to Java webapps, CVE-2013-4444 related to Tomcat, and so on. There were a bunch of bugs for POODLE and other SSL issues, similar to many other programming languages. Perhaps you could argue that CVE-2013-0422 was related to Unsafe, but since in this case it was an internal standard library class, presumably Oracle would continue to give itself permission to use whatever unsafe-equivalent they come up with.

If new APIs are needed are needed to replace uses of Unsafe, Oracle could simply add them gradually over time. This would be low-risk and easy for everyone. I can't see any reason why Unsafe needs to be "kill[ed]... as quickly as possible."

There's a lot of backlash here against the author with comments like:

"You used a sun.* package? You deserve to die in a fire as well!"

These comments look like they are coming from people who have never used Unsafe. First, Unsafe provides some immensely useful features to the Java ecosystem. Second, there is no alternative to Unsafe on the JVM. That includes JNI (native code integrated within the JVM).

So far, Oracle has stated their intent to remove Unsafe from Java 9. To all the users of Unsafe, Oracle is basically saying, "tell us your use cases for Unsafe, and if we decide they are valid we may try to create an alternative for you. If you're lucky, we may even ship that alternative in Java 9 or 10! Have a nice day!"

Many people, myself included, find this unacceptable. For instance, it looks like several groups are currently collaborating on a document discussing uses cases for Unsafe and whether alternative features satisfying those use cases are expected to appear in Java 9:

https://docs.google.com/document/d/1GDm_cAxYInmoHMor-AkStzWv...

As you can see, the majority of the cells in the column "Expected in Java 9" flat out say "no". There is a good reason people are up in arms about the removal of this API.

Personally, if I had to choose between Oracle keeping Unsafe and Oracle implementing Java 8 Lambdas, I would have picked Unsafe in a heartbeat. At the end of the day, Lambdas are mostly syntactic sugar, especially in their current implementation. Unsafe, on the other hand, actually empowers you to do something more.

That link is a very useful contribution to this discussion (and possibly the first really useful comment in this thread), thanks. Everyone's entitled to their opinion, informed or not, but if you want an informed opinion, the question needs to be what the current uses of Unsafe are, why they're used, and whether there are plans for a migration path.

It sort of seems like the major use case (judging from those numbers) is reading and writing raw byte buffers, possibly mapped from other objects/files, with native performance. Is there a potential sensible design for a bounds-checked raw data API ("unsafe but only within these pages") that maintains the JVM's usual safety guarantees for all other data, and can be implemented in a performant way?

The document is also linked in the blog post, on the very first line...

It was not obvious to me that the other side of that link would be a living document on current uses of Unsafe.

If the author is correct (which I am not questioning or doubting) and very high profile projects are using this feature... and Oracle refuses solicit input from the community... those developers with clout can and should maintain a fork of OpenJDK to include this feature. If things like Netty only run on the forked version of the JDK, and the developers are committed to not porting or working around a lack of Unsafe support, Oracle will be forced to reconcile with the community or risk burning down their own house.

Do not be surprised if they remove this feature from the GPL'd OpenJDK but maintain an analogous feature in their proprietary JRockit JVM. There is a good chance that this is a "shot heard around the world", and now is the time to start fighting back.

Oracle did a survey last year exactly as input for what to do.

Good, that seems like a smart move. It is of course possible for them to still ignore the results of that survey, or to game the results, or that a majority of respondents don't care or understand the issue anyhow. My point is that if this feature is important enough to people, they can take real action and don't have to kiss the ring and beg at the feet of power in order to achieve the result that they want.

They are not ignoring the result, and are doing exactly what the author wants them to do, except they started doing it before this hysterical piece was written.

I kind of figured as much... and that was my point actually... that writing and whining about it useless, and that if they actually do start misbehaving the community has the capability to take real action.

Here, Java ONE 2014, one year ago!

"Safety Not Guaranteed: sun.misc.Unsafe and the Quest for Safe Alternatives"

https://www.parleys.com/tutorial/safety-not-guaranteed-sun-m...

Every Java developer worth its salt knows about Java ONE.

Bet you I'm worth some salt... and I don't pay attention at all to "Java ONE"

Except that those of us on the Java community that follow Java development are fully aware of the 2014 survey, done by Oracle about how Unsafe is being used and what is our opinion on it.

Of course, many HNers that hate Java aren't aware of those discussions and post comments without knowing the full background.

What is the purpose of enforcing module boundaries? That appears to be why none of these internal classes will be accessible. Outside of a sandboxed environment, who cares what you do? Is it for some magic optimization? Or just some folks on a rampage? (The quote from the guy saying it needs to die, and I read the mailing list, well it seems to be baseless.)

There are documents and presentations discussing that.

Yes, it has a variety of benefits, including security, performance (e.g. a static linker with DCE becomes a lot easier), not exposing internal guts to apps that then come to depend on them, etc.

All that said, if you want to disable module enforcement, then they seem to be saying there will be command line switches to do that. Much like there are switches to disable other forms of rule enforcement.

One of the purposes of Jigsaw is offering a way to define a public API based on multiple jar files, similar to what OSGi does.

Similar in concept how Assembly Modules work in .NET, with friendly internals, which incidentally not many devs know about.

In Java's case a linker will be part of the SDK, so that only the Java code required for a given application will be deployed.

Since Oracle needed to clean up dependencies to implement Jigsaw, they are taking the opportunity to also remove most internal APIs, Unsafe being one of them.

They are also in the process of replacing JNI with something more programming friendly, P/Invoke style, but it will only come in Java 10.

There is also the reason that in a memory safe language every use of JNI and Unsafe is another security bug waiting to happen. If the compiler/runtime are able to offer safe APIs with similar performance, then those exploits can be avoided.

So the explanation is:

  Let me be blunt -- sun.misc.Unsafe must die in a fire.
  It is -- wait for it -- Unsafe. It must go. Ignore any kind of theoretical rope and 
  start the path to righteousness

And what the author takes away is:

  This engineer hates the Unsafe class for no reason at all.

The reason the engineer hates it is that it's unsafe. Disagree with it, sure. But it's a reason, and it's a good reason (imho). I understand that it's commonly used by apps I use, but it's hard to keep reading after this kind of knee-jerk.

In addition, it's in the sun.misc package, if it's not in java.* or javax.* it is ripe for pruning, you'd think people would have learned by now to stop using internal classes since pretty much every JRE/JDK release known to mankind has changed them or removed them.

It's sort of amazing that the diatribe here is against Oracle for wanting to remove an undocumented internal API because of typesafety, and not against the open source projects that use a specifically unsafe, undocumented internal API. I mean, nothing against these projects -- I understand that type safety can carry a performance penalty -- but the anger of this article is amazing.

The issues is 2 fold for people.

1) People use Apache Storm and Cassandra daily. These need Unsafe to function at current performance changing its functionality may change these tools. This can be considered existiential threat to their job.

2) The one big selling point of the JVM is that JVM 1.0 code will run in JVM 8.0 without issue, and often much faster. With this change that isn't necessarily true, and potientally when your .jar was compile will weigh on which JVM is used or HOW that jvm is started.

1. Have the maintainers of Storm, Cassandra, etc. said whether they architecturally need Unsafe in its current form, and whether the replacement APIs under development will fulfill their needs?

There's a lot of software out there that uses MD5 or DES right now. Saying that the software needs MD5 or DES is a different claim entirely.

>Have the maintainers of Storm, Cassandra, etc. said whether they architecturally need Unsafe in its current form [...] ?

As there is no alternative to unsafe serialization in terms of performance. It goes without saying.

>There's a lot of software out there that uses MD5 or DES right now. Saying that the software needs MD5 or DES is a different claim entirely.

I agree, but you are putting the trees before the forest.

Saying implementation X is flawed is one thing. Saying We are removing implementation X and offering no alternatives is another. While there are plan to offer an alternative, they are simply plans not an immidate alternative, thus a problem. When DES/MD5 were removed it was clear that they were already inferior implementations, and superior alternatives existed. That isn't true for java.

> As there is no alternative to unsafe serialization in terms of performance. It goes without saying.

I don't think that's true, though perhaps I don't understand what you mean by "unsafe serialization".

First, I suspect one can use JNI and get at least as good performance. There are other tradeoffs, like the ease of packaging, but in order to evaluate those we need a clear problem statement.

Second, premature optimization is the root of all evil, and we should be looking at realistic benchmarks to determine that there is a performance gain. There are many documented examples of software safety checks adding negligible performance overhead because they get branch-predicted away. Alternatively, new hardware support like Intel's MPX adds bounds-checking at the instruction level, so if we're talking about raw writes to a bounds-checked buffer (which is a vast improvement over raw writes to anywhere), this can probably be implemented efficiently on new hardware. On old hardware, the JVM could implement these as unchecked writes, which preserves the exact same performance, but the API would now have information to add safety where it is performant.

Finally, sun.misc.Unsafe covers a lot of ground. If we can restrict it to the specific uses that these projects make of it, that's still an improvement. I don't think these libraries use the whole thing, do they?

Very few things in science go without saying, and software performance and correctness is science.

Seeing as you've admitted you don't understand what I'm talking about. Here is some reading material:

http://www.javacodegeeks.com/2010/07/java-best-practices-hig...

Thanks, I definitely understand this topic imperfectly -- I am familiar with high-performance serialization, but I'm not familiar with Java or best practices with doing this sort of thing in Java. So it's definitely helpful for you to be clear in what you're discussing and what Unsafe is currently used for.

Is the specific Unsafe use here to serialize a Java object into a byte array? The linked article doesn't describe using Unsafe, so I'm still not sure what functions are at issue.

This is what I was referring to about a bounds-checked buffer in my parent comment: it seems like you could state that accesses to an area of memory are unsafe and unchecked, except to check that the reads/writes are within the buffer. This preserves safety for the JVM as a whole, but gets you native performance within the buffer. Intel MPX should be able to implement this efficiently, and the standard techniques in other languages for efficient bounds-checked memory access should all apply.

Am I completely off-base here?

You are. Here's a great idea, if you've never implemented a JVM don't tell somebody how it works.

Java VM isn't native code. Serialized objects are "Field"(s) within the JVM. Which is short hand for a "Growable Page File" more or less. Which starts at a fixed 32bit address. How does that work? JVM implementation takes care of that, so you decide.

On top of that whats inside the Field itself doesn't actually matter to anyone provided the same read/write opcodes do the right thing(s).

The serialization process starts by jumping to your initial field, copying the serialized data into another buffer. While checking integrity, and assuring that your data conforms to the serialization standard b/c its in memory representation doesn't. The fun part is when ever it encounters a RetAddr type it has to jump to that Field, and serialize THAT Field also, as that Field is part of the object your serializing. Of course you don't actually know whats in the Field without consulting its ClassField which gives you a basic prototype of what-is-where.

Furthermore what ever optimizations the JIT makes, it'll end up treating Fields more-or-less like stack frames. So you have to keep an active record of how it'll mangle memory, so you know how to untangle that when you want to do serialization.

Understand now?

About #2: there are many internal changes even between minor releases and if you relying on those internals, your code is NOT portable.

sun.misc.Unsafe effectively became a non-internal API a long time ago. Yeah, it's flagged internal, but it doesn't get broken without a damned good reason (I can't remember the last time it meaningfully broke, even).

1. it's quite likely that the Unsafe classes will be accessible with a flag, so for this kind of projects it's not really a big problem, as they provide the shell scripts for launching the jvm with all the necessary parameters.

On #2 - Java 1 code using the public API (java.*) will run in Java 8. Using internal APIs was never considered forward compatible.

The problem with that, is unlike most of the other internal classes, there is zero other way to do the things people are doing with unsafe without it.

It's not a matter of expedience, it's a matter of necessity.

Yes, removing Unsafe would make things like PowerMock and Robolectric impossible to (re)implement. So what? Those libraries never should have been possible, and we can live without them.

Languages need to be selective about what features they support, and Java was never intended to support things like mocking static methods, or creating instances without invoking any constructors.

Without the current unsafe package most performance sensitive applications (whether throughput, latency, or pause sensitive) will have no recourse to meet their current performance levels.

That will leave people to choose between staying on Java 8, worse (and often unacceptable) performance, and migrating to a different language.

At this point, there is a very good chance if your library/application is known for performance and exists on the JVM it uses unsafe and without a migration plan all of us who care about performance will likely have to move off the language.

With some effort, I'm sure the folks behind Cassandra, Hadoop etc. could achieve acceptable performance with allocateDirect(). It doesn't support explicit freeing, but you can always reuse buffers and write your code to allocate slices within them.

We can live without Robolectric? How do we write tests for Android code without it?

Turn on unitTests.returnDefaultValues, write some mockable wrappers around core classes like SystemClock, unit test what you can, and write instrumentation tests for everything else.

An increasing number of teams are moving away from Robolectric anyway, since it's so complex and buggy, and there are some things it will never properly support such as COLLATE LOCALIZED.

That is a pretty crappy explanation, on the engineer's side.

To be clear, any explanation that is not immediately followed with examples to how current valid uses of Unsafe can be accomplished without it, are simply witch hunts by an engineer that hasn't had to meet the same requirements.

It's a) unsafe, and b) an internal, undocumented API.

When you start importing from sun.misc.* you've made the conscious (or ignorant) decision to write non-portable code which may well break due to any JVM release. It sucks that these people are going to have to update their code (or that end users won't be able to upgrade to Java 9 without adding special command-line flags), but it's not due to fickle Oracle developers, it's due to technical debt that's finally catching up to them.

You quite simply have NO option to do on the JVM most of the things people are using unsafe for in Java 9/10 without a migration plan from Oracle.

What Oracle is close to "hey all you performance sensitive folks, we aren't going to support unsafe anymore, can you tell us what you are using it for and how we can migrate to that explicitly?"

It just so happens that one of the leading proposals is "leave in java 9 but make it explicit to turn on". This is probably the right way to do this (in my opinion) but it does have the downside of making neither camp happy (it leaves unsafe so the purists are mad, and it causes headaches for people using unsafe).

Whats wrong with:

- deprecate in Java 9

- "make it explicit to turn on" in 10, offer alternative official APIs as fallback

- remove entirely when most people have switched

Nothing at all. That's what I assume will happen at the end of the day. But people saying that we can just turn it off in 9 because you shouldn't be using internal api's don't seem to get that there is no external solution.

Why should third-party libraries pay for Oracle's technical debt?

It's not Oracle's technical debt, it's the developers of the third-party libraries that are importing from sun.misc.*.

I think the author's point about being able to make safe Unsafe replacements and keeping Unsafe around (for the time being) to enable better compatibility was thoughtful.

Entire companies exist around functionality provided in Unsafe. I don't see this going well if they remove it without providing access to the functionality it gives.

Java has plenty of problems, but backward compatibility over the years has been simply amazing. This would chip away at one of the few legitimate reasons to use Java in 2015.

sun.* has ALWAYS been exempt from that guarantee though.

yeah they went the fast and cheap route instead of doing the right thing, and they are now going to be bitten by it, and they can go cry a river for all that I care.

those are the sloppy engineers that are holding back evolution of software at large, from little things like obscure functions to huge operating systems like http://blogs.msdn.com/b/oldnewthing/archive/2005/07/28/44439...

You forgot to explain what the "right thing" that they should be doing is? Even according to most articles on the topic there is very to any pleasant alternatives that are cross platform e.g. https://dzone.com/articles/understanding-sunmiscunsafe

jni, and don't tell me now that c isn't portable.

C isn't portable.

You cannot write it once and run it everywhere, each platform needs a bunch of macros and flags. Plus you'll now either be delivering it in binary form (which may or may not work due to bitness, OS incompatibilities, and so on) or you'll be delivering it as code (which may or may not work due to compiler incompatibilities, build system components available, and missing libraries).

C is only portable in the sense that "every platform ever made has a C compiler." But if you've ever written a C application which is just meant to run on Windows, OS X, and Linux you'll know that it is a painful experience (double that if they're meant to compile it themselves).

*bad C isn't portable. and you can pack as many platform as you wish in your binary distribution, so single download still works everywhere (everywhere you built beforehand).

I think the issue is a disagreement over what portability means. To me, portability doesn't mean write once, run everywhere, it just means you can make it run on a different platform with a bit of effort.

In a C/C++ program, if you're careful, you can isolate the platform-independent parts so that you only have to write them once. VM languages abstract further by creating a new language for these platform-independent parts. However, the platform-dependent parts are still going to be written in C/C++, or assembly, or generated by a JIT compiler written in C/C++ (there are rare exceptions).

So if anything, I would say that C/C++ enables portability, simply because there aren't any other viable choices besides assembly when it comes to generating platform-specific machine code.

no the issue is that while it's true that complex library accessing native stuff in c are hard to port, you don't need the whole stuff ported under jni, and here is the missing link, the functionality provided by unsafe is limited enough (think of it as a building block for higher level stuff) that it could be reasonably done in c, using posix stuff and some patching in for window which while non posix enough is at least stable trough versions.

so it basically become write twice run on many things, which is good enough.

the problem is that it's not 'rewrite all the libraries in jni' but only 'rewrite the two calls you need in jni' - scoping is relevant, as we are not arguing about jni in a vacuum.

Java is very explicit that anything not part of the public API may not work in the next version. So I'd say them disabling it, they're in the right. Since so MANY projects use it though, I think it might need some special attention.

Entire companies existed around DOS extenders and TSRs. Microsoft rearchitected and everyone is better off for it.

The argument needs to be that the use of Unsafe by these companies (or the projects listed on the blog post) both is good and has no alternatives, not that the use exists.

There should be some argument made that the use by these companies is causing trouble, as well. DOS had real identifiable reasons not to use it. The rhetoric used so far just has Unsafe upsetting some engineers for existing.

Once again, The Platonic Engineer gazes hatefully at the gronky, ugly and awkward things that make things work in real life, rub his hands together and says...

"Burn it to the ground."

Or you could actually stop for a second and find out why your current model needs so many ugly and awkward workarounds and try to adjust it accordingly - which seems to be what Oracle is trying to do in the long term.

This is an odd one, though. I can not remember seeing/hearing any real issues with Unsafe. For the most part, people don't use it. There are a few that do, to ridiculous benefit. As things stand right now, without it, you can not get anything close to the performance people are seeing with it.

So, what many of us are seeing here is someone is upset with basically the shape of an API. They have no evidence that I have seen of trouble caused by it. Only a desire that such things not exist. This feels almost puritanical in how it is being exercised. Which is just plain silly for a technical debate.

puritanical

I think the correct term is platonic: each of us has an Inner Platonic Engineer who cannot abide the ugliness of real world systems and insist on The Great Rewrite. A lot of progress relies on that engineer.

However, in large, functioning and mature software systems he becomes the enemy.

I question just how much progress relies on that engineer. Be honest here, there are no identified problems that are being solved here. Merely an undesirable in the system that is very effective at its job.

Progress, on other hand, advances something in the way of the problem space. Not merely rearranges something in the solution space.

But a new model should be that: a new model, not a suicide pact. Wise engineers keep the old API working while offering the new API (which, soon enough, will acquire its own gronkiness.) See the x86 and OS9/OSX evolutions, for example.

Thinking about the domain problem in the abstract is smart. Thinking about the current system in practical terms is wise.

Java 8 is still around, and will be around for long enough. It's the Classic to your Cocoa, and the analogy works well: you need a full, separate virtual machine, because the software could be doing unsafe things with memory that you don't want to affect modern apps, but that VM will be around for a few years.

The equivalent of Carbon in this analogy is software written to documented APIs, which will run on both Java <= 8 and Java 9.

Java 8 support will run out in a while.

I'm pretty sure it won't "be around for long enough" except if you decide to stay on unmaintained versions full of unfixed security issues.

It'll run out by September 2017, if not later:

http://www.oracle.com/technetwork/java/eol-135779.html

Do we believe that the affected software will not be able to move to the replacement APIs in the next two years? Honestly I'd expect that by fall 2017, the only versions of Cassandra, Spark, etc. that need sun.misc.Unsafe will themselves be unmaintained and full of unfixed security issues.

Herein lies the problem. As other commenters have pointed out there is no alternative way to do thing performantly without Unsafe. These "core" (as in used everywhere not core to java) libraries don't use Unsafe for fun but often because there is no other way to do something without compromising way too much on speed or other things. If Java 9 had replacements for Unsafe things it would be one things but it doesn't.

JNI

JNI has a very high per-call overhead. It is not an effective alternative to many things in Unsafe. For example, in Hadoop when using short-circuit local reads, the client determines if the file descriptor it has cached is still valid by doing a read from a memory-mapped file. The only way to do this with decent performance is with Unsafe#getLongVolatile. In contrast, JNI methods can't be inlined, often require copying memory back and forth, and don't work well with the just-in-time compiler.

In many cases, the overhead of JNI is too large to even use it, unless you move a large portion of the program out of Java. If Oracle goes through with this removal it will be a huge problem for Java and the JVM ecosystem.

Yeah, I don't get it either. Even the Lisp Machines had unsafe subprimitives you could call if you needed to and knew what you were doing.

I just read an article that says in order to even use Unsafe, you have to use reflection to unwind the stack and go searching for a reference to the Unsafe instance, because the Java authors worked so hard to make it private. So it seems to me that they'd be well within their rights to remove the class.

Normally when you go to that level of epic hackery, you can't expect for official support or backwards compatibility. So the author of this article using scare tactics to keep that feature seems rather disingenuous.

Could you provide a link to the article? I'm not saying you're wrong (I'm not a Java guy) but that sounds convoluted.

This is how you get access to Unsafe:

  Field theUnsafe = Unsafe.class.getDeclaredField("theUnsafe");
  theUnsafe.setAccessible(true);
  Unsafe unsafe = (Unsafe) theUnsafe.get(null);

Reflection, but no stack unwinding.

Chill.

There is no disaster in the making. Oracle is very much aware of the issue, and is actually already taking the very same steps suggested by the author. It has assembled a public working group that is coming up with a spec for a public "Unsafe" API[1] (that is safer than Unsafe).

In the meantime, use of Unsafe in Java 9 will be possible with a command-line flag in order to allow the working group more time to create the new API and for Unsafe consumers to migrate, and of course to allow existing libraries to run on Java 9.

[1]: https://docs.google.com/document/d/1GDm_cAxYInmoHMor-AkStzWv...

The point is if you have to pass a flag to allow the use of Unsafe, you're breaking "works-out-of-the-box" backwards compatibility.

"Works-out-of-the-box" backwards compatibility -- without flags -- has never been guaranteed (even the default GC is about to change), let alone for non-public classes. Whenever you compile a class that uses Unsafe you get a warning (that you can't disable!) that says "this class may go away at any time". Still the reality is recognized, and you'll still be able to use Unsafe -- despite all warnings -- and all you need to do is add a flag. I think that's very reasonable.

I agree in most cases that it's perfectly reasonable. But from a pragmatic position, I think this needs to be treated as a special case because of all the people that are using it. I still like the idea of having a flag that opts people in to using internal classes. It's like an agreement almost that they know what they're doing is not supported across versions.

Thankfully, Project Jigsaw (if it lands) in JDK9 should remove the problem of internal classes entirely — they simply won't be available to use.

It already did. Java 9 early alphas are available.

Backwards compatibility is only guaranteed for official Java SE and Java EE API's existing in the java.* and javax.* packages, anything in sun., com.sun., etc is internal to the JDK and has no such guarantees.

Every time you hear of an application that only works on a specific release of the JRE you can know for sure it's because they use undocumented internal API's — keep in mind this not only impacts compatibility with other releases of OpenJDK/Oracle JDK, but the sun.* API's are not typically part of third-party JVM's like JRockit or the IBM JVM.

Stop using internal API's and then bitching when Oracle makes changes to them, you only have yourself to blame.

This is exactly the same line Microsoft took when they broke poorly architected applications with the new (and much better!) security model in Vista. "You're the one who wrote an app that needlessly requires root privileges, you have only yourself to blame."

It... didn't go well.

And MS did a lot of work to detect such apps and make them work OK. What's the real argument against Unsafe? People are upset that it exists? Really?

Sorry I wasn't clear enough in saying that I agree with you here in most cases. But since Unsafe is used in so many applications that it should probably be an exception to this rule for now. That a better solution is to create a public API, then keep track of popular projects as they move to it. When it's moved then enable the flag.

> This is exactly the same line Microsoft took when they broke poorly architected applications with the new (and much better!) security model in Vista. "You're the one who wrote an app that needlessly requires root privileges, you have only yourself to blame."

And lo and behold, a decade later everyone has sucked it up and done the right thing. Sure, there were growing pains, but it was for the best and in hindsight everyone can agree with that.

Sometimes the boot just has to come down, for better and for worse, everyone will adapt and life will continue.

> Stop using internal API's

So what's the alternative? It's not like people are using sun.misc.Unsafe for the fun of it.

C++ and while your right, those same people were told this 17 years ago minimum by Sun, "don't use sun. classes". The other option is to keep using the old JVM. I mean I work on code all the time that require Java 6, how many people can say there Python runs on 3.x not 2.x. I'd say before anyone has to worry about Java 9 they have plenty of time to think about this problem. Add in the fact that Sun said use these at your own risk 17 years ago.

Many people seem to be upset with Oracle, but I think they should be upset with the vendors who ship applications/libraries that depend on the "sun.*" packages.

Oracle (then Sun) have since _1998_ strongly discouraged the use of them: https://web.archive.org/web/19980215011039/http://java.sun.c...

Fine, but why isn't the argument: "This then needs to be standardised in a Java.* package and correctly documented."

Unsafe is useful, and is evident by how popular it is. Heck even Sun/Oracle must agree because they created it for their own internal usage. So why not just go ahead and standardise it.

What is "Unsafe?" Is it unsafe, as its name implies? If it is unsafe, why to so many programs use it?

It's part of the OpenJDK/Oracle VM implementation, not an official part of the Java standard library. It allows for all sorts of fun stuff like native compare and swap operations (used for concurrent collections iirc), object allocation without calling the constructor, accessing blocks of memory directly without object references, and other things. The reason it's called unsafe is because it allows developers to bypass a lot of the safety guarantees that java is supposed to provide as a memory managed language. But it's useful for getting more performance out of the VM.

There's currently an effort underway to push through a variety of Java enhancement proposals that would add a lot of the unsafe functionality to the official standard library. This would enshrine the unsafe functionality as official, remove access to some of the more dangerous and less useful parts of unsafe, and would also mean that the functionality would be properly documented, which it is currently not.

Thanks for the thorough explanation - you summarised the issue well, as well as explaining the impact and flow on effects.

I have to say, I agree with the engineer.. I'm cringing just knowing this sort of practice is seen as acceptable, and it reinforces my distaste for Java. If you have to resort to insecure practices to achieve performance, there's something wrong with the framework.

Reading this thread as C programmer has me scratching my head.

Of course you have to resort to low-level functionality to achieve performance. That's how computers work. We can't all live in fuzzy wuzzy land.

Surely Unsafe is safer than writing the whole thing in C and using JNI?

Yes. Also there are many "safe" operations on "unsafe". Such as memory fences.

Multithreading is somewhat unsafe in general. And so is ability to do most I/O, you might overwhelm the system.

Sure. Unsafe in terms of program correctness. By that measure, new Thread() is unsafe.

But that's not a reasonable definition of "unsafe". This could crash the JVM is a reasonable definition. And fences and ordered instructions cannot.

I completely agree - I have no problem with leveraging low-level functionality, but there's usually a safe and and unsafe way to approach it - and from what I've read 'unsafe' doesn't take the secure approach.

(separately: why the downvote? I get downvoted for having the opinion that it's worth a tiny bit of performance for improved security?)

This sort of practice is present in the implementation of every mainstream programming language. 'Unsafe' operations are the bread and butter of computation.

As far as use in libraries, you wont get the same performance without it in most of the places it is used. This is much like Python's 'unsafe' use of C modules for performance.

As a corollary, many Java programs rely really heavily on very thick layers of abstraction (in my day-to-day on a multi-node "enterprise" Java app I regularly see stack traces of 200-500 function calls), and it's important to keep each slice of those abstractions as thin as possible.

As another reply pointed out, much better to do some hinky things with the JVM then to call out through the native interface every time you need high performance for a tight loop.

Why remove it though? What harm is being caused?

I haven't seen anything official about it, but I imagine it's been planned for some time. Sun/Oracle have explicitly warned against its use for a while. It seems less necessary when GC improvements in the last two releases have meant it's much easier to get great speed and latency with idiomatic and documented implementations. Further, what performance improvements are offered by Unsafe are obtained by breaking a variety of Java's guarantees, and I imagine that it is tiresome as an OpenJDK dev to hear about bugs "caused by the JVM" in libraries that depend on an unsupported undocumented API.

Just speculation, of course.

This seems a bit hyperbolic. Especially the comparison of Python 2 to Python 3. Its not like you could run Python 2 and Python 3 under the same interpreter by passing a flag.

Why on Earth is it a big deal that you need to pass a flag to the JVM at startup?

I understand that backwards compatibility is valued, but if I went to my CTO and said "we're gonna drop Java 9 in production, and we're not gonna test beforehand"...the reaction would not be pretty.

Document it, make it public. It's very much needed!

Compression in my library is 3x faster with unsafe. I think it is lesser evil then native code integration.

At least he didn't name the article "Unsafe considered harmful."

> Oracle plans to remove sun.misc.Unsafe in Java 9. This is an absolute disaster in the making and has the potential to completely destroy the entire ecosystem around Java.

You know your language has a problem if the entire ecosystem threatens to fall apart as soon as you actually try to enforce the language's conceptual model and try to make use of the guarantees it provides.

With the caveat that you need to believe in the hyperbole of a chicken running around with its head cut off.

Ok moving past the rhetoric of don't use internal classes, there is a value to sun.misc.Unsafe. It has been well known by Oracle for some time that it has to go away right.

Realise that those of us that do use it, are fully aware that it is a hack, and that we should be doing cleaner things. However know that those cleaner things _dont exist_; and that you, Dear application developer might not quite realise some of the shadow puppetry in making your favourite library work, please lay off the bile.

As others have said, this blog post stops short, the full quote is part of a very long and ongoing process to rid ourselves of Unsafe, replacing its majors usages with new API's.

This is a process Oracle started last year after the previous sun.reflect.Reflection.getCalleeClass mess. Yes, it no one should have been using an internal class. This method did not go away; it was made only available to the JVM internals, locked out from end users. The upshot of this was that, with no sane alternative, suddenly logging became between 2x and 100x more expensive where class names become involved. The JDK team somewhat back peddled on this change, and we are stuck here in a strange limbo land.

Was it right of Oracle to do this? Absolutely, everyone who goes down the internal package route is fully aware that these are open-state secrets, sure we all know them, but we should not be talking about them. Support for these damn things requires version to version checking, hideous reflection hacks and being prepared to read a lot of JVM source code for when bugs occur.

In light of all of this, and being the general level of awesome the JDK folks are they started a very direct community outreach to remove safely Unsafe. Last year they publicly asked people on the mailing list, and privately via email to do a survey (http://mail.openjdk.java.net/pipermail/core-libs-dev/2014-Ja...) on usages of Unsafe. I know this because I got an email from Paul Sandoz asking me to comment on Unsafe abusages. Out of previous thinking from the JDK folks, as well as from the aforementioned survey, we have a bunch of JEP's for improving things regarding Unsafe.

Where things have gone slightly sideways is that we have, currently no unified working group to deal with these changes. What this document is, is a proposal to make a working group, focused on getting the right changes into the JVM. In the same ways as Project Jigsaw is for modularising the JVM (and is indirectly responsible for removing Unsafe), or the MLVM project handled making the VM better support other languages.

Unsafe is dead, long live Unsafe! I fully expect that we will achieve the right changes in during Java 9 to allow those of us that need to break the rules, to do it in a supported fashion. I am _looking_ forward to the day when I can do setMemory or getAndSetInt via a supported API, and not one where I have to know the words Unsafe.theUnsafe. The above is a storm in a teacup, its open-source democracy at work.

What about shipping sun.misc.Unsafe as 3rd party package? Is there anything in it that can't be done as through JNI?

No, you can do everything in Unsafe via JNI, but that comes with all the disadvantages of JNI (including .dll's and .so's is not so easy for a fat-jar deployment).

It'd be unusably slow. "sun.misc.Unsafe" compiles in the instruction stream. JNI implementation would need to go through a complicated calling mechanism. I think it'd be up to 100x slower.

At least in OpenJDK it is in fact implemented as a JNI package, so it is already going through the JNI calling convention: http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/s...

That code shows that it's implemented as native methods, but not necessarily as JNI - these could be VM intrinsics, like Object::getClass and so on. This is the native code for Unsafe:

https://github.com/openjdk-mirror/jdk7u-hotspot/blob/master/...

It's certainly JNI-ish. Whether it is actually called via the JNI mechanism, i can't tell.

I think these are a strong hint they're not truly JNI:

Line 61:

  * <p> Most methods in this class are very low-level, and correspond to a
  * small number of hardware instructions (on typical machines).  Compilers
  * are encouraged to optimize these methods accordingly.

Line 90:

  /// peek and poke operations
  /// (compilers should optimize these to memory ops)

Why libraries need Unsafe, after all? What problems does it solve for them?

It's almost entirely about performance. Unsafe allows you access to concurrent intrinsics as well as memory layout functions.

It's true that it is widely used, but the people that use it also understood the ramifications of using it. If you were trying to make jvm agnostic code you didn't use it. If you were trying to make code that could run on a variety of jvm versions, you didn't use it.

A lot of what people use it for is being rolled into documented/standard libraries and this is just the migration pains coming out.

Why those concurrent intrinsics are not available with "safe" code? They are not compatible with Java Memory Model? They are not implemented at some platforms? What's so unsafe about concurrent primitives? There is API in java.util.concurrent.atomic, for example.

I understand that Unsafe is used for direct access to memory, but why this direct access is needed? May be there are other ways to solve those problems.

AFAIK there aren't other ways to get the performance required to be competitive in the financial space. People that are using these libraries don't want to use undocumented, Unsafe code. However, their hands are forced given the races they are competing in. They could switch to C++ but that's not really a great option for firms that have java code bases going back 10+ years.

Plus, personally, and for a lot of others, the jvm plus a dash of sun.misc.Unsafe, is a lot better than going over to C++.

More concretely, Here are two sets of trading libraries that use unsafe for direct memory access: https://github.com/real-logic/simple-binary-encoding https://github.com/OpenHFT (various libraries in here use it, once you drink the Unsafe Kool-aid it's hard to turn back from the performance benefits you obtain).

If Oracle just rips off the Unsafe band-aid without a replacement, the business decision for these firms is still sound. They will just stay on 8 and start making migration plans (likely to the C++ bandwagon).

However, It sounds like Oracle plans to give a sane alternative which is good and what everyone using Unsafe would prefer (along with deprecating it the meantime to give them some time to adapt).

One real world example: The CME (Chicago Mercantile Exchange) reworked the entire way they distributed market data (v3 of their Market Data Protocol) to mirror the cutting edge in performance oriented serialization libraries (e.g. Cap'n Proto). They also commissioned an Open Source library for parsing that data (real-logic SBE), it uses unsafe to approach the performance of C++ code.

Exactly. Though I'd add that its not just the financial space that is doing this for performance. Lots of people that are trying to scale high throughput systems use unsafe for similar reasons:

Akka for instance uses it: https://github.com/akka/akka/blob/0de9f0ff40fc5e43540df58718...

This is related and unrelated and maybe even meta to some degree:

There are a couple of companies that are guilty of using their free software update mechanism to try and trick people into installing crap they don't want (and might even mess-up their systems): Sun and Adobe.

I almost happened to me last night as I updated Flash on a test system and some bullshit anti-virus crap-ware was going to be installed because I was moving fast and neglected to un-check the check-box on the site. I was able to stop the install process and start over. No harm done.

These companies need public shaming to stop using this bullshit method to make a few more bucks. Are they so hard-up that they need to trick users to install crap to make money? I sure hope not.

Sun, Adobe, please STOP trying to trick users into installing crap-ware they were not looking for in the first place.

You understand of course that Sun has been out of business for a very long time right?

Sorry, brain fart, I meant Oracle

java hasnt been bundled with crapware since oracle bought it

Some installations I've seen have crapware pre-selected. I haven't looked at it lately, maybe they changed this.

I read this article yesterday and I really dislike how it is worded. The "this engineer" has a name, Donald Smith, this is what he said:

"If you're using Unsafe, this is the year to explain where the API is broken and get it straight.... Please help us kill Unsafe, kill Unsafe dead, kill Unsafe right, and do so as quickly as possible to the ultimate benefit of everyone."

It reads like a request to have a discussion on Unsafe, to talk about it and find solutions. Unfortunately, I don't see anyone pointing out why Unsafe should be kept around on that thread (http://mail.openjdk.java.net/pipermail/openjfx-dev/2015-Apri...).

And this quote from Oracle shows an effort to do things right:

http://mail.openjdk.java.net/pipermail/discuss/2013-October/... "This is definitely something we plan to propose for SE 9. I expect to see a JEP from the current maintainers of sun.misc.Unsafe..."

This (http://blog.codefx.org/java/dev/how-java-9-and-project-jigsa...) seems like a more balanced article on the situation.

"So far we focused on the problematic aspects of Project Jigsaw. But that should not divert from the exciting and – I think – very positive nature of the planned changes. After reading the documents, I am impressed with the scope and potential of this upcoming Java release. While it is likely not as groundbreaking for individual developers as Java 8, it is even more so for everyone involved in building and deploying – especially of large monolithic projects." near future. (This is far too small to require a whole JSR.)"

Literraly meme language

Still using Java in 2015 – A disaster in the making

What would a MS fanboy suggest as an alternative?

Reply on news.ycombinator.com