Can you speak to why you were behind latest on the JVM? I can think of less than a handful of breaking changes over the last 15 years. I'd say it was more stable than Go.
* In order to upgrade the JVM, all the software you run has to work with the new JVM. If there's even one minor library that doesn't work or is suboptimal, you can't cut over. This is not an issue with Go because everything is compiled to an x86_64 binary there, whereas in Java everything is a jar file that must be run under (almost always a single) JVM.
* During a JVM version change, often Oracle removes or modifies non-public APIs that you need to achieve acceptable performance. For example, there is still no public way to free a DirectByteBuffer or create a FileChannel from a FileDescriptor, so you have to use the non-public APIs. Go almost never has this sort of problem since they tend to provide public APIs for everything you need, including platform-specific things.
* We run really big JVM heaps (>100 GB), and so minor changes in the GC behavior or default settings can cause major issues. For example, JDK8 changed the defaults for many GC tunables. It takes weeks of work at least for us to validate that there are no significant regressions. This is an issue that I would expect Go to have as well since they are changing the GC.
* Enterprise customers are extremely risk-averse, and they're not enthusiastic about deploying a new JVM. Operationally, they don't see any upside, only downsides. Of course JVM upgrades have to happen eventually, but they usually happen when new software is rolling out as well. Oracle's decision to stop shipping security updates for older JVM versions has "helped" in a sense by making the issue seem more urgent.
* Open source projects don't like dropping support for users running older software. There is usually someone around to argue against dropping support for anything that rolled out within the last 5 years.
The private APIs thing is indeed one of the most common issues, along with needed upgrades to bytecode rewriting libraries.
I think both Go and Java have been good about avoiding backwards incompatible changes in the standard library. Both languages have an explicit policy of avoiding these whenever it is at all possible.
Frankly, the Go standard library is a lot better written than the Java one. For example, I dare you to figure out how to call statvfs from Java, or figure out how many hardlinks there are to a specific file inode. Or make an asynchronous DNS lookup. Even simple things like creating a socket without doing a DNS lookup are very difficult to achieve in the Java standard library.
However we also have projects where the Java version is married to whatever the Websphere deployment of the day supports.
And on Android, well there is no upgrade at all. Which is yet another reason to use the NDK, even with all the 3rd class developer treatment, at least the C++ compiler gets updated and doesn't depend on the Android version of the target devices.
I suspect this really boils down to operational complexities of Linux distros, not Java. If your package manager only lets you install one JVM then maybe this seems "complicated" relative to Go, but that's not a Java problem.
WRT the standard library, yes the Java standard library doesn't expose UNIX specific syscalls. It exposes stuff at a higher level instead because it tries to be portable. That's a different tradeoff to what Go makes but I wouldn't say that makes it badly written. To me badly written would mean buggy, confusingly designed, too small or too big etc. If you want to write non-portable software then that means you may have to link in an extra library or so (like JNA).
Creating sockets does not do DNS lookups in Java. You may be thinking of the URL class, which does, and there's a URI class that avoids that.
The desire for a single JVM comes partly from the architecture of Hadoop itself. Hadoop is structured as a framework (you give your MR job to YARN and it runs it by creating new JVMs for you).
The Java standard library is weak in many areas. The "write once, run anywhere" ideology is part of it, but there are also just... weak parts.
I don't think the situation is all that different. All the Java code ends up being binary as well, at least after running a while.
At the moment, I think most people's Go projects simply have fewer dependencies. If you poke about on some of the popular Go projects, you'll see some of the bad coding practices that will result in upgrade breakages like checking for exact error strings (unfortunately needed sometimes, I know), so I foresee the same issues over time.
* when "enum" keyword was added, if you had a variable named enum you had to rename it.
* If you use any of the sun.* packages you always ask for trouble on major upgrades.
* GC changes over the years can change how your program runs (latency changes, OOM issues). This probably applies to Go as well.
So while your awesome CMS tweaks from JDK7 become worthless once you switch to G1 on JDK8, with Go all you can do to optimize the GC is to create less garbage to begin with.
Not trying to say one approach is better than the other: Go's approach is operationally simpler but far less sophisticated than the JVM's.
This isn't actually true, Go has a single tunable: "GOGC" https://golang.org/pkg/runtime/
Using sun.* packages or relying in GC behaviour is a way to make Java code not portable across certified JVMs.
For example I took part in some projects that were married to IBM JVM, because they were relying on its features.
You don't really have a choice in the matter. If you're writing high throughput or low latency applications you are dependent on the JVM's GC behavior, period.
Just as a very basic example, a certified JVM 8 is not required to have G1.
As for high throughput or low latency applications, yeah actually one is dependent on the whole stack, hence why HPFT is already moving into FPGAs.
IIRC, that caution about using sun.* packages was mentioned by Sun in docs of early Java versions, like 1.2 / 1.4 etc. Not sure about more recent ones.
So that means you either run different JDKs for different services or you stick with the lowest common denominator.