> I can't see any reason why this requires the entire program to be subject to GC.
It doesn't. Lots of Java low-level programs (DBs, caches etc.) manage most of their heap manually. But an "internal" GC would have to be just as good as a "whole program" GC anyhow in order to make the non-blocking data structure robust (I generally don't use the term lock-free, as it's a very specific case of a non-blocking data structure; many interesting nonblocking DSs are only obstruction-free rather than lock-free).
Also, GCs have other advantages, such as a much higher memory allocation-reclamation throughput than any general manual scheme, especially (though not only) in the multi-threaded case. Many GCs may introduce pauses, but all the good ones would give you a much higher throughput than anything you can achieve manually.
> Maybe to you the JVM gives enough benefits to be worth giving up all this control, but to me it it's just nuts that you'd put yourself on top of a platform that you have to work against so much.
Yes, to anyone doing hardcore multicore programming, the JVM is probably the best choice. You are not giving up control, though, because C++ has far worse problems in those areas (forget about safepoints - C++ doesn't even have a memory model, or didn't until C++11, and most C++ programmers don't even know what ordering guarantees their particular compiler provides). Also, if you think you're not "giving up control" in C++ (or C; or Assembly), then you're sorely mistaken. You have little or no control over ILP, prefetching, branch-prediction, outstanding cache misses and all the magic modern CPUs do at runtime. You could think of the JVM as an extension of the CPU only in software – modern CPUs do so much crazy stuff to your code anyway, that you can't possibly know how exactly it will be executed even if you program directly in machine code. You can no longer get truly close to the metal, so being one step rather than two steps removed is not much of an advantage any more.
> "Oh my god, [because we derive from java.util.Collection] we have put more lines of code into supporting a method that nobody in their right mind would call for a concurrent queue, than almost everything else combined. Between iterators and interior remove, that's probably 80% of our concurrent queue code.
Sure. The code is often complex to support moot use-cases. Still, it's leaps and bounds beyond anything in pretty much any other language. Again, the reason is mostly coincidental. From its beginnings, more or less, Java (or Sun) has attracted the most prominent concurrency researchers. Java 1.5 was the first general purpose programming environment with a memory model, iron-level access to concurrency primitives (fences, CASs), and some world-class GCs, all requirement for state-of-the-art concurrency research. To this day, concurrency researchers prefer the JVM, despite its annoyances (and what language doesn't have those?) to any other platform out there. That's why most concurrent DSs get implemented in Java first, and their Java implementation is usually the most polished and robust.
But to quickly go back to where we started, a higher-level PL often brings about performance improvements. There are famous examples of sorting algorithms in C++ easily outperforming their C implementations. Why? Because of templates vs. function pointers. More precisely - due to inlining, as inlining is "the mother of all optimizations" (it opens the door to other optimizations). Now, of course you could achieve the same in C, by writing the sort function many times, but it's cumbersome and error prone.
The JVM is "the inlining magic machine". It inlines code that a C++ compiler never could. There are some areas where C++ is more performant, namely computations on arrays of complex numbers, but hopefully that will be addressed in Java 9. But this inlining capabilities of the JVM is why it often outperform C++ even in single-threaded code. Again, you could achieve the same in C++ by writing the same code several times (or by generating machine code at runtime, which is what the JVM does). Also, this effect is usually perceived in larger applications, because then you use virtual methods.
Anyway, that's pretty much what I have to say on the matter. I have a lot of respect for C++. I was a C++ programmer for many years. The JVM brings many interesting performance improvements over C++, as well as performance regressions in comparison. All in all, Java usually comes up ahead in large codebases, and falls behind in specific use cases (mostly numeric HPC). In addition, there is no other programming language/platform out there that even comes close to the JDK's power when it comes to hardcore concurrency; some successful implementations find their way to TBB a few years later, and some don't. If you're trying to push the envelope on concurrent software development (my day job), Java is pretty much the only sensible way to go.
Still, there are valid reasons to prefer C++ over Java in many circumstances, but sheer performance is seldom one of them.