Surely these two languages can't be compared? Java is a much higher level language than C++. It is interpreted at run time (byte code) rather than compiled. Should people not be making comparisons to C# or other similar languages?
Speeding up the JVM has always been on the critical path - and it is important to recognize the distinction between Java and the JVM. They are two entirely different types of things. Java is measured by expressiveness. The JVM by speed and memory efficiency and concurrency and other performance abstractions.
Rich Hickey offers garbage collection as the compelling reason that many cpp programmers found Java attractive enough to switch.[1]
In the early days, and to some degree today, Java programmers felt the speed hit of JVM managed code in comparison with managing memory directly in their program. Thus, speed of execution has always been a intercommunity criticism - even though a significant fraction of applications are constrained primarily by other things...like getting written in the first place.
The rather understandable misidentification of Java and the JVM painted the language with the features of the machine. Java has not improved its performance over the years (though it has perhaps become more expressive). The JVM has, on the other hand, been improved in terms performance.
The bytecode is complied to native code by hotspot after a few runs but C++ still has a big advantage with direct memory management and numerical functions. Though for most applications these aren't a necessary requirement.