What make sparse matrices harder to JIT is proving that the loop bounds will not be exceeded, so all accesses re bound checked. In any case in my experience array bounds checking and escape analysis never gave the boost that theory and JVM fans promise. So even normal matrix multiply will trail behind. That said Hotspot JVM is possibly one of the most optimized VMs we have got.

A structural problem of JVM is that its runtime semantics is over-specified, there is very little room for the JIT to do its stuff. For example function arguments are evaluated right to left, there goes an opportunity for parallelism.