This is mostly true, but the primitives are value types and you can get some things done with them. (Not enough to make Java good for these use cases, no.) I.e. write float[] instead of Float[] and you have a contiguously allocated region of memory that can be efficiently accessed.
What do you do in a dynamic language where no such separation can be enforced? Or in a static language where you just don’t want that kind of ugly anti-generic bifurcation of your type system? The classic static PL answer is to just make all objects have reference behavior. (Which is the approach Java takes with a pragmatic ugly exception of primitive types.) This is the approach that ML and its derivatives take, but which is why they’re unsuitable for numerical computing—in ML, Haskell, etc. objects have “uniform representation” which means that all objects are represented as pointers (integers are usually represented as special invalid pointers for efficiency), including floating point values. In other words an array of floats is represented in ML et al. as an array of pointers to individual heap-allocated, boxed floats. That makes them incompatible with BLAS, LAPACK, FFTW, and just generally makes float arrays inefficient.
So what does Julia do? Instead of having mutable value and reference types, which have observably different semantics, it has only reference types but allows—and even defaults to—immutable reference types. Why does this help? Because immutable reference types have all the performance and memory benefits of value types! Yet they still have reference-compatible language semantics, because there’s no way to distinguish reference from value semantics without mutation. In other words immutable structs can be references as far as the language semantics are concerned but values as far as the compiler and interop are concerned. And you can even recover efficient mutation whenever the compiler can see that you’re just replacing a value with a slightly modified copy—and compilers are great at that kind of optimization. So all you give up is the ability to mutate your value types—which you don’t even want to allow for must numeric types anyway—and which you can simulate by replacement with modification. This seems like a really good trade off and it’s a little surprising that more languages don’t make it.
The Julia approach is indeed elegant. An in-between position taken by F# is statistically resolved type parameters, where type safety is maintained and semantics largely preserved by constraints on members. It's a pain to write but trivial to consume though errors can be obstruse. More type level flexibility on the CLR (which seems to be planned) will further improve things when it comes to generic programming.