I use the Colt (Java) matrix libraries for applied math and graphics applications, but it's still quite costly compared to optimized libraries like Intel's MKL. A singular value decomposition takes about eight times as long in Colt vs MKL.
The java version of netlib produced by f2j is unfortunately the unoptimized "reference implementation", so it's not terribly performant for large matrices.