Eh, I'm not sure where you're getting your numbers. Here's a quick test using node 0.10.40 on my machine:
mmalone$ node
> var f = function() {
... s = Date.now()
... for (var i = 0; i < 1000000; i++) {Math.random()}
... return Date.now() - s
... }
undefined
> f()
3
> f()
1
> f()
1
>
So I'm getting a couple milliseconds for 1,000,000 numbers. I'm also toying around with a PRNG package in Go currently so I have some numbers there too:
BenchmarkXorshift128p-8 500000000 2.98 ns/op
BenchmarkXorshift1024s-8 300000000 4.62 ns/op
BenchmarkXorshift4096s-8 300000000 4.44 ns/op
BenchmarkXorshift4096ss-8 200000000 6.29 ns/op
BenchmarkUint64sFromCryptoRandIdiomatic-8 200000 10736 ns/op
BenchmarkUint64sFromCryptoRandUnsafe-8 200000 8081 ns/op
Go's crypto source pulls from urandom. This is more like what I'd expect in terms of performance. The difference is orders of magnitude, but we're still talking about microseconds so I think your argument is still valid. A proper CSPRNG is not "as fast" but it is probably "fast enough" for most use cases.