> Why aren't there e.g. 16 GB of L1D$ inside our CPUs? It'd be blazingly fast! :)

I didn't say the memory necessarily has to be moved into the GPU. The other way around would also be a possibility.