This is wrong. Training data can be streamed through GPU memory during training. It's your parameters that can't exceed GPU memory. You can get GPUs with 12 GB of memory, and they also support float16 so they can be twice as memory efficient as CPUs. If your model has more parameters than that, then you'll be waiting months or years for a single model to train using CPUs, even distributed.

Furthermore, almost any technique you use to distribute and scale training will work just as well regardless of whether the computations are happening on CPUs or GPUs.

It's not trivial to take any existing algorithm for training and scale it not only cross-GPU but also cross machine. Training tends to be very bandwith intensive. In fact that is the entire reason Nvidia is so heavily focused on this with their upcoming GPUs (Pascal). I am not saying it's not possible, just that it's a lot harder than "distributed makes it scalable".