Sorry what's your contention? or do you have one?

Backprop is backprop on CPU or GPU - it just runs (way) faster on GPU. So while you can do some stuff with NN's on CPU, you can do more stuff with NN's on GPU.

Further, the sources you reference are from before advances made in software and hardware for massively parallel GPGPU architectures (CUDA), so it's kind of a moot point anyways. Those researchers may have had access to such a system, but you definitely couldn't just `pip install pytorch` on your gaming GPU in the 90's...