The performance difference between CPU vs GPU is very substantial: the consumer-class Nvidia GTX 980 I use can deliver about 4612 GFLOPS [1] of raw performance, whereas the i7-6700k CPU I've got can only deliver about 113 GFLOPS. Whilst those numbers aren't really comparable (and hence the difference is not as large in practice) you still get a substantial speedup, maybe 10x-15x (this is only especially important during the training phase). The insight I've gained is that the kinds of APIs that Google/Microsoft/IBM give you are definitely not magic and they _can_ be replicated + tweaked to a reasonable degree, and actually I think the concentration of natural language understanding / visual recognition / machine learning and the datasets needed to make them work into the hands of a few very well-financed corporations is both a good thing (because it lets you get up and running quickly and the predictions are continuously updated) and a bit of a bad thing (since these types of applications are going to become a more essential part of computing, and for reasons of cost, performance and customisability). I've also written a bit about my experiences with Theano versus TensorFlow[3].
[1] http://techgage.com/article/intels-skylake-core-i7-6700k-a-p...
[2] https://en.wikipedia.org/wiki/GeForce_900_series
[3] https://medium.com/@sentimentron/faceoff-theano-vs-tensorflo...