Also, in some cases like small RNN/LSTMs, CPU's can be faster.