Fascinating. Wish he could have shown benchmarks on a larger image database (Imagenet or CFAIR-100), as mnist is extremely easy to train on. Great to know, especially the LSTM benchmarking.