Thanks for the tip. I'll see where I can apply it.

Most of the time is usually spent in the convolution layers. Convolution is not a matrix multiplication in the current implementation. I guess it would be a matrix multiplication in frequency domain or by using a Toeplitz matrix.

I've implemented a CPU Parallel version and gave a try at GPU implementation. But I'm not satisfied at all by the GPU version :)

https://github.com/cbovar/ConvNetSharp/blob/master/src/ConvN...

https://github.com/cbovar/ConvNetSharp/blob/Gpu/src/ConvNetS...

Pull requests more than welcome!