Hacker News Reader

Comment by im3w1l | Hacker News Reader

You should use FFT's for the convolution, see http://arxiv.org/abs/1312.5851

Also, while the original Blas was in Fortran, you probably want OpenBlas, which is in C. Or maybe Intels MKL.

FWIW, FFT-based convolutional methods aren't better for a lot of common sizes (including a bunch of AlexNet/VGG/GoogLeNet sizes). http://arxiv.org/abs/1412.7580 has a pretty thorough evaluation of im2col + gemm vs fused gemm (cudnn) vs fft.

Thanks for the article tip hadn't seen it. But I don't agree with your conclusion. In fig 1-6, you should look at the last column. Because really the other channel numbers are quite low. Knowing that, the space domain algo, only really has a chance at 3x3 filters which is quite small.

And since they claim fbfft was even faster than cufft, the situation should look even better for fft.

EDIT: And they are about tied in the 3x3, 64 case.

The current trend in convolutional neural networks seems to be moving toward more convolutions with smaller kernels. This year's second-place ILSVRC winner (VGG) is essentially a giant stack of 19 convolution layers with super tiny kernels, sandwiched between nonlinearities and pooling. "To reduce the number of parameters in such very deep networks, we use very small 3×3 filters in all convolutional layers" http://arxiv.org/pdf/1409.1556.pdf

The article actually touches on this issue a bit.

"The main competitor to the GEMM approach is using Fourier transforms to do the operation in frequency space, but the use of strides in our convolutions makes it hard to be as efficient."

For 'small' workloads, it's not uncommon to find situations where direct computation might make more sense. The author touches on that a bit, as others have pointed out.

According to Github, 51.6% of the code in OpenBLAS is Fortran:

https://github.com/xianyi/OpenBLAS

Reply on news.ycombinator.com