A friend did some research in this area, comparing FPGAs, CPUs and GPUs. He published a paper [1] in regards to performance for several common Linear Algebra computations across a variety of input sizes. In particular, Figure 2 shows you where each of the platforms works best.

FPGAs are essentially re-programmable hardware, so they tend to outperform CPUs/GPUs when you program them for a specific task. They don't have to deal with most of the overhead that the more generalized platforms deal with which is why they dominate in the small input sizes. However, with FPGAs you're trading space (silicon) for that re-programmability so you can't have as much hardware in the same area as say a GPU. Thus, when the data sizes have saturated the available hardware of the FPGA for computation, the GPU begins to outperform. Due to the decreasing node sizes (28nm, 22nm, etc), we can fit more programmable logic into the same area, which causes the chart I mentioned above to shift more into the FPGA's favor.

[1]: http://www.researchgate.net/profile/Sam_Skalicky/publication...