In theory, an application with a computation-heavy task could program the FPGA to provide part of that task in hardware (think the hot innermost loop body).
What I am worried about is the infrastructure that is needed to make this happen: Is there even support for this in our compilers? What would support look like?
This is absurdly far away; it reminds me of the decades of assumption that 4GLs are going to obsolete programmers or the decades of trying to cross-compile C to FPGAs, badly.
It doesn't help that there's a huge infrastructure barrier caused by closed tools. Imagine if Intel brought out a processor with a proprietary instruction set where you were only allowed to use their FORTRAN compiler (no C, let alone anything more modern or JIT) with a per-seat license. That's where FPGA tools are.
We won't see a Cambrian explosion in FPGA tooling until they are made properly open. Building using open-source tools needs to be actively supported by the manufacturer.
There are also conceptual obstacles; FPGAs are sufficiently different that programmers have to re-learn and re-write idioms in order to get usable results. It's as big a jump as going from Javascript to CUDA.
I agree with you about the problems of closed-source - but I feel like the end of Moore's law will encourage creative solutions to performance problems - which FPGAs would at least make technically possible. And, even if it's like the old days when people bought compilers and access to source, people will still do it - since it'll be the best way to get an edge on the competition.
Here's one of the few references to that project that I can now find online, a WIRED era piece with all the gushing nerd optimism of the pre-dot-com-bust: https://www.wired.com/1999/02/the-super-duper-hypercomputer/
I suppose this is where Intel is headed now with these first few tentative steps.
FPGA offload has been a very active area of compiler research for more than a decade. I like this paper for example https://dri.es/files/fpl05-paper.pdf
In many applications data access is the bottleneck, and an FPGA coprocessor will not give any improvements in that area.
Using an FPGA to do arithmetic computations (e.g. as in audio/video coding) could provide a benefit, but I fail to see how this improves things much over having an extra CPU core.
The only thing an FPGA does is replace registers by wires, basically, and there's not much to be gained there, I guess.