I think there are a few fundamental problems with FPGA compute offload. First is that, as niche products, FPGAs are always a node or two behind the leading edge, so their logic gates run slower than CPUs and GPUs; even their "hard" compute blocks are not as fast. Second, the fundamental nature of FPGAs as a "sea of logic" mean that routing delays reduce your max frequency, or make pipelining necessary thereby incurring latency. Third is memory; historically, FPGAs have not supported high-bandwidth GDDRx, and if you're crunching on something you generally want bandwidth. The latest high-end FPGAs do have HBM but they are quite expensive.
So why would anyone want something that's slower and more expensive? Well, you have to be doing something special, like a lot of custom parallel processing pipelines, or with hard realtime requirements. Basically, a niche, and that doesn't lend itself to economies of scale.