It's not the execution of FMAs that's the challenge, it's the ramp up / down.

And I assure you GPUs do have challenges with that as well. That's just less well known because (1) in GPUs, all workloads are vector workloads, and so there was never a stark contrast between scalar and vector regimes like in Intel's AVX-512 implementation and (2) GPU performance characteristics are in general less broadly known.