They don't for fp32 or int8 workloads until SVE IMO. High end Skylake as an example has 2 AVX-512-F ports (and not considering AVX-512-4FMAPS), which give you do 64 FLOP/cycle (512 / 32 * 2 * 2). Even 2 NEON units, assuming FMA's execute in 1 cycle/unit, gives only 16 FLOP/cycle. For ML inference workloads you can even get away with lower precision and even do away with floating point (which helps motivate AVX-512-4VNNIW).