Actually, it's much, much slower than that.
The Z80 takes at least one clock cycle to add or subtract an 8-bit number (sometimes several clock cycles, depending on addressing mode). For a 32-bit number those clock cycles add up quickly.
In contrast, a modern 3.5GHz CPU can add or subtract several 64-bit numbers every clock cycle
And don't get me started on multiplication or floating-point arithmetic. A modern CPU would be thousands times faster, even if it was running at the same clock speed.
Thus, I think that the feat is even more impressive than presented!