How can we quantify "not actually much"? Here's my attempt for multiply:

You'd have to do a imul instruction followed by a right shift. On, say, Ice Lake, the imul has a reciprocal throughput of 1 and a latency of 3. The shift has RT of 0.5 and a latency of 1.

So compared to regular integer multiply, a fixed-point implementation on Ice Lake would be 0.67x the speed if you were throughput bound and 0.75x the speed if you were latency bound.

But most (all?) other operations wouldn't be any slower than regular integer operations, so a real program would slow down by less than the figures calculated above.