At larger nodes, as far as I know the latency was mostly dominated by fan-out, which is how many inputs an output is connected to. The more inputs the larger the total gate capacitance; to compensate you can make the output transistors larger, but that adds its own delay. So if you need to fan out to a lot of inputs, you'd end up chaining stages and adding more delay.

Typical numbers for designs going for higher clocks (read: Pentium 4) would be 16 FO4 stages per clock. So of all you're doing is moving data from point A to point B on the chip, that still gives you quite a bit of leeway for adding buffer stages without having to add an extra cycle of latency.

Things have changed though, as other commenters have pointed out. These days everything is a lot more complicated, as you get to smaller nodes.