It was mostly because of the clock speeds, wasn't it? I seem to remember once hearing that below 20 Mhz you could play "TTL Lego" but above that you ran into problems with things like propagation delay.
So, 20 MHz seems pretty credible for a region where you would see problems due to propagation delay—maybe even somewhere below that with the original 7400 series.
That's also a reason that I hadn't thought about so much why putting the whole CPU on a single die would look more and more practical over time: propagation delay in wires!
If you had the money. Those chips weren't cheap back then (even the 74xx branch), and you still needed quite a few to do anything useful when it came to building a CPU.