The solution to these issues is to provide bus signalization, dedicated lanes, and/or queue jumping opportunities. These changes minimally inconvenience cars while significantly speeding up buses. It makes it so that a bus route, while longer, can be time-competitive with driving (or rather, closer enough that it overcomes the inconvenience of walking to/from stops and sometimes having to wait for your bus)
And having frequent stops makes buses go slower.
Traveling long distances doesn't make the bus any slower; it just limits the frequency you can revisit a stop on the route without increasing the number of buses on that route.
- Motorcycle to work: 15 - 20 minutes door to door (driving generally ~30 minutes, but 40 minutes or so during busy periods)
- Bus to work: > 60 minutes (7 minute walk, 5 minute wait, 45 minute average bus journey, 7 minute walk)
Cycling would, I reckon, take me about 45 - 50 minutes door to door until I got fitter, but that would be fine because I'd be getting loads of exercise, so the trade-off becomes worthwhile even though it takes longer. (The reason I don't cycle is there's no safe route.)
Presumably most congested roads are used mostly by cars, so I suppose you could say that "cars cause congestion and congestion makes buses slower." But of course, the congestion also makes the cars slower as well.