The other is the group of people who might ride the bus if it were convenient. Not just in terms of accessibility to a stop, but also accounting for the journey time. If someone tries riding the bus and finds that a 20 minute drive becomes an hour with stops every single block, they might never ride it again.
In most US cities (outside of the few big ones with decent transit), public transit is basically treated as a welfare service for those who cannot get around by any other means. Not saying that this service doesn't have value, but making all decisions in that mindset isn't going to attract more ridership from those who could choose to drive instead.