That is a self-reinforcing cycle. There have been long and successful campaigns by car companies and other self-interested entities in the US to associate public transportation with being poor. Just like how a city street is safer per-capita if there are more people on it, public transit is safer if it is more well-used.

I see this in seattle. When I am commuting in the morning or in the evening my bus is full of yuppies and working class people getting to their job. But if I take the bus on the weekend or during the off hours when well-adjusted people are not on it, the bus is a much less inviting place.

I don't know how to solve the problem other than to believe in the system and hope that other people do as well.