Public transportation is not necessarily all that "cheap," it's often subsidized directly or indirectly. Cars are increasingly taxed (inc. fuel tax).
It's hard to legitimately say which is cheaper. Buses don't need their own infrastructure, but they do need roads and these are often funded by various car taxes. Still, they often need subsidies to run. Many also have subsidized rates for poorer people on top of that.
Transport is generally expensive. I don't think it can be taken as plain fact that driving cars is the most expensive method, especially outside of dense urban areas. If you gave those same subsidies to poor drivers, many would be able to afford to drive.
Excluding taxes, driving a low cost vehicle has a fairly decent cost per-km. A litter of petrol can take you 15-20 km in an efficient, small car. Untaxed, that's <$1. If you travel 500km per month, that's about $30. Add $110 for purchase and repairs (cheap car), $40 for insurance and we are at $170 per month. $5.50 per day. Pretty close to the price of public transport in many European cities, maybe less.
This is driving relatively little, but most car owners just consume more transport (travel more/farther) that public transport users. Also, more than one person can ride.
It also doesn't take into account for infrastructure costs though, but public transport's purchase price often doesn't' either.
I'm not saying cars are better/cheaper, just that it's not a clear win for either mode. Transport is expensive. If you have no money, you can't afford much of it. We subsidize public transport for poor people and could do the same with cars.