Charging only those people who possess the means to drive, AND those people who choose to drive, sounds like it could reduce the harms to those of lesser means.
Right now, if I cannot afford to own a car and use my bicycle to get around town, I pay for someone else to use roads, because my taxes subsidize those roads.
So, if we paid for roads not out of general taxes, but out of tolls, it seems like it could be cheaper for those who don't drive.
Also, for what reasons do you suppose that demand is inelastic? I'm not sure what mechanism you're assuming is not sensitive to price.