Doesn't that basically describe access to all scarce resources?
If you don't like the idea of money being used as a way to allocate scarce resources then another way to look at it is forcing people to pay for negative externalities (traffic, pollution). And I don't see why poor people should have to pay less for creating the same negative externalities.
> Why should my taxes get used to build infrastructure that's going to be subjected to congestion pricing that prices me out of using that infrastructure?
I think the arguments here are
1. Rich people pay a much higher percentage of the cost of the infrastructure. If you're so poor then you might not be paying for any of it anyway.
2. You still benefit from the infrastructure - fire trucks, police cars and deliveries are all using the roads to your benefit, even if you don't even drive on them
3. This is very similar to someone saying "why should I pay for roads when I don't own a car?"
4. It's also similar to "why should I pay for schools when I don't have a kid?" These things better society as a whole even if you don't use them directly*