The solution to this problem is to upzone that land so that more people can live there. If more people wind up living close to their jobs, that's a good thing. Commutes are bad for workers and bad for society.
But that upzoning is a better solution doesn't negate the argument that congestion pricing hurts the wrong set of people.