Agreed. Though I can imagine some companies reasonably applying constraints in certain areas: working hours overlap, travel costs, and (especially where they don't do the fraudulent "honest this person whom we treat daily like an employee is not one" contractor lie) compliance costs and obligations.
That may indirectly imply some geographical constraints which could affect the shape labor market and the resulting supply/demand math leading to a salary. But it's definitely not what most companies do in their cost of living adjustments.