This isn't a clear "everyone is worse off" situation. The tradeoff gets you closer to a society where everyone can participate with reasonable accommodation. Whether this level of fairness is worth the cost is a legitimate point to discuss, but it's not really fair to elide the fact that people value a fair society.