There are societal components to such unrestricted labor movement, like displaced local workers, lack of housing, strain on public resources etc.

Capital movement is restricted - especially internationally.

Capital doesn’t bring along the culture of where it comes from or become part of the polity. Capital doesn’t need to be educated or housed and isn’t eligible for welfare benefits.

That's begging the question. We are not OK with either.