In the same way that most people don't want to pair program full time, but it's perfectly legitimate for a company to decide it wants to be a full-time pairing shop if they think it's important enough to limit their hiring pool to people who want to pair program, it's perfectly legitimate for a company to decide it does not want to be remote.