Hiring is more difficult. It's not that you can't find applicants, but the people capable of performing without direct supervision is (mostly) a subset of those capable of performing in an office. It can be difficult to identify those who will actually perform well in that environment. It takes some degree of maturity and responsibility to provide value as a remote worker because there's always that temptation to do something else.

That said, I wouldn't trade this lifestyle for anything. The only complaint I've ever had is that coworkers not capable of performing in a remote environment sometimes last too long at the company.