I think the hardest part is the planning process and communicating rapid change.

Startups and fast growing companies grow quickly and changes happen rapidly. If priorities are constantly changing and communication/the planning process isn't really robust you won't have remote people who are successful.