I used to have a garden office but my wife now needs her own space for her work so I took the rented space.
A dedicated space is a must if you're working remotely. Either at home or offsite.
I have no objection to going into the office for well defined, productive meetings like project kick-offs though. I still believe they're the best way to collaborate on big planning tasks, even after having worked remotely for over 10 years.