I've been able to take any day off I want, but I have NEVER been able to switch off. It took me about 4-5 years to finally be relatively OK with it. Annual leave is far better.
I've taken a bit of a "holiday" this year by taking a full time role for ~8 months and it's been a great opportunity to recharge and work with amazing people that are much smarter than me, but I'm already feeling the pinch to go back and do my own thing.
Having said all of the above, I'm very lucky that I'm 10+ years in the software industry and I've always got the option of a great full time role to fall back on.
All in all, so far I'm not sure it was worth it. I wouldn't have been as successful and fulfilled if I kept my full time role, but I'd be a hell of a lot happier.