That sounds like a loss of freedom to me. How about the freedom to not answer your phone when you don't want to be bothered?
A lot of people seem to think that turning your phone off while you take a hike is as the same degree of freedom as never having any of your extracurricular activities interrupted - ever. It's not.
Are there downsides to not owning a phone? Of course. I was just commenting that habitually being incommunicado is, for me, a peaceful state to find oneself.
Possessing a phone seems to give people the addiction of reaffirming their social value with little "New Text Message" nuggets throughout the day. Or the need to cater to every correspondence whim by immediately returning calls and texts.
I think the loss of freedom is less contingent on getting a phone, and more indicative of the general social insecurities most of us share.