Right on. I quit Facebook about 2 years ago for the exact same reason and never looked back. I think you're on to something here. Network effects work both ways. If each new user on a platform compounds it's utility and desirability, then the reverse is also true. This means even a relatively small, but non-trivial, number of people shunning Facebook can drastically reduce it's desirability and power.
I saw a little tiny glimpse of this when an organization I'm in wanted to use Facebook to schedule events and communicate. There were perhaps 30 of us in the group and myself and one other person spoke up and said we weren't on Facebook, so we would either need to be emailed the events or simply not participate. The head of the org decided it was easier just to send everyone an email. So two stubborn people out of 30 changed the way an organization decided to interact with it's members.