You'll get left out of real life. If your social circle organizes events on Facebook, and you're the only weirdo that can only be contacted by email and phone, you'll be forgotten by all but your strong friends (who don't need Facebook's friend-menu to remember you).
That might not be a problem if your social circle doesn't rely on Facebook much or you're (for instance) an introvert who doesn't care about parties thrown by acquaintances (which are a nice way to meet new people), but there are a lot of people who those don't apply to and who would have to sacrifice to avoid Facebook.