Same experience here, as soon as I dropped out of facebook, I was immediately blacklisted in every social circle that used it, even if the primary mode of communication was texting/email.
Some op-ed considered dropping out of facebook as a radically anti-social act, and I'd have to agree: you're not considering the group's views on communication, and you're forcing them into a personal world. When the entire social circle uses facebook as a communication platform, it's pretty pompous to expect it to use what a single person wants. This is my argument boiled down: expecting a group of people to conform to a single person's views is rude, and the delusion that the one person is so important that everyone else will conform is anti-social.
The majority opinion on HN seems to be that this isn't, oddly enough. The last time I stated my experience it seemed to piss a lot of people off, and there was a knee-jerk reaction that "true friends will work around it." Maybe, but weak connections are important as well, and since facebook is the place for weak-connection friends, defiantly not using it seems off and a little anti-social.
Another argument was that interaction existed before facebook, so not using it is fine. That's dancing around the point: facebook is the current defacto standard of social communication, not participating it is seen as aggravating to the other party, not that it destroys communication completely. Try telling your friends that you'll only interact with them in orkut. Or, for the Gen X'rs here, that you're dropping your phone, and you expect them to say "hi, how's the job" through post mail and in-person visits. There's a series of fall backs here that's reversed for the average hacker. The hacker sees facebook as personal, and email/phone public. The average person sees facebook as a public place, and email/phone as a personal channel.