Off-hand the following categories occur to me: co-workers, classmates, random people at user groups you attend (or any other club-like group), family members. In all these cases, you don't have control over the complete set of people you interact with, so "wanting to be in the company" of them is not relevant.
You're correct. Just because I've worked with someone it doesn't mean that I want to hear anything from them. Or talk to them.
I'm much more restrictive about who I spend my face time with than who I'll accept a FB invite from.