You're friends are not introvert, are they? I say "Yes" more to those who seem to understand an introvert more. I say "No" more to those I don't have much in common. In my case, I say "Yes" more to invitation from techies than from sport lovers.