I doubt people are presuming that you smoked because you've been to college. That's not a common assumption, at least in my circles.

Regular pot smokers have a certain vibe. Not necessarily red eyes, but a way of being very casual and slightly disconnected. If you disagree with me, then think about how you 'just know' when someone is a pot smoker. There are subtle but telling mannerisms.

Which tells me that long-term pot use has an effect on personality.