Exactly. People didn't care much about the granular stuff, just that only their friends could see what they were posting. With the changes FB has made to its settings (namely, the 'Everyone' setting that is now the default), that's no longer true.

For what it's worth, I was in the exact same group as you — Facebook came out while I was in college and I joined the day it came to my school. Exclusivity definitely played a part in helping it catch on, but I think Facebook's reputation as a "safe" site is what helped it make the jump from high school/college kids to everyone else.