Exactly right, it's the potential abuse in the future that's scary. In 30 years time I see it playing out something like this:
Potential Politician: "I want to run on a platform of reform, government transparency and freedom of speech"
NSA: "We have these emails you sent to a friend in London when you were a student 20 years ago, where you intimated how you slept with another woman than your girlfriend, now your wife..."
PP: "I resign to spend more time with my family"
All it takes is for one bad apple to come along and really screw things over for everyone. Almost everyone has something that could be used against them in the court of public opinion. Mass surveillance just primes the pump for abusive individuals to blackmail the world.
Do you want your government run by elected individuals or the unelected intelligence agents who are blackmailing them?
PS: Pre-snowden, I'd have said this is crazy conspiracy theory talk. Nowadays, I dunno...