The scary part of Glass-like devices taken to their logical conclusion for me isn't the little individual invasions of privacy (though those are also unfortunate), but the massive amount of real-time, real-world information collected in the aggregate.
If Glass-like devices are ever mainstream and controlled by one entity they create the easy possibility for an information mesh network that makes the one shown in "The Dark Knight" look absolutely quaint in comparison. Ubiquitous video + audio + advanced feature detection destroys personal privacy when in public basically across the board, even if you yourself are not a participant.
eg:
EvilOverlord: "Glass-MCP-10k, where is George McBay?"
Glass-MCP-10k: "Scanning... Subject George McBay last seen entering residential building 10.5 seconds ago at 32.925924,-117.227812. Data based on facial recognition match with 96.75% confidence. 5 Gargoyles report seeing him within 1 mile radius of this location within the last 10 minutes."
I see the massive benefits of Glass-like-devices but I am also terribly concerned about the possible privacy implications. The potential for abuses could be staggering and given that I think it is a good idea to actually discuss the potential for problems and not just assume everything will be okay because the tech is "neat" (yes, it is!, but...).