This is the worst argument in my opinion. People can record you in private spaces using cell phones and watches without you ever knowing. Not to mention security cameras.
Glass is new form of something that is currently: done covertly (hidden cams, sneaky smartphone use), obtrusively and temporarily (holding up a camera/smartphone) or by entities that already eschew social convention for their own business ends (security).
I see the act of wearing glass as a social confrontation. A dare that says the wearer think upping the intrusiveness and pervasiveness of recording should be fine because Google said so. And the reaction to the first roll out shows that people feel similarly.
If I wanted to take a photo of you with Glass, you'd have plenty of time to turn away, put your hand up, or whatever. And you'd certainly know I was doing it.
I mean, 50 years ago it would have been ridiculous to suggest the government could transcribe and search all phone calls - consider the incredible cost of all those typists, filing cabinets and reel-to-reel tapes! If someone had said "because we already have technical constraints, there's no need for ethical or legal constraints" we'd be in a difficult position when the technically impossible became possible.
In the vein that you can buy a pen camera that'll do a much better job for $30, worrying specifically about Glass seems silly.