Exactly. I think it boils down to the fear that we don't know when the camera is active. When we're holding out our smart phone in a certain way, people know we're taking a photo. Google could build glass with a bright flashing on-air light, which would help, but absent obvious physical gestures, we'll wonder wether the device has been rooted to disable the light.
Don't get me wrong, I do think its unfortunate, because there are real positive sides to the device, and you touch on them in your blog, but for me, unless I feel better about the hair raising on the back of my neck, I'm going to be hesitant to accept the devices. (Caveat: I'm a bit of a privacy nut, that has never used Facebook, twitter, uses brivate browsing whereever possible, uses SMIME to encrypt most of my emails, etc., so my experience may not be very indicative)