It's the /services/ that Glass secretly enable for its users that I find far more worrying than mere scrapbooking of your life. Just start thinking as sleazy as possible here, for a moment. If Glass can log conversations, it can also relay conversations to other parties to comment upon.
I just can't stop imagining things like MTurk-like services staffed with popped-collar sociopaths fresh off the set of Jersey Shore, all working with your transcripts to be your 'virtual wingman' as you work your 'game' on unsuspecting people. Ridiculous, but just sleazy enough I could see it.
I imagine if we all really just wanted to depress ourselves today, we could keep coming up with more and more ideas of how to exploit AR for social advantage/manipulation.