For you (and others like you and I who picked 'no'), clicking 'turn off buzz' does seem to actually turn off the service for all intents and purposes. No info is made public. Sure, some people 'auto-follow' me, but only those who already had your email address. No real harm done.
I don't care if it was made public. If I had a profile, and it was made public, that would not have made things better. That would be worse. I should not need to understand the subtle privacy interplay of several different and seemingly unrelated Google products just because I use one of them (Gmail).
I entirely agree with this, although I'll point out that IIRC Google Chat did the same thing in terms of modifying gmail but no-one seemed to really object to that. Buzz seems to be different because of the privacy issues, i.e. Google didn't communicate well what would happen (specifically a public profile that included your follower/followee lists) if you clicked "yes". That there's a new link on the left-hand menu can't really be the main issue, is it?
As far as FUD, I'm specifically referring to comments like this:
There is absolutely no way to opt out of Buzz.
http://twitter.com/etherial/status/9023625277
It IS possible to opt out of Buzz, ~ its just that its counter-intuitive and once you've hit the "Yes" button, somewhat difficult to undo.
Because in combination with the default of having your follow list public, that exposes who you've been emailing to the world.