Don't let the FUD fool you. It's "active" in that you can see other people's buzz posts and you get a buzz item on the left menu, but until you create a public profile none of your information (such as follower/followee lists) has been made public.

For you (and others like you and I who picked 'no'), clicking 'turn off buzz' does seem to actually turn off the service for all intents and purposes. No info is made public. Sure, some people 'auto-follow' me, but only those who already had your email address. No real harm done.