This could backfire if you ever plan on selling your house. Many prospective buyers will check the streetview, and the surrounding neighborhood.
I wonder if Google could/would implement a toggle option, so that homeowners could turn Street View on or off.