There's nothing shameful in being associated with Wozniak or Draper or any of the great technological innovators.
There is something shameful in being associated with the "l33t h4x0r kr3w" (or Iranian Cyber Army) and those who participate in mass defacements etc.
edit:
Also, I really don't like the idea that you're pushing, that you have to work 'outside the system' in order to be a hacker. I completely agree that important attributes of being a hacker are the ability to see problems from non-traditional viewpoints, to come up with unusual solutions, to challenge the de rigeur way of doing things, to disrupt. And those attributes are shared by many subversives.
But the important difference to me is that a hacker is constructively subversive.