Because usually it's people that do shady things, not bad things maybe, just things that people don't want you to see, no matter if it's just trading pot or else, or even if you have the best intentions. If you do need/want to hide something, people will assume you are doing something bad, and therefore you are a scumbag. Personally I don't care but this is how the world usually works.