It really depends on your office environment. In most companies in the bay, a hoodie and jeans are completely fine and it will likely be distracting if you dress 'too nice'.

Take cues from how most people dress in the office and target that level of formality. If most people are t-shirt and jeans, just stick with what you're comfortable in. If most are jeans and a button-up shirt, do that.

Yeah, it's conformity, but if you're concerned your attire is affecting your perception, the easiest way to fix it is to match everyone else.