I wonder this quite often too. It would be interesting to see a study involving programmers and their confidence in their skills compared to what other people think and also how much they read about programming online.
My gut instinct is to say that because we want to read a lot of new/interesting stuff online, we're exposed to (seemingly) tons of people who know TONS of things that we don't know in the aggregate, so it affects our self esteem...
You may not be up to your job, but probably more likely it's that.
The most interesting thing I've seen written was on here, pointing out a problem in the logic. "If they haven't noticed you're as stupid as you think you are, they themselves must be even stupider. Therefore imposter syndrome at some point requires that your colleagues are morons, yet if you believed that you probably wouldn't feel as inadequate."
Alas my brain still believes.
Ah, the old "I'm not ok, you're not ok" section of the OK Corral[1].
[1] http://changingminds.org/explanations/behaviors/ok_not-ok.ht...