I find that it's usually a combination of both.
What usually convinces me that we're reaching into the "I'm just not smart enough" territory is when I listen to multiple explanations from multiple sources and I'm still stuck (although ... I find that on certain subject people are just parroting each other a lot, which makes me often question whether the person doing the explaining actually understands the topic at all).