Amusingly, I've noticed the same thing: I do claim that a surprising amount of things are simple. And you know why? Because they are. A whole bunch of things (most recently F-algebras and F-coalgebras) really are simple: you start with a very simple concept (say sets and functions) and build up to them, taking very simple steps along the way. Maybe you have to bring in a couple of other concepts that are similarly simple.
EDIT: I should have made this clearer: I say something is simple before trying to explain it rather than instead of. I mainly do it to avoid scaring people without reason, and everyone gets far too scared of words like "F-coalgebra" or even "monoid".
Every single step along the way? Virtually trivial. But put them together and you get something genuinely difficult to follow.
It turns out that enough trivial steps all building upon each other becomes difficult to follow for anyone. Yet, once you've finally understood it, you can't see why it was difficult any more: after all, all the components are so simple, and it all makes sense! (Now.)
I realized this in a pretty direct way: I was trying to explain the aforementioned algebras to some of the smartest people I know--people with much more mathematical experience and aptitude than me--and they all found it a bit hard to follow. Not because they couldn't follow any particular step, but because putting it all together in your head is difficult.
So really, the ultimate conclusion is: just because something is simple does not mean it is easy. Having some trouble with simple concepts like this is not an issue at all; everybody does. But that doesn't mean they aren't simple!
Another, more surprising, conclusion is just how shallow the stack of concepts has to be to cause people trouble. It's not like there are hundreds of parts in these definitions: there are maybe 10. (Say: set, function, algebra, identity, category, morphism, functor, dual.) And yet this is more than enough to confuse.