About pointers being difficult...I don't think that pointers are difficult to understand. To me it was more the syntax C uses that got in the way.
I do teach programming, and most of the time I see both the concept of pointers, and syntax is pretty understandable.
The real trouble begins when you expect students to use them to their advantage - but it really is beyond syntax.
As a newb, you get bogged down in the details and knowing the cases for when to use &,* seems like an added complication.
If you have a pointer (that is, the address), then you need to prefix it with a * in order to get the value in order to do useful things. If you have a value, you need to prefix it with & in order to get the value's address in order to pass it around more efficient (at least it will be more efficient if it is some large data blob).
Did anyone ever mention addresses and values and their difference when looking at Java from a users point? Not to me, to be honest.
Pointers are simply the first thing that forces most coders to consider that split. But, a reasonably competent JAVA developer moving to C can pick them up in little time. The problem is reading other peoples C code that looks more like line noise than structure. But, pointers are a tiny step along that path.