I was well into my twenties when I started programming, but I found the logic very natural - if this do that, otherwise do something else.
A friend of mine was skilled in natural languages (she spoke several languages reasonably well) and wanted to learn programming (it's just another form of language, right?), but she gave up after a semester - she found programming languages to be hopelessly nitpicky with nonsense rules and no relation whatsoever to languages used to communicate between humans.
For some segment of the population the basic premises of programming come easy, and it's just a matter of spending time enough to learn the details.
Others struggle mightily with the same basic premises, it seems like they can't really switch to the "extremely narrow minded one step at a time and you can't substitute uppercase letter O for numeric zero even though the look the same" frame of mind that is a prerequisite for doing any kind of programming in any language.
It's possible that programming is learnable by everyone, but it certainly seems to come more natural to a few.
I suspect that the difference is more deep seated than we normally give it credit for, possibly even somewhat analogous to the mix of "fast twitch" and "slow twitch" muscle fibre (http://sportsmedicine.about.com/od/anatomyandphysiology/a/Mu... ) that can give someone a natural advantage in certain kinds of sports.