1. Lack of awareness. In many of these environments "software engineering" doesn't really mean anything to anyone, especially if few households own computers in the first place. Growing up in Detroit, my Dad lugged in our first (huge, ancient) PC when I was in the 4th or 5th grade - if I had simply been pointed in the right direction, I could've started my learning much earlier. I remember lots of kids in our middle school getting super interested in HTML/CSS, but only having access to PCs for an hour or two after school.
2. Learning ability. Obviously not to say there aren't any smart underprivileged people, but it is not often made clear that one's ability to learn is, in itself, a powerful asset. I believe this is the strongest factor in one's ability to program outside of personality inclinations, and that anyone can increase this capacity (within whatever local spectrum their personality and lifestyle allows).