What made the difference for me were two things.
1) Parents that were role models. My mother didn't earn much money, but she had a tremendous work ethic (she grew up on a farm), and was a very disciplined person in terms of morality. She was always present in my life, and was quite firm and consistent in terms of discipline (if you did something wrong, you were punished; and you listened or else). My father was an entrepreneur, very creative, very intelligent, high ingenuity, exposed me to a lot of amazing experiences growing up (traveling, ideas, technology, etc). Neither parent drank, had drug problems, or had ever been arrested. I had a consistent, safe home environment, where being smart was encouraged, independence and success were outcomes to strive for.
2) A tremendous public education system, with high quality teachers across the board. Almost all of which were older women interestingly, most were very firm, attentive, and persistent. The school itself might have had several hundred students, but from K through 5, I had one male teacher out of maybe 15 or 20, and only one younger female teacher.
The kids that did the worst in school, almost universally had very bad home lives. They were not only extra poor, but they had terrible parents (or absentee parents); abusive, alcoholic, non-working, etc types.