Genes may play a factor, but they are certainly not the most important one, or even a significant one in this context. If you grow up in a poor family / poor neighborhood, you won't have access to the same education as those who grow up in middle-class areas. Education is far more important to what we consider "intelligence" in this context. Basic math, financial literacy, or even reading comprehension is far more a result of your education than it is of your genetic material.

It's easy, as a product of good education and a safe environment, to look at people in poverty and say "I guess they're not as smart as me." This hides a far more uncomfortable truth, which is that everyone who doesn't grow up in poverty has access to better education, better opportunities, and a better environment than those in poverty. Try as you might to claim that people in poverty are just "stupider", it's simply impossible to make this comparison given that people on opposite ends of the financial spectrum grow up in totally different environments. It's short-sighted to attribute this to genetics and nothing else.

I'm just going to hazard a guess that you didn't grow up in poverty, am I right? If you had, I'm sure you would have a very different idea about how difficult it is to grow up without all the advantages of a middle-class upbringing.