Well, one standard deviation is far more than just "a few IQ points". One general problem is that much of this stuff tends to be very US-centric. If you look at British data related to poverty [1] and the IQ of children [2], you get a somewhat different picture. You even have two different black populations with shared genetic ancestry, but different economic outcomes and different average IQ. (And Black Caribbeans weren't doing as well when they first arrived, but a few generations later, the gap seems to be closing.)

More importantly, as pointed out in an article by Kevin Mitchell [3], a professor of genetics and neuroscience in Dublin, we do not really know of any plausible mechanism by which such differences could have evolved. In the end, if you want to do science, you have to propose a theory by which an effect can be explained and then test that theory. What Murray & Co. have never done is to provide a testable theory that explains how cause and effect are supposed to be related.

[1] http://www.poverty.org.uk/low-income-and-ethnicity/

[2] http://www.unz.com/jthompson/intelligence-of-5-year-olds-in-... (not keen on citing James Thompson here, but I didn't find an open access source for the data)

[3] https://www.theguardian.com/science/blog/2018/may/02/why-gen...