I haven't looked at it extensively, but I don't believe this is true.
This study shows that SES (Socio Economic Status) doesn't explain much of the gap.
http://www.umich.edu/~rdytolrn/pathwaysconference/presentati...
This study (sorry, can't find a non-paywall version) shows that even holding income constant, blacks do worse than whites at the college level. In particular, low income whites perform as well as high income blacks.
http://www.jstor.org/pss/2963200
Also, even if you could explain the Black/white income gap via wealth, racism or other environmental factors, how would you explain the White/Asian gap?
http://en.wikipedia.org/wiki/File:Personal_income_race.png
And how would you explain that in spite of wildly different environmental factors (compare Singapore to Texas), there is a relatively small gap between Asian Americans and Asians? And similarly, there is only a small gap between white Americans and white Europeans?
(By the way, I'm not trying to claim there are no other factors at play. I'm just pointing out that race does seem to be a biggie.)