While I 100% think that SFH has often been used for this reason I think larger reason that "my house will go up in value if I restrict all other building" is more accurate. Keeping out black people was one of the early use cases with it but certainly not the only or the first.
SFH has existed heavily in New England, Canada, Iowa, pretty much every where in North America in spots where there are very few black people. Even heavily black neighborhoods like Baldwin Hills (aka Black Beverly Hills) historically had quite a bit of single family zoning. I think if that thesis was true you'd see large historical discrepancies with SFH in different states but you really don't.
I grew up in a suburban white town next to a city that was 95% white and our town had a lot of NIMBYs who prevented condos from going up. I do no think it was because they were cool with 95 white people moving in but scared of 5 non-white people. I think they just knew that it was bad for property values (for supply/demand reasons) + property taxes (for budgetary reasons) and were selfish.
The only reason I push back on this is because I don't think its helpful. I live in LA and the most common cited arguments against SB9/SB10 was that it would hurt "middle class minorities in south LA" and the representatives of these areas all voted against the bill (presumably because their biggest donors are black+hispanic+white landowners in these areas)