This is something which always gets me.

So it is widely accepted that there is no/negligible genetic difference, but there are cultural differences at least when it comes to prosperity.

However it is also the case that cultures are extremely slow to change - multiple generations - almost to the actual speed of evolution. Essentially conceding the point that even if it isn't genetic there is some kind of heritable negative/missing positive traits which can last multiple generations.