This is actually a really interesting point/question, and it's one that doesn't have an easy answer. It's true that, overall, most slaves (in pre-civil-war US) were owned by a small number of plantations. But in some southern states, nearly half the White families owned at least one slave. In those families, obviously the women and children didn't themselves "own" any slaves, but they nevertheless controlled that person's labor. Similarly, wealthy plantation owners frequently rented their slaves to poorer whites. So even a relatively unprivileged class of White people benefited from slave labor directly despite themselves getting a raw deal relative to the land owners.

But ultimately, the south fought a civil war in order to preserve the institution of slavery because it was the fundamental organization of society on top of which the economy was built, and many non-elite white men nevertheless allied themselves with this effort.

As for projecting the past, I don't know. It's certainly plausible that White people today are still reaping the benefits that were established by slavery and racism in a broad, systemic sense. As far as I know, my relatives didn't own slaves, but my relatives did do things like get mortgages that Black people couldn't do. That doesn't seem fair does it? It's a little bit like inheriting stolen property. But if you go back far enough, what isn't stolen property?