It is my sincere belief that African-Americans or Latinos if they were left alone would do much better.
Consider an alternative: those non-white races who predominantly arrived here as a result of passing modern immigration requirements, often having secondary education and a drive to succeed, tend to be more successful.
I technically agree with your statement, though. If we had never pushed out the people who were here before us, and never enslaved blacks, both populations would probably be doing much better.
The immigration system of the old days wasn't meritocratic; it was simply exploitative.
And really, so is the immigration system today, except that it exploits largely by keeping the threat of deportation hanging over the immigrant's head (including, yes, under the H1-B program for white-collar workers).
in the days of Ellis Island and had just enough education to work as bakers and butchers
So in other words, for the time in which they lived, they were pretty skilled.
The immigration system of the old days wasn't meritocratic; it was simply exploitative
That's irrelevant to the parent comments claim that societal meddling has left some population groups behind. It's also irrelevant to my point distinguishing between people who were already here and pushed out, or forcibly brought here, compared to population groups primarily composed of modern, legal immigrants.
However, it doesn't matter. Coming here in "the days of Ellis Island" required a tremendous amount of dedication and daring, and the places they were coming from were likely even worse off. It was self-selecting for driven people.