Slavery made a lot of individuals quite wealthy without doing any real work, but it's a poor system to build a capitalistic economy on. A consumptive economy like ours needs as many people as possible to have money and to spend it.
Slavery is one of the defining stories of the founding of America, but ending slavery and increasing equality has allowed the U.S. economy to go much further. Additional actions to further increase equality and get more diverse economic participation will only help the U.S. economy.
The U.S. also has a reprehensible history with American Indians and did not properly compensate them for land. And the U.S. doesn't always have the best past with immigrants to this country. But, again, getting more people involved in our economy has only helped the U.S.
It's one thing to say that the founding of America was built on the plunder of nonwhites; it's quite another to say that it's current wealth is based on that. Engaging more people in the economic system is precisely what allowed the U.S. economy to get so big.