The race to slavery connection was more an accident of history than any design. Early America had some black slave owners and they imported from Africa because Native Americans tended to die from disease or escape.
NYC did see slavery from 1655 (fist auction) to 1827 but issued a gradual emancipation act in 1799. Hardly hundreds of years of institutionalized slavery. Even before that there was various waves of freed slaves associated with the Revolutionary war. When black men made up one-quarter of the rebel militia in White Plains and where then freed after the war etc.
Source: https://www.washingtonpost.com/news/made-by-history/wp/2017/...
Also, segregation spread from the South after the Civil War so there was a significant gap from the end of NY slavery to the start of Radical segregation which means it was not a continuous period.
Remember the Civil War ended in 1865 slavery absolutely ended in NY in 1827, but was almost non existent at that time long having been phased out.
However, at the time women where not allowed to vote at all. Thus it was not pure racial discrimination.
Just for perspective North Carolina and New York did allow freed blacks to vote in 1790. And in 1860 two states still had property ownership requirements to vote I am saying deep cultural views are a more complex topic than you think.
PS: As one historian put it, economics pushed poor white farmers to abolitionism before the civil war. However, economics pushed those same farmers to discrimination after the civil war.
Look at the 16 or so classes of the "sistema de castas" that the Spanish American areas used., for example.
PS no hispanics were ever enslaved in Canada and the United States.
Indians who are deemed lower caste are converting to Christianity to escape the tag.