I did some digging because I was curious.

From the 1860 census [1]:

Total Number of slaves: 4m

Total Population: 31.4m

Percentage of population as slaves: 12.7%

From the 2010 census [2]:

Total Population: 309m

Total Incarcerated Population: 7m [3]

Percentage of population incarcerated: 2.2%

OP is absolutely not correct, even if we are generous and assume every incarcerated person is forced into labor.

[1]: https://en.wikipedia.org/wiki/1860_United_States_census [2]: https://en.wikipedia.org/wiki/2010_United_States_census [3]: https://en.wikipedia.org/wiki/United_States_incarceration_ra...