Yup. It's important to note that while slavery was bad economically for the South as a whole, it was certainly good for the tiny minority of rich white elite.
Interestingly enough, a virulent white supremacist at the time [1] pointed this out, and the South banned his book. He hated black people, but at the same time made the economic analysis that slavery was a tool for white elite to oppress not just black people but also poor white people. He wanted to end slavery not to benefit blacks (he wanted to build a railroad to send black people to South America), but to benefit poorer whites.