You're not wrong, but it's more complicated than that.
My town never recovered economically after the US Civil War. It was a fairly wealthy place before then because the main economy was tobacco plantations, which slave labor enabled. It literally had more residents in 1860 (including enslaved Americans) than at time since then.
A lot of the economic woes in the American South have very deep roots. A significant amount of wealth, pre-Civil War, was in the form of slaves. That was incredibly immoral, of course, but the economic effects of that "wealth" disappearing after the war was still a real thing. There were basically no large white landowners left, the farmers who remained were small landholders. And blacks ended up being a permanent underclass. They were also mostly farmers, but they were continuously undercut by de facto exclusion from the banking system. Also, public education as a whole was never great, even for whites, and for blacks it was pretty abysmal. The county next to mine actually closed its public school system for a generation instead of segregating, and the money that would have gone to the schools was instead embezzled by the county commissioners and funneled to a private white school instead.
Kneecapping your population like that has generational effects. The lack of investment in public infrastructure has two root causes: a significantly reduced tax base, and racism. Public services being available to everyone, that means you were taking money out of the hands of whites and using it to benefit blacks. And even though that is largely gone (imho), the effects of zero investment in public infrastructure isn't something that can be fixed quickly. We mostly missed out on the post-WWII economic boom for those reasons.
Even in the face of all of that, my county was doing okay up until the Carter-era agricultural reforms that incentivized large agricultural conglomerates and crippled family farmers. That, plus the interstate not going through town, pretty much doomed the place.