If textile manufacturing became 100% local again, local wages would also rise by a similar amount. $400 wouldn't seem unaffordable for a sweater. Or the sweater would cost correspondingly less.
Note that owning fewer pieces of clothing overall doesn't significantly affect your yearly cost of clothing, it just affects your initial cost to fill out your wardrobe.
What drives up yearly costs is that most new low-cost clothing doesn't last for many wears - it either wears out quickly due to being of relatively low quality, or is eventually donated/thrown out after languishing in a closet for years.
If we could have cheap manufactured goods and also pay living wages to the workers who produce them, we would not have outsourced.
You can produce cheap goods with cheap labor (or robots), or expensive goods with expensive labor.
Maybe sweaters will be $200 and wages will rise enough to be able to afford one every year. But we won't have $3 sweaters on Alibaba. From an environmental perspective, having more clothes than we need at dirt-cheap prices isn't all that great.
Some people may call this a reduced standard of living, and they are right from a reductive viewpoint where less stuff = lower standards. But isn't there more to quality of life than filling your closets with cheap shit you don't really need?
I understand that you think it best if the government force me to support my inefficient local textile manufacturer, my local cotton mill, and my local cotton grower. However I don't think that's best.