Get rid of the unsafe factories and poor people won't be able to afford clothes anymore. And the workers of those unsafe factories won't be able to afford anything.
There is no easy answer.
poor people have always been able to afford clothes, it was called saving for a while to buy a good pair of pants and mending it for many years until it pretty much fell apart.
Poor people would not be able to afford new clothes all the time, sure, but clothes in general most certainly.
And let's not forget about thrift stores, go into any goodwill and you will be able to buy plenty of good quality extremely lightly used clothes for very little money.
The benefit they gain through slightly cheaper clothes is more than offset by the benefits from the jobs they could have had if the manufacture of clothing (like everything else) was not offshored.
The real winner is Stefan Persson and the $28 billion he made from H&M's sweatshop labor arbitrage.
"There is no easy answer."
There is. Make trade deals contingent on good labor and environmental conditions and ruthlessly enforce that. That will bring jobs home and improve working conditions in 3rd world countries.
Then, deal with the fact that that new top is going to cost $4 more and Stefan Persson won't be quite so spectacularly wealthy.
Most people ignore this subsidy from poor countries to rich.
Even if they can, the extra money they would have to spend is money that they won't be able to spend in food and other essentials. Time they spend making clothes is time they can't spend making money for buying other things, or in education.
Replace or just wash? The latter has been absolutely common across many cultures that had access to water and hot enough climate to dry clothes quickly.
Sturdy cotton lasts 10 years or even more.
Ever heard of a thrift shop? Hand-me-downs?
There are plenty of ways for poor people to get clothes that don't involve buying stuff produced in a Bangladeshi sweatshop.