Because everyone wearing plastic fabrics is somehow better?
The moral benefit comes from not farming animals to only use their fur. It's cruel and wasteful, and we have synthetic fur that's actually better than the real thing for warmth. The reason real fur is still used is because it's cheaper, which should tell you a lot about the conditions the animals are farmed in.
Since neither of us are giving an empirical analysis, it's just one opinion vs another.
But definitely buy less clothes, take better care of them, and make sure to give/sell the good ones you have but don't wear to second hand shops instead of just throwing them away.
[1] Compared to viscose etc.