Plus the expense of maintaining two different wardrobes.
A bunch of suiting trends come from this. Suit jacket with ruined pants? Now it's a blazer, so, casual wear. Handmedown worn 3-button suit that you're wearing casually at college, with the top button pretty much permanently unbuttoned? You've just invented the 2-roll-3 jacket style. And so on.
We've replaced this behavior with simply having all our clothes be disposable-cheap. Mending a hole and adding patches to keep wearing something's not really worth it when it cost $20 to begin with.
To defend myself a bit, though, most conversations I see around this topic are referring to comfort.