This is like saying "stop making everything political." It's not even possible.

Everything is a statement, the lack of intention can be doubly so. There are entire utilitarian schools of thought in fashion just like there is in everything else.

Also fashion doesn't exist solely to please others! I wouldn't dismiss an entire massive industry as inane. Vanity is a big part of it like a lot of other things, but if you're making a conscious choice in what you wear... fashion is playing some small role.