I think most people who believe they don't care about fashion do. Cargo pants, t-shirts and hoodies might not be the typical image of fashion or putting effort in one's appearance, but me wearing them at work and free time is still a semi-conscious social marker.

I don't think I put much more or less effort in my appearance than someone who's everyday attire includes button up shirts and designer eyewear, but we will signal different interests with the way we dress. They say to dress for the job you want. It could be that you don't actually base your respect on the effort someone puts in their appearance, but rather the kind of appearance you value.