I'm actually very opposed to visible branding on a lot of things, especially clothing. At first I felt like I was being a little obsessive-compulsive but then eventually realized it's an ethical issue for me. Every time someone puts on a garment with a logo, they're being used to serve the interests of the corporation and not their own interests in a tiny way. The fact that this seems so natural says a lot about the state of our society in this regard.
As for your argument, I think it's reasonable but I would turn it on its head: to me every bit of money that's going into visible branding in some way is a signal that the company is putting effort into advertising itself on the outside of the clothing rather than improving quality. I can think of a couple of companies I buy from whose quality has declined in the last couple of decades overall in certain regards, and it's been directly associated with an increase in visible branding on their goods. Basically they started cashing in on their reputation, focusing on their brand identity rather than the quality of their items. Judging from comments in various places I'm not alone in the sense their quality has declined.
My impression is actually that it is not difficult to find minimally branded high-quality goods in many areas. In fact, the highest quality goods in a some areas are those with minimal or no branding.
I admit there are also many areas, like athletic wear, where this isn't usually the case. On the other hand, I've often noticed that if you actually compare some types of very high end activewear to less expensive variants, the difference is often just the logos. You're in effect paying more to have the logos off rather than less.
I personally feel that companies should be shamed out of including visible logos on their clothing.
Maybe we're talking about different things though. I do think having brand identifiers of some sort somewhere is important, as it increases accountability. And some things that are likely to be shuffled away, like laundry detergent, I don't really care about. But things that are likely to be out and about? Especially on you as a person? Brand visibility shouldn't be a factor imho.