The point of visible branding is that you leverage the brand value and marketing yourself. By purchasing the product and showing the label, you rely on other people's opinions of the brand to colour their opinion of you. This can work either way, of course, but you get to choose the brand and, indirectly, the type of people you're trying to change the opinion of.

Chinos are a brand all of their own, just not a trademarked brand. They're usually (if they're pastel coloured) relatively conservative, business casual, more formal than jeans but less formal than a suit. A bit middle-aged, or something you'd expect to see a businessman in on the weekend. You can't really escape branding when you appear in public. Even if you showed up naked, it would be a brand.

I think there's a decent listing in https://www.upcounsel.com/trademark-vs-brand on the elements of brand:

    Identity
    Image
    Personality
    Character
    Culture
    Essence
    Reputation

You mark out which group you're a member of in part by the clothes you choose. Some groups are anti-labels and you're partially opting in to them by eschewing labels.