There are actually a few etymologies for the term in architecture. It does often imply a deliberate crudeness and a harshness, supposedly an honest response to the character of the materials. To the untrained eye this is close enough to "ugly" that most people don't distinguish. There's also a very strong argument that brutalism represents a turn away from the values of western humanism (typically expressed, historically, in classical architecture) toward a vulgar, bleak conception of modernity. Another aspect is that it tends to be "sublime" rather than "beautiful", which, again, translates to negative vibes for many people.
It's not a term that I'm happy to see being applied now to some essentially cosmetic aspects of web design.
I share a similar peeve as you with the use of "women" as an adjective. All through grade school and high school, the use of "women" as a noun was pummeled into my head. Now, I routinely see presumably college-educated journalists use it as an adjective ("women scientists", "women engineers", etc.)
If you would now go ahead and use it to describe a concept of a different scope where it does not apply at all, say web designs, you could do that, but you would expose that you just tried to name something in a sciency way to sound sophisticated without having any idea what the term actually means.
Indeed, this is by far the usual case. People collectively can, and do, reject almost all proposed novel usages, which are quickly forgotten. We just remember the few cases where things change due to survivorship bias.
"People" aren't really a useful measure here because there is slang that achieves popularity out of the subculture that produces it, subcultures composed of people, just the same as it may not propagate, in which case it naturally remains useful to the in-group. To reject slang as valid language until it achieves mainstream acceptance is a) not measurable as there is no line that is crossed, and b) antisocial.
Anyway, I came up with a better definition: accepting a word/usage is believing that it's a good way to express whatever it was.
I myself will never use "literally" to mean "very," which differentially lowers the chance of this new meaning becoming standardized. Over time, either most people will do the same and the proposed novel usage of "literally" to mean "very" will die out, or else most people will collectively adopt the new meaning.
Empirically, almost all novel usages die out very quickly and are forgotten. Read some slang from the 1920s -- it's already almost incomprehensible to a modern reader.
As for the term itself, there's always a risk of imprecision when laypeople discuss the concepts used in certified professions.
There's a 2004 paper along the lines you mention by Simon Yuill: http://www.lipparosa.org/essays/code_brut.pdf
Edit: Somebody should write a manifesto about it.