There are actually a few etymologies for the term in architecture. It does often imply a deliberate crudeness and a harshness, supposedly an honest response to the character of the materials. To the untrained eye this is close enough to "ugly" that most people don't distinguish. There's also a very strong argument that brutalism represents a turn away from the values of western humanism (typically expressed, historically, in classical architecture) toward a vulgar, bleak conception of modernity. Another aspect is that it tends to be "sublime" rather than "beautiful", which, again, translates to negative vibes for many people.
It's not a term that I'm happy to see being applied now to some essentially cosmetic aspects of web design.