Brutalism is one of those things that means many things to many people, and gets
brutally simplified (heh) in the public eye. There's more to any brutalist building than the architect's own idea of brutalism, and of course each architect's idea of what brutalism can be is more than the stripped-down least common denominator definition.
For example, one of the ideas behind using raw concrete for public buildings instead of something sleeker and more expensive is to make public institutions more welcoming and accessible to ordinary people. Traditionally, public buildings were built to project the prestige and power of the class of people who ran elite institutions, at the expense of common people who themselves might feel a bit raw and brutal compared to the sleek and expensive buildings where the sleek and expensive elites ran the world in their sleek and expensive suits. Raw concrete is supposed to communicate that the building, and the institution inside, is (at least supposed to be) working for the public, not spending their money to elevate and aggrandize the people in power.
In other words, in those cases, concrete is meant to create a relatable and accessible rawness, like Gritty the Philadelphia Flyers mascot.
I think it works out a lot better in reality than people think, when a building is well-designed. Architectural photography can hide ugliness — almost every building has its equivalent of the flattering "Myspace angle" — but it can also manufacture it, and with the right lighting, angle, and exposure, it's easy to make any tall concrete building look like a gloomy looming hulk.
Local feelings are a better guide to whether a building is good or not than the feelings of people who have never seen a building in person and have been primed by tendentious photography.