It's almost always because they're almost all expensive maintenance nightmares decaying and crumbling to dust and rubble after just a few years of use resulting in unpleasant, dirty looking buildings and public spaces around them.
I would be highly surprised if even 10% of brutalist era buildings survive their first 50 years without being torn down.
They're almost universally poorly designed and built, feeling more like a parking garage or a public restroom, with interior floor layouts based on mental hospitals or mazes.
There are beautiful brutalist buildings. But I'd argue that even only a small percentage of those on the fuckyeahbrutalism blog would be considered so by the public. FYB even calls out City Hall in Boston as an example of great architecture, but it's pretty much reviled by anybody who's even come in cursory contact with it and the surrounding square. There's lots of terrible public spaces on that blog to be honest, clever photography is what makes most of them passable.